Hybrid expert LLM inference engine architecture and its scheduling method

Abstract

The application discloses a hybrid expert LLM reasoning engine architecture and a scheduling method thereof, and the architecture comprises a double transmission path, a cross-clock domain isolation layer, an internal core computing domain, a phase scheduling controller and a flow control scheduling module. The first transmission path and the second transmission path are respectively used for establishing transmission paths of a hybrid expert weight and a key-value pair cache which are independent of each other; the cross-clock domain isolation layer is used for synchronizing asynchronous data flow between a system bus and a core computing clock domain; the core computing domain selectively guides data flow to a computing array through a crossbar switch; the phase scheduling controller controls the double paths to pass through the crossbar switch in phases A and B in mutual exclusion; and the flow control scheduling module connects a request configuration end of the first transmission path based on an obtained cross-clock domain isolation layer on-chip cache fullness state signal to realize closed-loop control of bus requests. The application effectively solves memory access conflict and congestion in long context reasoning, and significantly reduces end-to-end reasoning delay.

Description

Technical Field

[0001] This invention relates to the field of edge computing and large language model (LLM) hardware acceleration technology, specifically to a Mixture of Experts (MoE) LLM inference engine architecture and its scheduling method. Background Technology

[0002] As Large Language Models (LLMs) migrate to edge devices, Hybrid Expert (MoE) architectures have become mainstream due to their advantage in balancing parameter and activation computation costs. However, edge hardware resources are limited, and sparse reads of MoE weights and continuous streaming accesses to the key-value cache (KV Cache) in a shared bus architecture lead to intense resource contention. Traditional arbitration mechanisms cannot provide deterministic latency guarantees, and in complex System-on-a-Chip (SoC) environments, asynchronous interference between different clock domains further increases the uncertainty of inference latency.

[0003] Therefore, developing an inference engine architecture that can achieve physical isolation of data paths and has deterministic scheduling capabilities has become the key to improving the interactive experience of edge AI. Summary of the Invention

[0004] This invention aims to address the technical problem of uncertain inference latency caused by asynchronous interference between different clock domains in complex system-on-a-chip (SoC) environments for current large language models. It provides an inference engine architecture and its scheduling method, which provides deterministic latency guarantees to achieve agile adaptation during model updates. While maintaining the deterministic performance baseline of core data flow, it significantly reduces the iteration cost and deployment threshold of edge AI hardware.

[0005] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, embodiments of the present invention provide a hybrid expert-based LLM inference engine architecture, comprising:

[0007] The system has two transmission paths, including a first transmission path and a second transmission path, which are used to establish an independent hybrid expert weight transmission path and a key-value pair cache transmission path, respectively.

[0008] A cross-clock domain isolation layer has its input terminals connected to the first and second transmission paths, respectively, and its output terminal connected to the internal core computing domain. It is used to synchronize the asynchronous data flow between the system bus clock domain and the core computing clock domain.

[0009] The core computing domain includes a crossbar switch and a computing array connected thereto. The input terminals of the crossbar switch are coupled to the first transmission path and the second transmission path respectively via the cross-clock domain isolation layer, for selectively directing data streams to the computing array.

[0010] A phase scheduling controller, whose control output terminal is connected to the first transmission path, the second transmission path and the cross switch respectively, is used to control the first transmission path and the second transmission path to pass through the cross switch mutually exclusively in phase A and phase B by outputting a phase switching control signal;

[0011] The flow control scheduling module has its input end connected to the cross-clock domain isolation layer to obtain the full / empty status signal of the internal asynchronous queue, and its output end connected to the request configuration end of the first transmission path. It is used to realize elastic backpressure closed-loop control of bus requests based on the data buffer margin of the asynchronous queue.

[0012] Secondly, embodiments of the present invention provide a hybrid expert-based LLM inference scheduling method, wherein the scheduling method is applied to an architecture provided by any possible implementation of the first aspect, and the method includes the following steps:

[0013] Step S1. Phase hard-cut scheduling: The phase scheduling controller receives the layer counting signal and divides the single-layer decoding cycle into phase A and phase B by hard-cutting the cross switch;

[0014] Step S2. Data Mutual Exclusion Transmission: During phase A, enable the first transmission path to transfer the mixed expert weights and suspend the second transmission path; during phase B, enable the second transmission path to read the key-value pair cache and suspend the first transmission path.

[0015] Step S3. Double-pass block attention calculation: The main computing cluster alternately reads ready data slices and corresponding activation expert weights from the first or second on-chip cache in a streaming manner. In an intra-layer temporal overlapping manner, the key-value pair cached data in the data slice is divided into N blocks and the first pass of block score calculation and the second pass of weighted accumulation calculation are executed in sequence. At the same time, the expert network matrix multiplication calculation dominated by the activation expert weights is executed in a coordinated manner to complete the double-pass block attention pipeline calculation.

[0016] Step S4. Dynamic flow control coordination: The flow control scheduling module monitors the full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer in real time. When the backlog of the internal asynchronous queue reaches a preset threshold or triggers a blocking condition, a pause or wait signal is sent to the first transmission path to suspend the current weighted data transfer.

[0017] Compared with the prior art, the hybrid expert-type LLM inference engine architecture and its scheduling method provided in this embodiment of the invention have the following technical advantages:

[0018] 1. Traditional edge-side large-model inference engines typically share a single bus interface. In long-context scenarios, massive key-value pair data streams can severely collide and interlock with bursty expert weight data streams on the bus. This invention achieves off-line transmission between high-bandwidth dynamic expert weights and deterministic, low-latency key-value pair caches through physically independent or logically isolated dual transmission paths. This fundamentally solves memory access conflicts during long-sequence inference from the underlying topology, greatly improving overall throughput.

[0019] 2. This invention eliminates the need for deploying an extremely complex, fully dynamic software routing network on-chip. Instead, it applies time-division gating directly at the hardware layer through a phase scheduling controller. Under different computation phases, it forces the weighted stream and key-value pair to "travel off-peak," achieving asynchronous streaming data interaction and continuous, seamless operation of the computing array within a very limited on-chip cache space with extremely low hardware requirements and zero software scheduling overhead.

[0020] 3. The on-chip network (NoC) of edge chips is highly susceptible to sudden congestion caused by interference from other peripherals, resulting in data write failures or inability to retrieve data. This invention uses a flow control scheduling module to capture the full / empty state of the lowest-level cache in real time (such as FIFO level or bus handshake state), establishing a fully hardware-adaptive backpressure control loop. When the external bus experiences severe jitter or congestion, it can immediately suspend requests or the compute array (Stall) to prevent on-chip buffer overflows or metastable crashes; it recovers instantaneously after the congestion is cleared, ensuring system-level hard real-time deterministic latency during long-sequence inference.

[0021] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this application in any way. In the drawings:

[0023] Figure 1 A block diagram of a hybrid expert-type LLM inference engine architecture provided for an embodiment;

[0024] Figure 2 The block-based online double-pass normalized exponential data flow graph in the hybrid expert-type LLM inference scheduling method provided for the embodiments;

[0025] Figure 3 This is a timing diagram of the DeltaNet state four-slice ping-pong pipeline in the embodiment;

[0026] Figure 4This is a schematic diagram of the state transition of the elastic prefetch and reverse pressure state machine in the embodiment;

[0027] Figure 5 This is a schematic diagram of the hardware-level gating logic of the phase scheduling controller over the entire inference engine in the embodiment. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0029] The core technical challenges faced when deploying large-scale MoE-LLM inference engines in edge computing scenarios:

[0030] 1. Bus contention between "delay-sensitive streams" and "bandwidth-sensitive block transport": In a single-chip SoC architecture, the KV Cache (key-value pair cache) reads during the Attention phase have strict real-time requirements and zero-jitter characteristics, while MoE expert weight transport involves large-block burst transmissions. If both share the system bus (NoC), it will cause uncontrollable congestion jitter and degrade decoding throughput.

[0031] 2. On-chip SRAM capacity is insufficient to accommodate each token: The state amount of a single layer of a hybrid expert model (such as DeltaNet) (~512KB / layer) far exceeds the physical limit of on-chip SRAM of eFPGA (usually ~128KB / block). How to complete the complete state update under limited on-chip storage becomes a bottleneck.

[0032] 3. Uncontrollable propagation of NoC congestion to the computing core domain: When system-level NoC becomes congested due to CPU preemption or memory controller arbitration, the eFPGA computing pipeline in traditional architecture is directly exposed to bus latency jitter, causing computing units to frequently idle and wait, resulting in significant fluctuations in throughput.

[0033] 4. Coordination of Dynamic Routing and Deterministic Computation in Hybrid Expert Models: MoE's Gating network dynamically determines the activated experts (Top-K) for each token, and the transfer of expert weights must be initiated after the routing is determined. Maintaining continuous saturation of the computation pipeline under uncertain latency is a major challenge.

[0034] 5. The contradiction between "hardware architecture solidification" and "rapid model iteration": LLM architecture evolves extremely rapidly (expert scale, attention variants, quantization strategies, and routing gating logic are frequently adjusted). Traditional ASIC hardware is solidified, and model updates require re-tape-out or equipment replacement. On the other hand, pure software logic eFPGA solutions lack a deterministic data flow scheduling skeleton, and after reconfiguration, clock frequency and pipeline depth are difficult to guarantee, making it impossible to balance flexibility with the stringent latency / power consumption constraints on the edge side.

[0035] Large language models have made groundbreaking progress in fields such as natural language processing and multimodal understanding. MoE models, represented by DeepSeek-V2 / V3, employ a "sparse activation" strategy—each token activates only a few experts among all experts (e.g., 8+1 out of 256), significantly reducing the equivalent computational cost of a single inference. Meanwhile, some implementations use linear attention mechanisms such as gated DeltaNet to further optimize computational efficiency in long-context scenarios. However, deploying such MoE models with hundreds of billions of parameters to edge devices (power consumption constraints <50W, limited physical space) still faces significant challenges.

[0036] The closest existing technologies include the following solutions:

[0037] 1. Pure GPU (Graphics Processing Unit) / ASIC (Application-Specific Integrated Circuit) inference acceleration architecture (such as NVIDIA TensorRT-LLM).

[0038] NVIDIA TensorRT-LLM is an official product from NVIDIA that utilizes a unified, high-capacity HBM (High Bandwidth Memory) memory pool. It stores weights, key-value pairs (KV) cache, and intermediate states in the same storage tier via NVLink (NVIDIA Link) / PCIe (Peripheral Component Interconnect Express) interconnects. All data transfers occur through the same interconnect network, and the attention mechanism and MLP operators are executed through CUDA (Compute Unified Device Architecture) / Tensor Cores.

[0039] 2. FPGA-accelerated Dense LLM (Dense Large Language Model) inference scheme (such as Xilinx FINN+DPU architecture): The quantized Dense LLM computation graph is mapped to the FPGA fixed dataflow accelerator, and accessed through a single AXI Master interface to the onboard DDR (Double Data Rate Synchronous Dynamic Random Access Memory). All operators and data transport share a bus and use a simple FIFO (First-In-First-Out) buffer.

[0040] 3. CPU+GPU hybrid offloading scheme (e.g., Mixtral-Offloading / MoE-Infinity): On the x86+GPU platform, inactive expert weights are offloaded to host memory, and then moved to GPU memory via PCIe when a route is selected. Expert residency is managed using LRU (Least Recently Used) / LFU (Least Frequently Used) cache, and all transfers occur via the same PCIe bus.

[0041] 4. In-memory computing / near-memory computing acceleration architecture: Deploy MAC (Multiplier-Accumulator) arrays near DRAM Banks (physical blocks of memory arrays) to reduce weight transfer requirements, but have limited hardware support for MoE dynamic routing and complex attention mechanisms (Chunked Attention / DeltaNet).

[0042] Disadvantages of existing technology:

[0043] 1. GPU / ASIC solutions consume too much power (≥100W) and are not suitable for edge computing scenarios; the unified bus architecture lacks a physical isolation mechanism, and latency jitter is unavoidable under high load.

[0044] 2. The FPGA single-bus architecture cannot distinguish between "delay-sensitive flow" and "bandwidth-sensitive block"; it does not support MoE dynamic expert routing; the on-chip SRAM management strategy is crude and cannot handle large state models.

[0045] 3. The CPU+GPU solution has excessive PCIe latency (microsecond level), resulting in millisecond-level latency accumulation with each expert switch; calculation and data transfer are sequential, and the GPU idles while waiting for data.

[0046] 4. The in-memory computing technology has low maturity and high design complexity; there is insufficient hardware support for complex operators such as DeltaNet outer product update and ChunkedOnline Softmax.

[0047] 5. Traditional acceleration chips have long iteration cycles and high sunk costs in tape-out; existing FPGA chip acceleration solutions mostly use all-soft logic to implement scheduling and computation, lacking an architecture isolation design of "fixed control skeleton + reconfigurable computing domain". After reconfiguration, the critical path increases and the clock frequency drops sharply (often below 200MHz), resulting in large performance fluctuations and loss of deterministic latency advantage.

[0048] The embodiment, a hybrid expert-type LLM inference engine architecture, includes dual transmission paths, a cross-clock domain isolation layer, a core computing domain phase scheduling controller, and a flow control scheduling module.

[0049] The dual transmission paths include a first transmission path and a second transmission path, which are used to establish mutually independent hybrid expert weight transmission paths and key-value pair cache transmission paths, respectively.

[0050] The input terminals of the cross-clock domain isolation layer are connected to the first transmission path and the second transmission path respectively, and the output terminal is connected to the internal core computing domain, which is used to synchronize the asynchronous data flow between the system bus clock domain and the core computing clock domain.

[0051] The core computing domain includes a crossbar switch and a computing array connected thereto. The inputs of the crossbar switch are coupled to a first transmission path and a second transmission path via a cross-clock domain isolation layer, respectively, for selectively directing data streams to the computing array.

[0052] The phase scheduling controller has its control output terminals connected to the first transmission path, the second transmission path, and the cross switch, respectively. It is used to control the first transmission path and the second transmission path to pass through the cross switch mutually exclusively in phases A and B by outputting a phase switching control signal.

[0053] The input of the flow control scheduling module is connected to the cross-clock domain isolation layer to obtain the full / empty status signal of the internal asynchronous queue, and its output is connected to the request configuration terminal of the first transmission path, which is used to realize the elastic backpressure closed-loop control of bus requests based on the data buffer margin of the asynchronous queue.

[0054] In some embodiments, the first transmission path includes a bus request generation module (such as a descriptor generator). The output of the flow control scheduling module is connected to the bus request generation module of the first transmission path.

[0055] It should be understood that, in some embodiments, the closed-loop control of bus requests based on the data cache availability of the internal asynchronous queue across the clock domain isolation layer can specifically manifest as follows: when the internal asynchronous queue cache is insufficient, the flow control scheduling module drives the descriptor generator to stream direct memory access (DMA) descriptors, thereby initiating a data transfer request to the external system memory. In other equivalent alternative embodiments, bus requests can also be initiated by directly configuring the bus registers using a finite state machine (FSM) or by streaming through hard-wired request signals.

[0056] The embodiments of the present invention can provide deterministic delay protection. The physical isolation of the rigid dedicated line of the second transmission path makes the reading delay of the Attention stage completely deterministic and unaffected by the system NoC load.

[0057] In some embodiments, the dual transmission paths can be physically isolated bus interface subsystems, with the first transmission path including a first interface and the second transmission path including a second interface, respectively used to establish a hybrid expert weight transmission path and a key-value pair cache transmission path.

[0058] The first interface (M0) connects to the system-level DDR via a network-on-chip (NoC) and embeds a Scatter-Gather (SG) DMA (Direct Memory Access) controller. It supports INCR256 burst transmission and descriptor chain prefetching for resiliently moving Mixture of Experts (MoE) weights and model states (such as routing state, synchronization state, etc.).

[0059] The second interface (M1) is directly connected to the private DDR, PHY (physical layer interface circuit), and interface controller via a dedicated physical line, and is dedicated to deterministic streaming reads of the key-value cache (KV Cache). The dedicated address generator of M1 consists only of a base address register (BaseAddr) and an offset accumulator (TailPtr), realizing an absolutely flat physical address mapping for the KV Cache and eliminating page table traversal and virtual address translation.

[0060] In some embodiments, both the first interface and the second interface are AXI Master interfaces. AXI (Advanced Dextensible Interface) is a bus protocol defined by ARM.

[0061] In some embodiments, the transmission path may also be manifested as virtual channels with different Quality of Service (QoS) under a shared physical bus.

[0062] As an example, if physical isolation between the two buses cannot be achieved due to limitations in eFPGA pin resources, the two transmission paths can adopt a single bus plus hardware-level QoS virtual channel (VC) priority scheduling, i.e., different QoS virtual channels under a shared physical bus. High-priority VCs are assigned KV streams, and low-priority VCs are assigned mixed expert weight transport. Although this example cannot provide absolute determinism for the second interface (M1) leased line under heavy congestion, it can significantly reduce latency and jitter.

[0063] In some embodiments, such as Figure 1As shown, the flow control scheduling module is a flexible prefetch and backpressure state machine. The cross-clock domain isolation layer includes a system asynchronous FIFO located between the first transmission path and the cross switch; a private asynchronous FIFO located between the second transmission path and the cross switch; cross-domain processing logic used to perform clock synchronization processing on the phase switching control signal output by the phase scheduling controller; the full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer is specifically the real-time water level signal of the system asynchronous first-in-first-out queue; the flexible prefetch and backpressure state machine compares the real-time water level signal with the preset cascade water level threshold to execute dynamic backpressure.

[0064] like Figure 1 As shown, the system bus interface is coupled to the system-level DDR and the distributed-aggregated DMA through the first interface (M0); the private bus interface is directly connected to the private DDR through the second interface (M1). When crossing clock boundaries, the two buses are metastable isolated through the system asynchronous FIFO (SYS Async FIFO) and the private asynchronous FIFO (PVT Async FIFO) in the clock domain layer, respectively.

[0065] In some embodiments, the core computing domain employs a hybrid precision scheduling strategy of 8-bit fixed-point (INT8) full-link quantization and 16-bit half-precision floating-point (FP16) normalized exponent (Softmax). The computing array integrates a dequantization module, which is deployed at the output tail of the normalized exponent unit to dynamically convert the streamed fixed-point data into 16-bit half-precision floating-point (FP16) format for subsequent accumulation, in order to suppress numerical precision collapse in long context scenarios.

[0066] In some embodiments, the architecture is deployed in an Embedded Field Programmable Gate Array (eFPGA) domain; the core computing domain is built using the reconfigurable logic resources of the eFPGA.

[0067] eFPGA reconfigurability and performance guarantee mechanism: The architecture is divided into a "deterministic control skeleton" and a "reconfigurable computing domain". The deterministic control skeleton includes physically isolated dual-bus interfaces, time-division hard-cut crossbar logic, waterline reverse voltage FSM (Finite State Machine), cross-clock domain Gray code synchronizer, and hard-wired address generator, implemented using ASIC static routing to ensure clock frequency stability.

[0068] The reconfigurable computation domain includes the clustering ratio of the multiply-accumulate unit (MAC) array, the depth of the sorting tree in the gate control section, the chunk / slice partitioning counter, the asynchronous FIFO depth, and the quantization and dequantization modules. When the model architecture or operator strategy is updated, it is only necessary to resynthesize the reconfigurable computation domain and download the configuration bitstream. The deterministic control skeleton remains static and fixed, which can guarantee the pipeline cavitation rate and clock frequency after reconfiguration.

[0069] This invention enables extended hardware lifecycles and agile deployment. Specifically, based on a "fixed skeleton + reconfigurable computation" eFPGA architecture, it supports rapid adaptation to different numbers of experts (e.g., Top-2 → Top-8), chunk sizes, DeltaNet slicing strategies, and attention variants (e.g., FlashAttention / ChunkedAttention) by updating the bitstream, without requiring hardware tape-out. While ensuring a deterministic latency baseline, it shortens the model iteration cycle from months to days, significantly reducing the sunk cost of edge AI hardware and providing underlying hardware flexibility for private model inference. The embodiment also enables low-power reconfigurability at the edge: the eFPGA programmable logic flexibly adapts to the MoE topology, and the overall inference IP power consumption meets the low-power system constraints of edge devices.

[0070] The architecture provided in this embodiment also achieves a balance between computing power and bandwidth limits in long contexts: it natively supports long contexts (determined by the eFPGA's local private DDR RAM capacity). Attention computing power and bandwidth are matched at the microsecond level, with no single dimension becoming a bottleneck. End-to-end low latency for the first token: the latency of the first token is determined by the system prefill stage, and the remaining latency is mainly contributed by the CPU asynchronously executing 508MB LM Head projection (commonly known as language model head projection) in the system DDR. There is stage overlap between the eFPGA pipeline and the LM Head projection, and the overall process does not block the prefetching of the next token.

[0071] In some embodiments, the core computing domain also includes two independent on-chip caches (Bank A and Bank B), which are coupled to the outputs of a crossbar switch. Bank A and Bank B can be directly connected to M1 and can be used specifically for rolling storage of KV Cache data matrices pulled in from external private memory.

[0072] In one specific embodiment, the model state is the hidden state matrix that DeltaNet dynamically updates during inference; the physical capacity of the first on-chip cache and the second on-chip cache are each independently configured to 128KB, while the data volume of a single layer of DeltaNet hidden state is 512KB; by dividing the single layer of DeltaNet hidden state into M slices (M≥2) according to the head dimension, the data volume of a single slice is adapted to the physical upper limit of the first or second on-chip cache, thereby trading time slicing for space occupancy.

[0073] As an example, such as Figure 1 As shown, unlike traditional FPGAs that use small blocks of BRAM, some embodiments of the present invention directly use larger capacity on-chip SRAM (Static random-access memory) to implement DNet SRAM (Bank A / B), score cache (Score SRAM) (dual-port), and result cache (Result RAM).

[0074] The computing array is a reconfigurable MAC array in the eFPGA, divided into primary computing clusters (such as...). Figure 1 The computational cluster A (which can be 64 MAC) and the gate control dedicated computational cluster (such as...) Figure 1 In the computation cluster B, gating and position decoding are both considered, such as Figure 1 Cluster B in the middle can be 64 MAC + RoPE. RoPE stands for Rotary Position Embedding.

[0075] In this embodiment, the hybrid expert-type LLM inference engine architecture further includes a partial sorting tree unit, a gate control dedicated computing cluster for calculating and outputting gate control routing decision results, the output of the gate control dedicated computing cluster being connected to the partial sorting tree unit, the partial sorting tree unit being used to generate expert activation indexes based on the gate control routing decision results, and configuring the expert activation indexes to the first transmission path to drive the corresponding bus descriptor submission.

[0076] The core computing domain also includes a first on-chip cache and a second on-chip cache that are independent of each other. The input of the main computing cluster is coupled to the first on-chip cache and the second on-chip cache via a cross switch. When the first or second transmission path performs a streaming pull operation across the cross-clock domain isolation layer, it alternately reads ready data slices and corresponding activated expert weights from the first or second on-chip cache. It also controls the main computing cluster to dynamically execute the double-pass block attention pipeline computation dominated by data slices and the expert network matrix multiplication computation dominated by activated expert weights in an intra-layer timing overlap manner.

[0077] The flow control scheduling module performs elastic backpressure control to pause or resume the first transmission path of the submitted bus descriptor based on the full / empty status signal of the internal asynchronous queue in the cross-clock domain isolation layer, thereby jointly realizing closed-loop control of bus requests.

[0078] In a specific embodiment, the number of multiply-accumulate units in the multiply-accumulate array (MAC) can be proportionally reduced or expanded according to the target computing power requirements (e.g., reduced to 32 MACs or expanded to 128 MACs or more). Simultaneously, the chunk size of the attention mechanism can also be dynamically adjusted. Reducing the chunk size can significantly reduce the size requirement of the score SRAM, but increases the row miss overhead caused by double-pass data flow; increasing the chunk size can improve the amortization efficiency of the row miss overhead. This invention allows for multi-objective collaborative optimization between chunk parameters and on-chip cache area based on the power consumption limit and bandwidth limit of the target chip. In this embodiment, the score cache can be 32KB, and the result cache can be 16KB.

[0079] In this embodiment, the dedicated computing cluster for gate control (Gating dedicated cluster) performs expert scoring matrix multiplication. The output of the core computing domain is coupled to a Top-K selection module, which is used to complete the streaming screening of the top K most probable candidate tags (Top-K) through a partial sorting tree, where K is a positive integer preset through the APB control interface. The calculated expert activation routing result directly drives the SG DMA descriptor submission, thereby initiating the expert weight parameter transfer of the target activation expert network, and making the weight parameter transfer and the expert network matrix multiplication calculation being performed in the main computing cluster for the previous data slice form a temporally overlapping pipeline.

[0080] A phase scheduling controller, whose control output is connected to a first transmission path, a second transmission path, and a cross switch, is used to control the first and second transmission paths to pass through the cross switch mutually exclusively in phases A and B by outputting a phase switching control signal.

[0081] In the embodiments, such as Figure 5 As shown, the phase scheduling controller interface uses a token / layer count signal to output a phase switching control signal that hard-switches the crossbar. Each Transformer Block decoding cycle is divided into mutually exclusive phase A (flexible window) and phase B (rigid window). During phase A, the crossbar connects M0 to the MAC array, and M1 is clock-gated; during phase B, the crossbar connects M1 to the MAC array, and M0 is suspended. A built-in hardware timer monitors the phase duration, allowing for ±10% fine-tuning of the flexible window.

[0082] In some embodiments, the phase scheduling controller is deployed in the core computing domain.

[0083] In this embodiment, the flexible prefetch and reverse pressure state machine (not shown in the figure) is controlled by a water level line. The input is connected to the real-time water level signal inside the cross-clock domain isolation layer, and the output is connected to the descriptor generator of the first transmission path. It is used to realize closed-loop control of bus requests based on the remaining data buffer of the internal asynchronous queue.

[0084] The flexible prefetch and reverse-pressure state machine monitors the FIFO (Gray code synchronization) level within the cross-clock domain isolation layer. The level-driven DMA flexible prefetch state machine (FSM) is as follows: Figure 4 As shown, when the water level is lower than the preset lower threshold, the DMA descriptor is automatically submitted to initiate prefetching; when the preset upper threshold is reached, a pause signal is sent to the SG DMA to stop the DMA from initiating new requests, thus achieving complete decoupling between the computing core and NoC congestion.

[0085] like Figure 4 As shown, the specific transition and control logic of the elastic prefetch and reverse pressure state machine is as follows:

[0086] 1. Initialization and prefetching triggering phase (IDLE → CHECK_LEVEL → SUBMIT_DESC):

[0087] Idle state (IDLE): When the engine is reset or the inference task is not started, the state machine remains in the idle state, and the bus channel is closed.

[0088] Cache water level check state (CHECK_LEVEL): Once inference starts and the long context data stream begins to flow, the state machine enters the water level check state. When it is detected that the cache water level of the on-chip Async FIFO is lower than the lower threshold (FIFO_LEVEL < TH_LOW), it means that the on-chip raw material warehouse is about to run out, and a prefetch signal is immediately triggered.

[0089] Descriptor Submission State (SUBMIT_DESC): The state machine transitions to the submission state, driving the descriptor generator of Scatter_GatherDMA to stream new pickup lists (submit DMA descriptors).

[0090] 2. Steady-state flow transport stage (Running):

[0091] Data transfer running state: Once the descriptor is ready (DESC_READY), the state machine immediately enters the running state. At this time, the first host interface (M0 bus) is operating at full capacity, pulling MoE expert weight data from the external system memory and continuously inputting it onto the chip.

[0092] 3. Dynamic back pressure control stage (RUNNING→PAUSE_WAIT→RESUME)

[0093] Pause-Wait State: This is the core defense mechanism of this invention against system congestion. When congestion occurs in the on-chip network (NoC) or the computing array slows down, causing asynchronous FIFO backlog and cache water level to exceed the upper limit threshold (FIFO_LEVEL > TH_HIGH), the state machine immediately switches to the pause-wait state. At this time, DMA transfer is forcibly stopped (suspended back pressure) to prevent on-chip cache overflow.

[0094] Resume transmission state (RESUME): The state machine continues to listen in the paused state until the computing array consumes the backlog of data, causing the cache water level to drop safely below the median threshold (FIFO_LEVEL < TH_MID). The state machine then transitions to the resumed state, reactivating bus streaming and pulling the pipeline to continue rolling.

[0095] 4. End of cycle and loop closure (RESUME → IDLE):

[0096] Closed-loop regression: Once the weight data of the activated expert network (i.e., the candidate expert network selected by Top-K) corresponding to the current slice or the current layer has been completely transferred (DMA_DONE), the state machine safely returns from the recovery state to the idle state (IDLE), waiting for the next token or the scheduling cycle of the next layer.

[0097] In some embodiments, the flexible prefetch and reverse pressure state machine is embedded in the spread-aggregate DMA controller of the first transmission path, or reverse pressure is achieved by the finite state machine in the core computing domain through direct control of the bus handshake signal; it performs reverse pressure decision by receiving the FIFO level signal fed back from the cross-clock domain isolation layer, and coordinates to receive the computing tick signal output by the chunk state machine (Chunk FSM) in the core computing domain, so as to dynamically control the triggering and suspension of the descriptor generator, and realize the closed-loop matching between the bus data pull speed and the on-chip core computing speed.

[0098] In another alternative embodiment, if the flexible prefetch and backpressure state machine based on direct memory access (DMA) and pause mechanisms is not used, the bus handshake control logic of the network-on-chip (NoC) can be decentralized. The finite state machine (FSM) within the core computing domain directly controls the valid / ready handshake signals of the AXI bus and implements the bus retry logic itself. When the valid / ready handshake signals cross the boundary between the external bus clock domain and the core computing domain, they undergo level pacing or pulse clock synchronization processing through a cross-clock domain isolation layer to eliminate metastability caused by asynchronous sampling. The synchronized handshake state then represents the full / empty state of the internal asynchronous queue. This scheme effectively reduces the hardware area of ​​the asynchronous first-in-first-out queue (Async FIFO) in the cross-clock domain layer, but it introduces protocol complexity into the computing core, requiring a trade-off between specific verification costs.

[0099] In some embodiments, the architecture further includes a hardware-level streaming decompression unit embedded in the data path of the first or second transmission path, used to perform real-time, pipelined decoding of compressed data during the transport process.

[0100] In some embodiments, a fully pipelined streaming codec unit is embedded in the first transmission path and / or the second transmission path; the streaming codec unit is used to perform real-time hardware-level cascaded compression or decompression before / after data access on-chip storage, and its decompression delay overlaps and masks the computing pipeline of the computing array.

[0101] As an example, if limited by DDR physical bandwidth, NoC channel capacity, or on-chip SRAM resident size, hardware-level streaming compression / decompression units can be embedded in the M0 and M1 data paths. These units employ a fully pipelined architecture, with data being encoded and decoded in real-time before and after transmission via the AXI interface or reading / writing to on-chip storage. Decompression output can be triggered without requiring the entire data block to reside, making it transparent to the upper-layer scheduling controller. Compression effectively reduces data volume, directly lowering the actual bus load on M0 / M1; decompression latency is completely masked by the overlapping of the "pre-decompression-computation" pipeline. This solution can significantly alleviate NoC congestion and storage capacity / performance walls in bandwidth-sensitive scenarios, serving as a low-cost or performance-enhancing supplementary path to dual-bus isolation architectures.

[0102] For ultimate reconfigurability and flexibility, in another variation of this invention, the scheduling logic of the crossbar and the direct memory access (DMA) state machine can be entirely software-based and implemented using programmable logic resources (eFPGA Fabric) to support dynamic reconfiguration of the entire bus topology at runtime. Although this approach may result in increased critical path latency and a slight decrease in clock frequency, it endows the architecture with extremely high topology evolution capabilities.

[0103] This application also provides a hybrid expert-based LLM inference scheduling method, applied to the hybrid expert-based LLM inference scheduling architecture provided in the above embodiments, the method comprising:

[0104] Step S1. Phase hard-cut scheduling: The phase scheduling controller receives the layer counting signal and divides the single-layer decoding cycle into phase A and phase B through a hard-cut cross switch;

[0105] Step S2. Data Mutual Exclusion Transmission: During phase A, enable the first transmission path to transfer the mixed expert weights and suspend the second transmission path; during phase B, enable the second transmission path to read the key-value pair cache and suspend the first transmission path.

[0106] Step S3. Double-pass block attention calculation: The main computing cluster alternately reads ready data slices and corresponding activation expert weights from the first or second on-chip cache in a streaming manner. In an intra-layer temporal overlapping manner, the key-value pair cached data in the data slice is divided into N blocks and the first pass of block score calculation and the second pass of weighted accumulation calculation are executed in sequence. At the same time, the expert network matrix multiplication calculation dominated by the activation expert weights is executed in a coordinated manner to complete the double-pass block attention pipeline calculation.

[0107] Step S4. Dynamic flow control coordination: The flow control scheduling module monitors the full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer in real time. When the backlog of the internal asynchronous queue reaches the preset threshold or triggers the blocking condition, a pause or wait signal is sent to the first transmission path to suspend the current weighted data transfer.

[0108] In this embodiment, when each layer of decoding starts, a hardware counter triggers a phase switch. During phase A, M0 is connected to move the MoE weights and M1 is suspended. During phase B, M1 is connected to stream read the KV Cache and M0 is suspended. The two phases are executed mutually exclusively.

[0109] In some embodiments, in step S2, the hybrid expert weight data carried by the first transmission path and the key-value pair cache data read by the second transmission path are aligned and buffered in their respective independent asynchronous clock domains when passing through the cross-clock domain isolation layer.

[0110] In the embodiments, such as Figure 2 As shown, step S3 performs a two-pass block-based online normalized index calculation: During the first round of polling, the rolling maximum value and rolling sum are dynamically updated based on the data blocks of the streaming input; during the second round of polling, the scores are corrected using the global normalization coefficient and a weighted sum is completed. The key-value pair cache (KV Cache) is evenly divided into N blocks (N≥8). In the first pass, K is streamed, the MAC array calculates the QK^T score and writes it to the score cache port 0, and the rolling maximum value / rolling sum maximum value (Running Max / Sum) is updated in parallel; in the second pass, V is streamed, the resident score is output from the score cache port 1 and multiplied by V and accumulated, and the full-window attention calculation is completed by looping N times.

[0111] In some embodiments, step S3 employs an intra-layer multi-data-slice ping-pong pipeline scheduling method based on multi-head attention. During the attention calculation process, the main computing cluster uses a 6-bit cyclic counter to drive address generation and sequentially completes all context window calculations. The Softmax exponential function is approximated by a piecewise linear 8KLUT (Look-Up Table), and the summation reciprocal is approximated by a shift approximation instead of a hard divider.

[0112] In some implementations, for long context reasoning scenarios, steps S2 and S3 are executed in parallel using an intra-layer rolling pipeline. Specifically, this includes: dividing the key-value pair cache data matrix of the current Transformer layer into M data slices along the Token length dimension, where M≥2; controlling the second transmission path to stream the i-th data slice to the first on-chip cache, while simultaneously controlling the computing array to perform double-pass block attention computation on the (i-1)-th data slice already cached in the second on-chip cache; after both the pull operation on the first on-chip cache and the computation operation on the second on-chip cache are completed, mutually exclusive switching of their pipeline states is performed, and the pull, computation, and result writing of the next data slice are rolled over until all M data slices of the current layer are computed; where i is a positive integer and 2≤i≤M.

[0113] As an example, a multi-slice ping-pong pipelined approach within a layer is adopted. The state of a single layer is divided into M slices (M≥2), and a switching loop of "pull current slice → compute → write back → pull next slice" is executed between Bank A and Bank B. When NoC is congested, the MAC compute array is naturally suspended and the sent bus requests are completed normally, with zero deadlock.

[0114] Combined with the pipeline timing in the embodiments Figure 3As shown, taking a single-layer large language model containing 32 attention heads (32 head data) as an example, due to the limitation of the physical capacity of the on-chip cache, it is impossible to accommodate the entire model state at once. In this embodiment of the invention, time slicing is used to exchange for space occupation. The 32 attention heads are equally divided into 4 data slices (i.e., 4 slices: Slice 0, Slice 1, Slice 2, Slice 3) according to a preset dimension, so that each data slice precisely contains 8 attention heads (8 head data), thereby perfectly adapting to the physical capacity limit of the first on-chip cache (Bank A) and the second on-chip cache (Bank B).

[0115] like Figure 3 As shown, its specific temporal pipeline advancement and overlapping masking mechanism are as follows:

[0116] 1. Initial Fetch Phase (Phase 0): At the initial moment of system startup or switch to a new computing layer, the control second transmission path (M1 leased line) is started first, streaming the 0th data slice (Slice 0, containing 8 headers) of the current layer and writing it to and caching it in the first on-chip cache (Bank A). At this time, since there is no ready data on the chip, the MAC computing array is in a waiting suspended state.

[0117] 2. Overlapping Pipeline Phase (Phase i): After the data fetched in the previous round is successfully stored in the on-chip cache, the system enters the double-buffered ping-pong overlapping execution phase. At this time, on the one hand, the second transmission path (M1 dedicated line) is controlled to continue streaming the i-th data slice required for the next time step and streaming it into the second on-chip cache (Bank B) on the other side; on the other hand, the cross switches of the internal core computing domain are mutually exclusively activated, driving the MAC computing array to read the ready i-th data slice (containing 8 headers) from the first on-chip cache (Bank A) and sequentially execute the double-pass block attention pipeline computation.

[0118] 3. Mutual Exclusion Switching and Rolling Phase: The system hardware state machine triggers a mutual exclusion switching instruction only when the on-chip pull operation and the array computation operation of the i-th data slice are both completed. This instruction swaps the pipeline function states of the first on-chip cache (Bank A) and the second on-chip cache (Bank B) (i.e., the cache originally used for computation is switched to pull, and the cache originally used for pull is switched to computation). This allows the subsequent data slices to be executed in a rolling loop of "pull-computation-writeback" in groups of 8, until all 4 slices of the current layer have been fully computed.

[0119] 4. Resilient Backpressure Protection Mechanism: During the timing of any of the above-mentioned pipelined steps, if the system-level bus (NoC) experiences a sudden severe congestion, leading to deterioration in data write latency, the flow control scheduling module will immediately capture the water level signal of the asynchronous FIFO and issue a suspension backpressure command to the computing core domain. The MAC computing array will then naturally suspend and idle until the bus congestion is resolved and the next set of 8-head slices is fully ready in the opposite buffer before automatically resuming. This mechanism ensures that under severe congestion, the issued bus requests can be completed normally in a closed loop, resulting in zero deadlock in the entire chip system and minimizing the computing idle cycle.

[0120] In some embodiments where power consumption is not sensitive but high speed is required, if the aforementioned in-layer multi-data-slice rolling pipeline is not used, the full model state (or full key-value cache data matrix) of the current Transformer layer can be directly stored on-chip by significantly increasing the physical capacity of the first on-chip cache and the second on-chip cache; or, external static random access memory (SRAM) or dynamic random access memory (DRAM) can be used for multi-chip package (SiP) expansion.

[0121] In this embodiment, dynamic flow control coordination is achieved through elastic backpressure coordination: the elastic prefetch and backpressure state machine monitor the full / empty status signal of the internal asynchronous queue output by the cross-clock domain isolation layer in real time, thereby realizing real-time monitoring of the fullness of the internal asynchronous queue of the cross-clock domain isolation layer.

[0122] In some embodiments, when the backlog of the asynchronous queue inside the clock domain isolation layer reaches a preset threshold (e.g., the water level of the system asynchronous FIFO exceeds the upper limit threshold), or when a blocking condition is triggered due to a sudden bus congestion (e.g., the synchronized AXI bus READY signal is pulled low), the flow control scheduling module immediately sends a pause or wait signal to the first transmission path. When the backlog of the internal asynchronous queue reaches the preset threshold or a blocking condition is triggered, a pause or wait signal is sent to the first transmission path to suspend the current weighted data transfer.

[0123] In step S4, during the process of sending a pause or wait signal to the first transmission path to suspend the current weighted data transfer, the control mechanism includes, but is not limited to: in a preferred embodiment using descriptor-driven operation, the flow control scheduling module cuts off the transmission link of the first transmission path by intercepting, suspending, or delaying the issuance of subsequent data transfer descriptors; or, in a bare bus embodiment without descriptors, suspending bus transactions by directly pulling the bus's READY signal low. All of the above behaviors fall within the protection scope of this invention for suspending the current weighted data transfer.

[0124] Achieving collaboration between water level backpressure and routing pipeline: The Top-K routing decision results output by the gate control network (such as a gate control dedicated computing cluster) directly drive the DMA descriptor submission, and monitor the FIFO water level to dynamically trigger prefetching or send a pause (PAUSE) backpressure signal, thereby realizing overlapping pipelines of routing decision, weight transfer and computation execution.

[0125] This invention addresses the contradiction between hardware fixation and rapid model iteration. By employing a hybrid design of a "deterministic control skeleton + reconfigurable computing domain" using an eFPGA architecture, it achieves agile adaptation during model updates, significantly reducing the iteration cost and deployment threshold of edge AI hardware while maintaining the deterministic performance baseline of the core data flow. Through techniques such as dual-bus physical isolation, time-division hard-slicing scheduling, waterline backpressure state machine, hardware-based dual-pass chunked normalized exponential function (ChunkedSoftmax) computation, and DeltaNet slicing pipeline, it achieves deterministic low latency for KVCache reads, elastic high throughput for MoE weight transport, complete decoupling of NoC congestion, and seamless overlap between dynamic routing and the computing pipeline, all within a <50W edge power consumption constraint. This meets the efficient inference requirements of long-context MoE-LLM at the edge.

[0126] The hybrid expert LLM inference engine architecture and scheduling method provided in this application have been described in detail above. Specific examples have been used in this paper to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the concept of this application and should not be construed as a limitation on the scope of protection of this application.

Claims

1. A hybrid expert-based LLM inference engine architecture, characterized in that, include: The system has two transmission paths, including a first transmission path and a second transmission path, which are used to establish an independent hybrid expert weight transmission path and a key-value pair cache transmission path, respectively. A cross-clock domain isolation layer has its input terminals connected to the first and second transmission paths, respectively, and its output terminal connected to the internal core computing domain. It is used to synchronize the asynchronous data flow between the system bus clock domain and the core computing clock domain. The core computing domain includes a crossbar switch and a computing array connected thereto. The input terminals of the crossbar switch are coupled to the first transmission path and the second transmission path respectively via the cross-clock domain isolation layer, for selectively directing data streams to the computing array. A phase scheduling controller, whose control output terminal is connected to the first transmission path, the second transmission path and the cross switch respectively, is used to control the first transmission path and the second transmission path to pass through the cross switch mutually exclusively in phase A and phase B by outputting a phase switching control signal; The flow control scheduling module has its input end connected to the cross-clock domain isolation layer to obtain the full / empty status signal of the internal asynchronous queue, and its output end connected to the request configuration end of the first transmission path. It is used to realize elastic backpressure closed-loop control of bus requests based on the data buffer margin of the asynchronous queue.

2. The hybrid expert-based LLM inference engine architecture according to claim 1, characterized in that, The first transmission path embeds a spread-aggregate DMA controller for asynchronously transferring hybrid expert weight data from system memory; the second transmission path includes a dedicated address generator, which consists only of a base address register and an offset accumulator, for performing flat physical address mapping of private memory.

3. The architecture according to claim 1, characterized in that, The flow control scheduling module is a flexible prefetch and reverse pressure state machine, and the cross-clock domain isolation layer includes: System asynchronous FIFO: located between the first transmission path and the cross switch; Private asynchronous FIFO: located between the second transmission path and the cross switch; Cross-domain processing logic: used to perform clock synchronization processing on the phase switching control signal output by the phase scheduling controller; The full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer is specifically the real-time water level signal of the system's asynchronous FIFO; the elastic prefetch and backpressure state machine compares the real-time water level signal with the preset cascade water level threshold to perform dynamic backpressure.

4. The hybrid expert-based LLM inference engine architecture according to claim 1, characterized in that, The architecture is deployed in an embedded programmable logic array domain; the core computing domain is constructed using the reconfigurable logic resources of the embedded programmable logic array.

5. The hybrid expert-based LLM inference engine architecture according to claim 1, characterized in that, The computing array includes a main computing cluster and a gate control dedicated computing cluster. The architecture also includes a partial sorting tree unit. The gate control dedicated computing cluster is used to calculate and output the gate control routing decision result. The output of the gate control dedicated computing cluster is connected to the partial sorting tree unit. The partial sorting tree unit is used to generate an expert activation index based on the gate control routing decision result and configure the expert activation index to the first transmission path to drive the corresponding bus descriptor to submit. The core computing domain also includes a first on-chip cache and a second on-chip cache that are independent of each other. The input of the main computing cluster is coupled to the first on-chip cache and the second on-chip cache via the cross switch. When the first transmission path or the second transmission path performs a streaming pull operation across the cross-clock domain isolation layer, it alternately reads ready data slices and corresponding activated expert weights from the first on-chip cache or the second on-chip cache. It also controls the main computing cluster to dynamically execute the double-pass block attention pipeline computation dominated by the data slices and the expert network matrix multiplication computation dominated by the activated expert weights in an intra-layer timing overlap manner. The flow control scheduling module performs elastic backpressure control to pause or resume the first transmission path that has submitted a bus descriptor based on the full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer, thereby jointly realizing closed-loop control of bus requests.

6. The hybrid expert-based LLM inference engine architecture according to claim 1, characterized in that, The architecture also includes a hardware-level streaming decompression unit, which is embedded in the data path of the first or second transmission path and is used to perform real-time decoding of compressed data during the transportation process.

7. The hybrid expert-based LLM inference engine architecture according to claim 5, characterized in that: When the main computing cluster performs the double-pass block attention pipeline computation, it uses a 6-bit loop counter to drive address generation and uses a lookup table to perform piecewise linear approximation of the Softmax exponential function.

8. A hybrid expert-based LLM inference scheduling method, characterized in that, The scheduling method is applied to the architecture described in any one of claims 1-7, and the method includes the following steps: Step S1. Phase hard-cut scheduling: The phase scheduling controller receives the layer counting signal and divides the single-layer decoding cycle into phase A and phase B through a hard-cut cross switch; Step S2. Data Mutual Exclusion Transmission: During phase A, enable the first transmission path to transfer the mixed expert weights and suspend the second transmission path; during phase B, enable the second transmission path to read the key-value pair cache and suspend the first transmission path. Step S3. Double-pass block attention calculation: The main computing cluster is controlled to alternately read ready data slices and corresponding activated expert weights from the first on-chip cache or the second on-chip cache in a streaming manner. In an intra-layer temporal overlapping manner, the key-value pair cached data in the data slice is divided into N blocks and the first pass of block score calculation and the second pass of weighted accumulation calculation are executed in sequence. At the same time, the expert network matrix multiplication calculation dominated by the activated expert weights is executed in a coordinated manner to complete the double-pass block attention pipeline calculation. Step S4. Dynamic flow control coordination: The flow control scheduling module monitors the full / empty status signal of the asynchronous queue inside the cross-clock domain isolation layer in real time. When the backlog of the internal asynchronous queue reaches a preset threshold or triggers a blocking condition, a pause or wait signal is sent to the first transmission path to suspend the current weighted data transfer.

9. The method according to claim 8, characterized in that: In step S2, the hybrid expert weight data transported by the first transmission path and the key-value pair cache data read by the second transmission path complete data alignment and buffering in their respective independent asynchronous clock domains when passing through the cross-clock domain isolation layer.

10. The method according to claim 8, characterized in that: The core computing domain also includes independent first and second on-chip caches. For long context inference scenarios, steps S2 and S3 are executed in parallel using an intra-layer rolling pipeline approach, specifically including: Divide the key-value pair cache data matrix of the current Transformer layer into M data slices along the Token length dimension, where M≥2; The second transmission path is controlled to stream the i-th data slice to the first on-chip cache, and the computing array is controlled to perform the double-pass block attention calculation on the (i-1)-th data slice already cached in the second on-chip cache. After the fetch operation on the first chip cache and the calculation operation on the second chip cache are completed, the pipeline states of the two are switched mutually exclusively, and the fetch, calculation and result writing of the next data slice are executed in a rolling manner until all M data slices of the current layer are calculated; where i is a positive integer and 2≤i≤M.

Images