A zero-cycle data flow context switch circuit

Abstract

This invention discloses a dual-granularity reconfigurable spatial array for large model inference, a zero-cycle dataflow context switching circuit, a large model inference hardware accelerator, and a large model inference method based on the dual-granularity reconfigurable spatial array. The dual-granularity reconfigurable spatial array for large model inference includes several sub-arrays, a coarse-grained configuration table, and a fine-grained configuration table. The spatial array consists of N sub-arrays, each of which is an M×M processing unit (PE) grid. The coarse-grained configuration table is used to implement coarse-grained reconstruction, dividing and configuring the interconnection relationships between sub-arrays according to the operator kernel. The fine-grained configuration table is used to implement fine-grained reconstruction, dividing and configuring the interconnection relationships between PEs and the register file port mapping relationships within a single sub-array according to the dataflow. The coarse-grained and fine-grained configuration tables are independent of each other and support independent writing and switching.

Description

Technical Field

[0001] This invention relates to a hardware accelerator architecture for Large Language Model (LLM) inference, specifically to a dual-granularity reconfigurable spatial array for large model inference, a zero-cycle data flow context switching circuit, a large model inference hardware accelerator, and a large model inference method based on a dual-granularity reconfigurable spatial array. Background Technology

[0002] LLM inference can be divided into two phases: prefill (first-token throughput) and decode (token-by-token latency). In the decode phase, a single token takes about 1,000 clock cycles to complete all 80 layers of the model (taking LLaMA-70B as an example). The hardware overhead of dataflow switching between layers directly determines whether the reconfigurable array can realize its energy efficiency advantages.

[0003] Dense Transformers employ a single attention layer that favors OH-OW parallelism (Q / K tiles are expanded along the PE rows and columns), followed by FFN that favors MN-IC-OC parallelism (weight reuse). MoE models activate different experts per token, requiring multiple subarrays to concurrently execute different experts within the same cycle. SSM degenerates into MN streaming due to state updates along token order. Hybrid Transformer-Mamba allows two types of dataflows to coexist in the same forward pass. Existing reconfigurable arrays either bind a single dataflow at compile time, sacrificing the utilization of non-preferred layers, or incur switching overhead on the order of tens of cycles between layers.

[0004] In the decoding stage, which is extremely sensitive to latency, reducing the overhead of a single dataflow switch to single-digit cycles is a prerequisite for realizing the value of "layer-by-layer optimal dataflow" and is also the starting point of this invention.

[0005] Prior art related to this invention:

[0006] One of the existing technologies related to this invention is LEGO (HPCA'25), whose technical solution is as follows: multiple spatial dataflows (such as Conv IC-OC, OH-OW, GEMM MN-IC-OC) are characterized in the array with affine relations, and direct edges are merged by minimum spanning forest (MST) + breadth-first search (BFS) to generate an RTL that can switch between multiple mappings from the same FU graph;

[0007] If a dataflow needs to be switched at runtime, the interconnect routes within the array need to be reassembled according to the target dataflow, and the routing table in the PE inter-register file needs to be rewritten. According to the literature, the overhead of a single switch is between tens and hundreds of clock cycles in a 256×256 array.

[0008] Source: T. Tambe et al., "LEGO: Spatial Accelerator Generation and Optimization for Tensor Applications," HPCA 2025.

[0009] Disadvantages of existing technology 1:

[0010] 1. Switching LEGO requires draining the pipeline and then reinstalling the interconnection, with a single switch costing tens of cycles. The LLaMA-70B decoder has only about a thousand cycles of end-to-end budget for a single token. If switching is done layer by layer across 80 layers, the cumulative cost can reach thousands of cycles, which is slower than a single dataflow binding.

[0011] 2. LEGO only supports one granularity of reconstruction within the array (the entire array level). It cannot concurrently run different operators between sub-arrays according to the kernel, nor can it independently switch between PE levels within a sub-array. It is not suitable for concurrent scenarios with multiple experts in the same period in MoE.

[0012] Prior art related to this invention:

[0013] The second prior art related to this invention is the Intel Gaudi MME, whose technical solution is as follows: The MME is a systolic array that supports three levels of systolic geometry switching at the batch boundary—256×256 (suitable for square arrays GEMM), 512×256, and 1024×128 (suitable for M... N's slender GEMM increases GEMV utilization by approximately 15%);

[0014] Runtime switching is initiated by the top-level scheduler at batch boundaries and is achieved by reconfiguring the physical mapping of pulsating rows and columns; geometric gears only cover GEMM shapes and do not involve switching of dataflow dimensions;

[0015] Source: Intel Habana Gaudi-2 White Paper, 2023.

[0016] Disadvantages of existing technology 2:

[0017] 1. Gaudi MME switching can only occur at batch boundaries and cannot respond to the layer-by-layer dataflow differences between attention→FFN→state-update within the same forward process of LLM; each token in the decode phase is a batch, so there is no "batch boundary" at all.

[0018] 2. The three-level geometry is fixed, covering only GEMM shapes, and has no reconfigurable semantics for the two non-GEMM shapes: SSM flow cytometry and MoE multi-expert.

[0019] Prior art related to this invention:

[0020] The third prior art related to this invention is FexMo (MICRO'25), whose technical solution is as follows: introduce fuse execution mode (FEM) on CGRA, and allow multiple data flow graphs (DFGs) to share the same array through modulo cycle-level time division multiplexing, which improves the average utilization rate by about 36.9% according to literature tests;

[0021] Runtime switching occurs at kernel boundaries, with the top-level scheduler issuing instructions for different DFGs based on modulo cycle counts; no further splitting occurs within the same DFG.

[0022] Source: Y. Zhang et al., "FexMo: Flexible Execution Mode for CGRA-basedAccelerators," MICRO 2025.

[0023] The disadvantages of existing technology three:

[0024] 1. FexMo's time-division multiplexing granularity is limited to the kernel level, and the same DFG cannot be split; an LLM layer is often a DFG, and it is impossible to make different dataflows such as attention and SSM emit simultaneously within the layer.

[0025] 2. FexMo still does not directly handle the hardware overhead of dataflow switching, but only avoids the problem by using kernel-level time-division multiplexing; it is not suitable for layer-level or finer-grained processing.

[0026] Prior art related to this invention:

[0027] The fourth prior art related to this invention is Neura (ASPLOS'26), whose technical solution is as follows: the array is composed of several sub-CGRA spliced ​​together, and the sub-CGRA occupancy range is dynamically expanded or shrunk according to kernel migration during operation to adapt to the computing power requirements of different kernels;

[0028] Migration is triggered by the host and is performed on an entire sub-CGRA; it does not involve the reconstruction of PE-level interconnects within the sub-CGRA.

[0029] Source: H. Liu et al., "Neura: Hierarchical CGRA with Dynamic KernelMigration," ASPLOS 2026.

[0030] The disadvantages of existing technology four:

[0031] 1. Neura's smallest migration unit is the entire sub-CGRA; fine-grained refactoring at the PE level is not within its design goals; it does not support multiple experts running concurrently within the same sub-CGRA.

[0032] 2. Neura does not distinguish between dataflow types and only performs kernel migration based on computing power quotas; it does not meet the requirements of LLM attention / FFN to favor different dataflows. Summary of the Invention

[0033] This invention provides a dual-granularity reconfigurable spatial array for large model inference, a zero-cycle data flow context switching circuit, a large model inference hardware accelerator, and a large model inference method based on a dual-granularity reconfigurable spatial array, in order to solve the technical problems existing in the prior art.

[0034] To achieve the above objectives, the present invention provides a dual-granularity reconfigurable spatial array for large model inference, which includes several sub-arrays, a coarse-grained configuration table, and a fine-grained configuration table.

[0035] The spatial array consists of N sub-arrays, each of which is an M×M processing unit PE grid;

[0036] The coarse-grained configuration table is used to implement coarse-grained reconstruction, dividing and configuring the interconnection relationship between each sub-array according to the operator kernel;

[0037] The fine-grained configuration table is used to implement fine-grained reconstruction, dividing and configuring the interconnection relationship between PEs and the register file port mapping relationship within a single sub-array according to the dataflow.

[0038] The coarse-grained configuration table and the fine-grained configuration table are independent of each other and can be written and switched independently.

[0039] In one embodiment of the present invention, optionally, each sub-array integrates a data flow context switching (DCP) circuit;

[0040] The DCP circuit includes K sets of shadow registers, a context pointer register ctx_ptr, and commit logic.

[0041] Each set of shadow registers stores a complete set of configuration information, including interconnection routing table R, register file port mapping P, and accumulated path pointer A.

[0042] In one embodiment of the present invention, optionally, the ctx_ptr is a log2K bit register used to point to the currently effective shadow register group;

[0043] When switching data streams, simply modifying the pointer of ctx_ptr is sufficient to complete the configuration switch;

[0044] In-flight operations that have entered the pipeline carry the original mode label and are executed according to the R, P, A configuration before the switch. Newly launched operations are executed in the next clock cycle according to the new configuration pointed to by ctx_ptr. There is no need to drain the pipeline. The total clock cycle for a single data stream switch is no more than 5.

[0045] In one embodiment of the present invention, optionally, each sub-array is further configured with an in-layer modulus-taking cycle time-division multiplexing circuit;

[0046] The time-division multiplexing circuit includes a modulo counter, which is configured with a fixed cycle period.

[0047] The modulo counter selects different groups of shadow registers to take effect alternately based on the count value, so that multiple different data streams in the same layer network can be executed concurrently under the premise of sharing the PE mesh.

[0048] In one embodiment of the present invention, the invention may optionally include a MoE scheduling unit (MDU) arranged upstream of the entire space array.

[0049] The MDU receives the expert IDs (expert_ids) and token tags (token_tag) output by the large model programming model, and distributes the tokens to the corresponding sub-arrays using a heat-aware scheduling strategy.

[0050] Subarrays assigned to active experts operate normally, while idle subarrays that are not selected enter the clock-gated state.

[0051] In one embodiment of the present invention, optionally, the heat-aware scheduling strategy includes:

[0052] The activation popularity of each expert is calculated, and experts with high activation popularity are fixedly assigned to high-bandwidth sub-arrays, while experts with low activation popularity are assigned to ordinary sub-arrays.

[0053] The popularity of experts is updated in real time through a sliding window, and the binding relationship between experts and sub-arrays is dynamically adjusted.

[0054] The present invention also provides a zero-cycle data stream context switching circuit, applied inside the sub-array of the above-mentioned dual-granularity reconfigurable spatial array, including multiple sets of shadow registers, context pointer registers and commit logic units;

[0055] The shadow register group pre-loads configuration information, and each set of configuration information corresponds to a data stream format.

[0056] The context pointer register is used to select the currently enabled shadow register group;

[0057] The submission logic unit is used to isolate in-transit operations from new operations, ensuring that in-transit operations are executed using the old configuration and new operations are executed using the new configuration, without needing to drain the pipeline during the switching process.

[0058] The present invention also provides a large model inference hardware accelerator, which includes the above-mentioned dual-granularity reconfigurable spatial array and the above-mentioned zero-cycle data stream context switching circuit.

[0059] The hardware accelerator is adapted to the prefill and decode stages of large models and supports layer-by-layer switching and intra-layer concurrency of attention, feedforward network (FFN), state sequence model (SSM), hybrid expert model (MoE), and hybrid Transformer-Mamba heterogeneous data stream.

[0060] In one embodiment of the present invention, optionally, for the decoding stage of the LLaMA-type model, the data stream is switched by switching the context pointer at the boundary of each network layer, with a single switching clock cycle not exceeding 5, and the cumulative switching clock cycle of 80 layers not exceeding 320.

[0061] This invention also provides a large model inference method based on a dual-granularity reconfigurable spatial array, which uses the aforementioned hardware accelerator to perform inference, and includes the following steps:

[0062] S1: Based on the network layer type of the large model, the interconnection configuration between sub-arrays is completed through a coarse-grained configuration table, and the PE interconnection and register port mapping configuration within the sub-arrays are completed through a fine-grained configuration table;

[0063] S2: Preload the interconnection routing table, register port mapping, and accumulated path pointer corresponding to different data streams into multiple sets of shadow registers in each sub-array;

[0064] S3: When switching data streams between network layers, only the context pointer register is updated, and the pipeline is seamlessly connected by the commit logic unit to complete the fast switching of data streams.

[0065] S4: For scenarios with multiple heterogeneous data streams within a single network layer, enable the modulus counter and switch the shadow register group in a polling manner according to the modulus result to achieve time-sharing concurrency of multiple data streams within the layer;

[0066] S5: For MoE model inference, tokens are distributed based on expert popularity through MDU, and idle sub-arrays enter clock-gated state;

[0067] S6: Repeat steps S3~S5 to complete the full forward inference of the large model.

[0068] The present invention has the following beneficial technical effects;

[0069] (1) Make layer-by-layer dataflow heterogeneity feasible in the decoding stage;

[0070] (2) The cost of a single switch is ≤5 cycles, which is an order of magnitude lower than the tens of cycles of LEGO;

[0071] (3) Multiple experts in MoE operate concurrently at the sub-array level within the same period;

[0072] (4) Hybrid models can coexist in the same array, saving 40% of the area compared to the two-array scheme. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This provides an overview of dual-granularity reconfigurable spatial arrays (coarse-grained reconstruction by kernel / fine-grained reconstruction by dataflow).

[0075] Figure 2 The DCP (Dataflow Context Pointer) circuit structure is located within a single sub-array (K sets of shadow registers + ctx_ptr + commit logic).

[0076] Figure 3 shows the DCP zero-cycle switching timing (this invention) vs. the existing LEGO drain pipeline timing.

[0077] Figure 4 shows Layer-Internal Modulo-Cycle time-division multiplexing (modulo cycle 2, attn-OH / OW and SSM-MN flow cytometry concurrent within the same layer);

[0078] Figure 5 shows the heat-aware concurrent scheduling of the MoE Dispatch Unit (MDU) (Mixtral-8x7B, N=8 sub-array, top-2 activation).

[0079] Figure 6 shows the LLaMA-70B decode single token 80-layer layer-by-layer dataflow switching process (DCP flips the pointer 80 times, accumulating 320 cycles). Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] In response to all the shortcomings of existing technologies such as LEGO / Gaudi MME / FexMo / Neura, the technical problems to be solved by this invention include the following four aspects:

[0082] (1) Layer-by-layer dataflow heterogeneity cannot be realized: In the same forward feed of LLM, attention is biased towards OH-OW, FFN is biased towards MN-IC-OC, SSM is biased towards MN streaming, and MoE requires sub-array level concurrency. If a single array is bound to a single dataflow at compile time, the utilization of non-biased layers will decrease significantly.

[0083] (2) Switching overhead is unacceptable during the decoding phase: existing arrays switch between layers for dozens of cycles, with 80 layers cumulatively swallowing up the benefits;

[0084] (3) Insufficient concurrent granularity of MoE: Each token top-k expert needs to be executed simultaneously on multiple sub-arrays, while the existing architecture can only perform serial execution at the kernel level;

[0085] (4) Difficulty of coexistence of Hybrid Transformer-Mamba single array: attention and SSM in the same layer need to have their own dataflow and share PE computing power, which cannot be done by a single array. The area of ​​the two array scheme is doubled.

[0086] This invention relates to a spatial array hardware structure and a data flow context switching circuit that support runtime multi-dataflow switching. In recent years, this field has shown two main trends: one is that dataflow architecture has achieved significant energy efficiency advantages over GPU SIMT by explicitly laying out operator graphs and data paths on the spatial array; the other is that LLM workloads contain various heterogeneous forms such as attention, FFN, SSM, and MoE in a single forward pass, and each form has a different preference for the optimal dataflow.

[0087] First embodiment:

[0088] This invention provides a dual-granularity reconfigurable spatial array for large model inference, which includes several sub-arrays, a coarse-grained configuration table, and a fine-grained configuration table.

[0089] The spatial array consists of N sub-arrays, each of which is an M×M processing unit PE grid;

[0090] The coarse-grained configuration table is used to implement coarse-grained reconstruction, dividing and configuring the interconnection relationship between each sub-array according to the operator kernel;

[0091] The fine-grained configuration table is used to implement fine-grained reconstruction, dividing and configuring the interconnection relationship between PEs and the register file port mapping relationship within a single sub-array according to the dataflow.

[0092] The coarse-grained configuration table and the fine-grained configuration table are independent of each other and can be written and switched independently.

[0093] In one embodiment of the present invention, optionally, each sub-array integrates a data flow context switching (DCP) circuit;

[0094] The DCP circuit includes K sets of shadow registers, a context pointer register ctx_ptr, and commit logic.

[0095] Each set of shadow registers stores a complete set of configuration information, including interconnection routing table R, register file port mapping P, and accumulated path pointer A.

[0096] In one embodiment of the present invention, optionally, the ctx_ptr is a log2K bit register used to point to the currently effective shadow register group;

[0097] When switching data streams, simply modifying the pointer of ctx_ptr is sufficient to complete the configuration switch;

[0098] In-flight operations that have entered the pipeline carry the original mode label and are executed according to the R, P, A configuration before the switch. Newly launched operations are executed in the next clock cycle according to the new configuration pointed to by ctx_ptr. There is no need to drain the pipeline. The total clock cycle for a single data stream switch is no more than 5.

[0099] In one embodiment of the present invention, optionally, each sub-array is further configured with an in-layer modulus-taking cycle time-division multiplexing circuit;

[0100] The time-division multiplexing circuit includes a modulo counter, which is configured with a fixed cycle period.

[0101] The modulo counter selects different groups of shadow registers to take effect alternately based on the count value, so that multiple different data streams in the same layer network can be executed concurrently under the premise of sharing the PE mesh.

[0102] In one embodiment of the present invention, the invention may optionally include a MoE scheduling unit (MDU) arranged upstream of the entire space array.

[0103] The MDU receives the expert IDs (expert_ids) and token tags (token_tag) output by the large model programming model, and distributes the tokens to the corresponding sub-arrays using a heat-aware scheduling strategy.

[0104] Subarrays assigned to active experts operate normally, while idle subarrays that are not selected enter the clock-gated state.

[0105] In one embodiment of the present invention, optionally, the heat-aware scheduling strategy includes:

[0106] The activation popularity of each expert is calculated, and experts with high activation popularity are fixedly assigned to high-bandwidth sub-arrays, while experts with low activation popularity are assigned to ordinary sub-arrays.

[0107] The popularity of experts is updated in real time through a sliding window, and the binding relationship between experts and sub-arrays is dynamically adjusted.

[0108] The present invention also provides a zero-cycle data stream context switching circuit, applied inside the sub-array of the above-mentioned dual-granularity reconfigurable spatial array, including multiple sets of shadow registers, context pointer registers and commit logic units;

[0109] The shadow register group pre-loads configuration information, and each set of configuration information corresponds to a data stream format.

[0110] The context pointer register is used to select the currently enabled shadow register group;

[0111] The submission logic unit is used to isolate in-transit operations from new operations, ensuring that in-transit operations are executed using the old configuration and new operations are executed using the new configuration, without needing to drain the pipeline during the switching process.

[0112] The present invention also provides a large model inference hardware accelerator, which includes the above-mentioned dual-granularity reconfigurable spatial array and the above-mentioned zero-cycle data stream context switching circuit.

[0113] The hardware accelerator is adapted to the prefill and decode stages of large models and supports layer-by-layer switching and intra-layer concurrency of attention, feedforward network (FFN), state sequence model (SSM), hybrid expert model (MoE), and hybrid Transformer-Mamba heterogeneous data stream.

[0114] In one embodiment of the present invention, optionally, for the decoding stage of the LLaMA-type model, the data stream is switched by switching the context pointer at the boundary of each network layer, with a single switching clock cycle not exceeding 5, and the cumulative switching clock cycle of 80 layers not exceeding 320.

[0115] This invention also provides a large model inference method based on a dual-granularity reconfigurable spatial array, which uses the aforementioned hardware accelerator to perform inference, and includes the following steps:

[0116] S1: Based on the network layer type of the large model, the interconnection configuration between sub-arrays is completed through a coarse-grained configuration table, and the PE interconnection and register port mapping configuration within the sub-arrays are completed through a fine-grained configuration table;

[0117] S2: Preload the interconnection routing table, register port mapping, and accumulated path pointer corresponding to different data streams into multiple sets of shadow registers in each sub-array;

[0118] S3: When switching data streams between network layers, only the context pointer register is updated, and the pipeline is seamlessly connected by the commit logic unit to complete the fast switching of data streams.

[0119] S4: For scenarios with multiple heterogeneous data streams within a single network layer, enable the modulus counter and switch the shadow register group in a polling manner according to the modulus result to achieve time-sharing concurrency of multiple data streams within the layer;

[0120] S5: For MoE model inference, tokens are distributed based on expert popularity through MDU, and idle sub-arrays enter clock-gated state;

[0121] S6: Repeat steps S3~S5 to complete the full forward inference of the large model.

[0122] Second embodiment:

[0123] This invention provides a dual-granularity (sub-array level + PE level) reconfigurable spatial array and a zero-cycle dataflow context switching circuit. The specific technical solution consists of four elements:

[0124] Element 1 (Dual-Granularity Reconfiguration Architecture, corresponding to Figure 1): The array consists of N sub-arrays (N=4 or 8 in this example), each sub-array being an M×M PE grid (M=64 in this example). The coarse-grained configuration table (CoarseConfig Table) divides the interconnections between sub-arrays according to the kernel, while the fine-grained configuration table (FineConfig Table) divides the PE-to-PE interconnections and register file port mappings within sub-arrays according to the dataflow. The two configuration tables are written to and switched independently, without dependency on each other.

[0125] Element 2 (DCP data flow context switching circuit, corresponding to Figures 2 and 3): Each sub-array maintains K groups (K=4 in the example) of dataflow configuration shadow registers. Each group contains three types of fields: interconnection routing table R, register file port mapping P, and accumulated path pointer A, and is equipped with a log2K bit context pointer register (ctx_ptr). During switching, only ctx_ptr is flipped. The already transmitted in-flight operation is completed according to the old R / P / A based on the old mode tag it carries. The new transmission operation is transmitted according to the new R / P / A in the next cycle. No drain pipeline is required. The measured single switching overhead does not exceed 5 clock cycles.

[0126] Element 3 (Layer-Internal Modulo-Cycle Time Division Multiplexing Circuit, corresponding to Figure 4): Add a modulo counter to each sub-array controller (cycle 2 or 4 in the embodiment), and select two or more groups from K groups of shadow registers to alternately transmit in the same layer according to the modulo result, so that two or more dataflows can be executed concurrently in the same layer; a typical scenario is the coexistence of attention-SSM in the Hybrid Transformer-Mamba segment.

[0127] Element 4 (MoE Dispatch Unit, corresponding to Figure 5): An MDU is set above the array to receive expert_ids and token_tag produced by the moe_gate primitive of the P1 programming model. The token is routed to the corresponding sub-array according to the heat awareness strategy. Hot experts are always assigned to sub-arrays with higher bandwidth, and unselected sub-arrays enter the clock-gated state.

[0128] Figure 6 shows the layer-by-layer DCP switching process for a single token of 80 layers in LLaMA-70B decoding: even-numbered layers of attention follow OH-OW, odd-numbered layers of FFN follow MN-IC-OC, and the pointer flips ctx_ptr at the boundary of each layer for 4 cycles, totaling 320 cycles, which accounts for about 27% of the 1.18k cycles of the decoding end-to-end process.

[0129] Typical quantization results of this invention: LLaMA-70B decode weighted utilization increased from 52% to 74% (+22 percentage points), and end-to-end decode cycle time decreased from 1.45k cycles to 1.18k cycles; Mixtral-8x7B decode throughput increased by 2.3× compared to kernel-level serial experts, and static power consumption of the clock-gated 6 sub-arrays decreased by 56%; the Jamba single-array solution reduced area by 40% and Transformer segment latency by 8% compared to the two-array solution; in concurrent scenarios within the MambaFormer segment, latency within the same layer decreased by 35%, and PE utilization increased from 49% to 78%.

[0130] This invention has good scalability across three axes: number of sub-arrays N, PE side length M, and number of shadow register groups K: N=2~16 adapts to different chip areas, M=32~128 adapts to different peak computing power, and K=2~8 adapts to different model families (K=2 for Hybrid dual dataflow, K=4 for LLaMA+ speculative verification, and K=8 for multimodal inference).

[0131] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0132] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-granularity reconfigurable spatial array for large model inference, characterized in that, It includes several sub-arrays, a coarse-grained configuration table, and a fine-grained configuration table; The spatial array consists of N sub-arrays, each of which is an M×M processing unit PE grid; The coarse-grained configuration table is used to implement coarse-grained reconstruction, dividing and configuring the interconnection relationship between each sub-array according to the operator kernel; The fine-grained configuration table is used to implement fine-grained reconstruction, dividing and configuring the interconnection relationship between PEs and the register file port mapping relationship within a single sub-array according to the dataflow. The coarse-grained configuration table and the fine-grained configuration table are independent of each other and can be written and switched independently.

2. The dual-granularity reconfigurable spatial array according to claim 1, characterized in that, Each sub-array integrates a data flow context switching (DCP) circuitry. The DCP circuit includes K sets of shadow registers, a context pointer register ctx_ptr, and commit logic. Each set of shadow registers stores a complete set of configuration information, including interconnection routing table R, register file port mapping P, and accumulated path pointer A.

3. The dual-granularity reconfigurable spatial array according to claim 2, characterized in that, The ctx_ptr is a log2K bit register used to point to the currently active shadow register group; When switching data streams, simply modifying the pointer of ctx_ptr is sufficient to complete the configuration switch; In-flight operations that have entered the pipeline carry the original mode label and are executed according to the R, P, A configuration before the switch. Newly launched operations are executed in the next clock cycle according to the new configuration pointed to by ctx_ptr. There is no need to drain the pipeline. The total clock cycle for a single data stream switch is no more than 5.

4. The dual-granularity reconfigurable spatial array according to claim 2, characterized in that, Each sub-array is also configured with an in-layer modulus-taking cycle time-division multiplexing circuit; The time-division multiplexing circuit includes a modulo counter, which is configured with a fixed cycle period. The modulo counter selects different groups of shadow registers to take effect alternately based on the count value, so that multiple different data streams in the same layer network can be executed concurrently under the premise of sharing the PE mesh.

5. The dual-granularity reconfigurable spatial array according to claim 1, characterized in that, It also includes a MoE scheduling unit (MDU), which is located upstream of the entire space array; The MDU receives the expert IDs (expert_ids) and token tags (token_tag) output by the large model programming model, and distributes the tokens to the corresponding sub-arrays using a heat-aware scheduling strategy. Subarrays assigned to active experts operate normally, while idle subarrays that are not selected enter the clock-gated state.

6. The dual-granularity reconfigurable spatial array according to claim 5, characterized in that, The heat-sensing scheduling strategy includes: The activation popularity of each expert is calculated, and experts with high activation popularity are fixedly assigned to high-bandwidth sub-arrays, while experts with low activation popularity are assigned to ordinary sub-arrays. The popularity of experts is updated in real time through a sliding window, and the binding relationship between experts and sub-arrays is dynamically adjusted.

7. A zero-cycle data stream context switching circuit, applied within the sub-array of the dual-granularity reconfigurable spatial array according to any one of claims 1 to 6, characterized in that, This includes multiple sets of shadow registers, context pointer registers, and commit logic units; The shadow register group pre-loads configuration information, and each set of configuration information corresponds to a data stream format. The context pointer register is used to select the currently enabled shadow register group; The submission logic unit is used to isolate in-transit operations from new operations, ensuring that in-transit operations are executed with the old configuration and new operations are executed with the new configuration, without needing to drain the pipeline during the switching process.

8. A large-model inference hardware accelerator, characterized in that, Includes the dual-granularity reconfigurable spatial array as described in any one of claims 1 to 6 and the zero-cycle data stream context switching circuit as described in claim 7; The hardware accelerator is adapted to the prefill and decode stages of large models and supports layer-by-layer switching and intra-layer concurrency of attention, feedforward network (FFN), state sequence model (SSM), hybrid expert model (MoE), and hybrid Transformer-Mamba heterogeneous data stream.

9. The large model inference hardware accelerator according to claim 8, characterized in that, For the decoding stage of LLaMA-type models, data flow switching is completed by switching context pointers at the boundary of each network layer. The clock cycle of a single switch is no more than 5, and the cumulative switching clock cycle of 80 layers does not exceed 320.

10. A large-model inference method based on a dual-granularity reconfigurable spatial array, characterized in that, Performing inference using the hardware accelerator of claim 8 includes the following steps: S1: Based on the network layer type of the large model, the interconnection configuration between sub-arrays is completed through a coarse-grained configuration table, and the PE interconnection and register port mapping configuration within the sub-arrays are completed through a fine-grained configuration table; S2: Preload the interconnection routing table, register port mapping, and accumulated path pointer corresponding to different data streams into multiple sets of shadow registers in each sub-array; S3: When switching data streams between network layers, only the context pointer register is updated, and the pipeline is seamlessly connected by the commit logic unit to complete the fast switching of data streams. S4: For scenarios with multiple heterogeneous data streams within a single network layer, enable the modulus counter and switch the shadow register group in a polling manner according to the modulus result to achieve time-sharing concurrency of multiple data streams within the layer; S5: For MoE model inference, tokens are distributed based on expert popularity through MDU, and idle sub-arrays enter clock-gated state; S6: Repeat steps S3~S5 to complete the full forward inference of the large model.

Images