Three-dimensional hybrid bonding and reconfigurable data flow large model inference acceleration architecture and method

Abstract

The application discloses a three-dimensional mixed bonding and reconfigurable data flow large model reasoning acceleration architecture and method, the architecture adopts Logic-on-Bottom inverted three-layer stacking: the top layer is a DRAM near storage plane, the middle layer is a reconfigurable TGU array, and the bottom layer is a RISC-V control cluster; the three layers are vertically interconnected by means of WoW wafer mixed bonding, and the bandwidth between layers is more than 16 TB / s. The TGU can dynamically switch between Prefill pulsatile array and Decode broadcast reduction two types of data flow; the RISC-V completes hardware reconfiguration scheduling through a self-defined Xtgu instruction set, and undertakes irregular operator calculation such as RoPE. The application solves the problem of large model super-long context reasoning storage wall by relying on three-dimensional stacked high-density interconnection, improves the calculation efficiency in stages by hardware adaptation, optimizes heat dissipation and reduces transmission power consumption by inverted stacking; the architecture adapts to mature packaging and process of domestic products, and gives consideration to reasoning performance, energy efficiency and algorithm iteration flexibility.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit design and artificial intelligence computing technology, specifically involving a three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture and method. Background Technology

[0002] With the continuous growth of parameters in large language models such as GPT-4, DeepSeek, and Qwen, the business demand for context windows has reached the 1M Tokens level. Existing general-purpose GPUs and dedicated inference chips have four core problems: 1. Memory wall bottleneck: Inference KV cache expands linearly with the context length. Traditional GPUs rely on HBM and off-chip buses, and their bandwidth and capacity cannot support millions of contexts. Frequent data transfer between off-chip DRAM and computing units causes huge power consumption.

[0003] 2. Imbalance in computational efficiency across stages: Transformer inference is divided into two stages: Prefill and Decode. Prefill involves matrix-intensive computation, while Decode involves memory access-intensive, single-batch workloads. The unified hardware architecture of GPUs cannot simultaneously adapt to these two types of heterogeneous workloads, resulting in low utilization of computational units.

[0004] 3. Limited advanced processes and packaging: Core technologies such as CoWoS packaging and HBM memory are subject to export controls; there is a generational gap in domestic process technology, on-chip memory, and advanced packaging, and it is difficult to catch up with international performance by relying solely on process iteration.

[0005] 4. Fixed data flow and poor adaptability: Existing dedicated language processing units adopt a static data flow architecture. After compilation, the hardware topology cannot be changed, making it impossible to quickly adapt to the iteration of Transformer network structure and diverse control operators.

[0006] In summary, the industry urgently needs a large-scale inference hardware architecture that is compatible with domestic processes, alleviates the storage wall, balances high and low memory access loads, and has dynamic reconfiguration capabilities. Summary of the Invention

[0007] The purpose of this invention is to provide a three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture and method to solve the problems mentioned in the background art, such as insufficient storage bandwidth, poor hardware adaptation for two-stage inference, limited advanced packaging, and insufficient dataflow flexibility in the prior art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture: The near-memory storage plane, located at the top physical layer, integrates high-density DRAM wafers for storing model weights and KVCache; The Token Execution Plane, located in the physical middleware layer, contains multiple TGU slices of token generation units. Each TGU slice integrates a matrix multiplication array MXM, a vector operation unit VXM, and local SRAM. The TGU is software-reconfigurable hardware that supports switching between Prefill and Decode data stream modes. The computing control plane, located at the physical layer, contains RISC-V processor clusters and SIMT coprocessors; The hybrid bonding interconnect layer uses wafer-to-wafer WoW hybrid bonding technology to vertically interconnect three planar layers, with an interlayer aggregate interconnect bandwidth greater than 16TB / s.

[0009] Preferably, the TGU is configured as a pulsating array topology in Prefill mode to improve matrix multiplication throughput; and as a broadcast-reduction topology in Decode mode to reduce single-token decoding latency.

[0010] Preferably, the RISC-V cluster is equipped with a custom Xtgu instruction set extension, including the cfg_tgu instruction for configuring TGU mode, the set_tgu_shape instruction for setting the matrix operation dimension, the tgu_dma instruction for initiating data transfer, and the wait_tgu instruction for computation synchronization.

[0011] Preferably, it also includes on-chip UCIe interconnect, which interconnects multiple computing dies to build a unified UMA address space, allowing each die TGU to access all DRAM memory across dies.

[0012] Preferably, it also includes inter-chip YeS Link interconnection, equipped with multiple SerDes ports and compatible with RDMA-like protocols, to construct a fat tree or ring multi-card interconnection topology and realize direct memory access across cards.

[0013] Preferably, the RISC-V and SIMT coprocessors are responsible for RoPE, Softmax, and LayerNorm non-normalization operations during the Decode stage, while the TGU only performs matrix and vector normalization operations, thus realizing heterogeneous hardware division of labor in computing.

[0014] Preferably, the overall structure adopts a Logic-on-Bottom inverted stacking structure, with the high-heat computing control plane and token execution plane arranged close to the packaging substrate to optimize heat dissipation performance.

[0015] On the other hand, the method for accelerating large-scale model inference using three-dimensional hybrid bonding and reconfigurable dataflow, applying the architecture described in any one of claims 1-7, is characterized by comprising a Prefill step and a Decode step: Prefill step: The RISC-V cluster divides the model weights into blocks, loads the weights into the TGU's local SRAM through UCIe interconnect, configures the TGU in systolic array mode, reads the input data through WoW interconnect to complete batch matrix operations, and generates KVCache which is stored in the top-level DRAM. Decode steps: RISC-V and SIMT process the current token position encoding, switch TGU to broadcast reduction mode, TGU randomly reads the historical KV cache through WoW interconnect, performs autoregressive operation to generate the output token and update the KV cache.

[0016] Preferably, the Prefill step employs an SRAM double-buffering mechanism, where the TGU preloads the weights of the next block in parallel while performing the current block operation, and the calculation and data transmission overlap in parallel.

[0017] Preferably, hardware data flow is controlled by the Xtgu instruction set, and the compiler converts deep learning operators into instruction sequences consisting of cfg_tgu, set_tgu_shape, tgu_dma, and wait_tgu.

[0018] Compared with the prior art, the beneficial effects of the present invention are: (1)WoW hybrid bonding interconnect density is 10 times higher than traditional TSV, inter-layer bandwidth exceeds 16TB / s, picosecond-level vertical access latency, perfectly adapts to massive random small packet KV cache reading in the Decode stage, and can stably run 1M token ultra-long context reasoning in a single session.

[0019] (2) The inverted stacked heat dissipation structure reduces the chip operating temperature; the vertical copper bonding parasitic parameters are extremely low, which greatly reduces the power consumption of data transmission I / O; the hardware layering and division of labor avoids the GPU cache coherency protocol overhead, and the overall system energy efficiency ratio is significantly improved compared with general GPUs.

[0020] (3) By relying on the RISC-V custom Xtgu instruction set to dynamically switch the TGU hardware topology, the defects of static data flow solidification of traditional dedicated inference chips are eliminated, and various Transformer variants and new operator iterations can be quickly adapted, with high flexibility of software and hardware collaboration.

[0021] (4) Adopting the system-level innovation route of "multiple small computing dies + WoW advanced packaging", it does not need to rely on the most advanced logic process; relying on the mature 7nm logic process in China and domestic wafer hybrid bonding packaging, it can achieve inference performance comparable to international high-end chips and break through the CoWoS and HBM control restrictions.

[0022] (5) TGU is specifically responsible for the core regularization operation of token generation, and the control plane handles irregular operators with clear soft and hard boundaries; UCIe bare chip interconnection and YeS Link multi-card interconnection can flexibly expand storage and computing power, and support single-machine ultra-large model and multi-machine distributed inference. Attached Figure Description

[0023] Figure 1 This is a diagram of the overall logical structure of the Taiji architecture inverted stacking according to the present invention; Figure 2 This is a schematic diagram of the cross-sectional profile of the three-layer planar WoW hybrid bonding of the present invention; Figure 3 This is a diagram of the four-die chiplet interconnect architecture based on UCIe of the present invention; Figure 4 This is a diagram of the multi-card hierarchical networking architecture based on YeS Link of the present invention; Figure 5 This is a timing diagram of weighted blocks and UCIe / WoW data transmission in the Prefill stage of this invention; Figure 6 This is a schematic diagram comparing the data flow configurations of the two working modes of the TGU of this invention; Figure 7 This is a flowchart illustrating the software execution process of the RISC-V system controlling the TGU via the Xtgu instruction set, as described in this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figures 1-7 The present invention provides a technical solution: Option 1: 3D Hybrid Bonding and Reconfigurable Dataflow Large Model Inference Acceleration Architecture: The architecture consists of three physical planes, employing a Logic-on-Bottom inverted stacking. The three layers are vertically interconnected via WoW wafer hybrid bonding, achieving a combined interconnect bandwidth greater than 16TB / s. The overall architecture comprises three main parts: the three-layer stacked body, on-chip UCIe interconnects, and inter-chip YeSLink interconnects. Near-memory storage plane (top layer): integrates high-density 3D LPDDR5x DRAM wafers for unified storage of neural network weights and KVCache generated in real time during inference, with a single-chip storage capacity of up to 512GB.

[0026] Token Execution Plane (Middle Layer): Consists of multiple arrays of token generation unit (TGU) slices; each TGU slice integrates a matrix multiplication array MXM, a vector operation unit VXM, and local high-speed SRAM; the TGU is software-dynamically reconfigurable hardware, supporting two types of data flow topology switching: Prefill mode: Configured as a pulsating array to maximize batch matrix multiplication throughput; Decode mode: Configured as a broadcast-reduction topology, adapting to single-token random KV cache access, reducing autoregressive decoding latency.

[0027] Computational control plane (lower layer): Deploys a RISC-V multi-core processor cluster and a SIMT coprocessor, responsible for global task scheduling, non-integer operator operations, and TGU hardware mode configuration; it includes a built-in custom Xtgu instruction set extension, with core instructions including: cfg_tgu: Switches between TGU Prefill / Decode working modes and configures the precision of the calculated data; set_tgu_shape: Sets the row and column dimensions of the matrix multiplication and the block size; tgu_dma: Initiates WoW vertical interconnect data prefetching, moving DRAM data to TGU local SRAM; wait_tgu: Polls the TGU's computing status to complete the synchronization of computing across multiple hardware units.

[0028] On-chip UCIe interconnect module: Interconnects multiple computing dies in a 2×2 mesh topology to build a unified UMA and a unified physical address space. Any die TGU can access all on-chip DRAM without blocking, expanding the overall storage limit of the system.

[0029] Inter-chip YeS Link interconnect module: Equipped with multiple high-speed SerDes ports, compatible with RDMA-like direct memory access protocols; supports both fat tree and ring network topologies for interconnecting multiple accelerator card nodes, supporting pipelined parallel large-model distributed inference.

[0030] WoW Hybrid Bonded Interconnect Layer: Millions of micron-level copper bonding points are deployed between layers, with a bonding spacing of less than 1μm; DRAM and TGU, RISC-V and TGU are all vertically interconnected, eliminating the need for cross-layer cache coherency interaction, and reducing access latency to the picosecond range.

[0031] Thermal optimization structure: The inverted stacked design is adopted, with high-heat-generating hardware such as the computing control plane and token execution plane closely attached to the packaging substrate, optimizing the chip heat dissipation path and reducing thermal resistance.

[0032] Further division of labor mechanism: In the Decode stage, the RISC-V+SIMT coprocessor separately handles irregular operators such as RoPE rotation position encoding, Softmax, and LayerNorm; the TGU only performs normalized matrix and vector multiplication operations, realizing heterogeneous hardware division of labor to speed up the process.

[0033] Option 2: A method for accelerating large model inference, applying the above architecture, includes a prefill process and a decode process. Prefill step: (1) The underlying RISC-V scheduler parses the model parameters and divides all weights into fixed-size Tile blocks; (2) Call the cfg_tgu instruction to configure all TGUs to pulse array mode; (3) Weight blocks are loaded in parallel from remote DRAM to TGU local SRAM via UCIe interconnect, and a double buffering mechanism is adopted: when the current block is being processed, the next block is preloaded in the background, so that the calculation and data transmission are completely overlapped. (4) TGU reads the input Token sequence through WoW vertical interconnection and performs GEMM matrix operations in batches; (5) The output prefix KV Cache is written back to the top-level DRAM via WoW interconnect.

[0034] Decode steps: (1) The SIMT coprocessor completes the current Token RoPE location encoding calculation; (2) The RISC-V issues the cfg_tgu instruction to switch the TGU to broadcast-reduction data stream mode; (3) The TGU uses WoW high-density interconnect to randomly read all historical KV cache in the top-level DRAM; (4) Perform parallel operations on the attention layer, feedforward network FFN vector and matrix operations to generate a single output token; (5) The updated KV cache is written back to the top-level DRAM; (6) The decoding process is executed repeatedly until the text end symbol EOS is generated, at which point the inference terminates.

[0035] Example 1: Three-layer inverted stacked overall hardware architecture This embodiment of the Taiji architecture employs a three-layer vertical stacking structure: the physical top layer is a 512GB capacity 3D LPDDR5xDRAM near-memory plane; the middle Token execution plane arranges multiple TGU slices, each with an internal independent SRAM cache matrix for calculating intermediate values; the bottom computing control plane is equipped with an 8-core RISC-V cluster and a SIMT coprocessor. Millions of vertical interconnects are achieved between the three layers via a WoW wafer bonding layer, ensuring all DRAM and TGU data interactions follow vertical paths without being forwarded through the bottom control plane.

[0036] Example 2: Implementation of WoW wafer hybrid bonding interconnect The bonding layer has more than 10 million copper-copper micro-bonding points with a bonding pitch of less than 1μm; the DRAM wafer and TGU die are directly stacked face to face, eliminating the need for an intermediate silicon layer; vertical data transmission has no bus arbitration and no cross-layer cache synchronization overhead, and the random KV cache read latency is much lower than that of traditional 2.5D packaging solutions.

[0037] Example 3: Prefill and Decode Phased Execution Process Prefill stage: After the user inputs a long text prompt, RISC-V divides the 140GB model weights into tiles and configures the TGU as a pulsating array; relying on SRAM double buffering to load weights in parallel and perform matrix calculations, it generates complete prefix KV caches in batches and writes them to the top-level DRAM.

[0038] Decode phase: The SIMT unit calculates the RoPE position code for each token and switches the TGU to broadcast reduction mode; the TGU directly reads all historical KV caches randomly to complete the attention calculation, generates output tokens one by one and updates the KV cache in real time, and the loop continues until EOS terminates.

[0039] Example 4: Xtgu Custom Instruction Set Hardware and Software Collaboration The deep learning compiler automatically inserts Xtgu extension instructions during the operator compilation stage: in the decoding process, cfg_tgu switches the mode, set_tgu_shape sets the KV matrix size, tgu_dma initiates vertical data reading, and wait_tgu waits for the operation to complete. This enables the software to have full control over the hardware data flow and adapt to new models without having to re-fabricate the chip.

[0040] Example 5: Multi-GPU Distributed Expansion Deployment Eight accelerator cards of this invention are interconnected through YeS Link to form an inference node. A pipelined parallel strategy is used to allocate different network layers of Transformer to different accelerator cards. YeS Link supports RDMA direct memory access, and the transmission of activation values ​​between layers is not interrupted by the CPU, resulting in extremely low distributed inference latency.

[0041] During the compilation phase of the software-hardware collaborative control process, the high-level Transformer operators are automatically mapped to Xtgu instruction sequences. Relying on the combined instructions of cfg_tgu, set_tgu_shape, tgu_dma, and wait_tgu, the software achieves fine-grained control over the hardware data flow topology, data movement, and computation synchronization.

[0042] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture, characterized in that, The near-memory storage plane, located at the top physical layer, integrates high-density DRAM wafers and is used to store model weights and KV cache; The Token Execution Plane, located in the physical middleware layer, contains multiple TGU slices of token generation units. Each TGU slice integrates a matrix multiplication array MXM, a vector operation unit VXM, and local SRAM. The TGU is software-reconfigurable hardware that supports switching between Prefill and Decode data stream modes. The computing control plane, located at the physical layer, contains RISC-V processor clusters and SIMT coprocessors; The hybrid bonding interconnect layer uses wafer-to-wafer WoW hybrid bonding technology to vertically interconnect three planar layers, with an interlayer aggregate interconnect bandwidth greater than 16TB / s.

2. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, The TGU is configured as a pulsating array topology in Prefill mode to improve matrix multiplication throughput; and as a broadcast-reduction topology in Decode mode to reduce single-token decoding latency.

3. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, The RISC-V cluster is equipped with a custom Xtgu instruction set extension, which includes the cfg_tgu instruction for configuring TGU mode, the set_tgu_shape instruction for setting the matrix operation dimension, the tgu_dma instruction for initiating data transfer, and the wait_tgu instruction for computation synchronization.

4. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, It also includes on-chip UCIe interconnect, which interconnects multiple compute dies to build a unified UMA address space, allowing each die TGU to access all DRAM memory across dies.

5. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, It also includes inter-chip YeS Link interconnection, equipped with multiple SerDes ports and compatible with RDMA-like protocols, to build fat tree or ring multi-card interconnection topologies and realize direct memory access across cards.

6. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, The RISC-V and SIMT coprocessors are responsible for RoPE, Softmax, and LayerNorm non-normalization operations during the Decode stage, while the TGU only performs matrix and vector normalization operations, thus realizing heterogeneous hardware division of labor in computing.

7. The three-dimensional hybrid bonding and reconfigurable dataflow large model inference acceleration architecture according to claim 1, characterized in that, The overall design adopts a Logic-on-Bottom inverted stacked structure, with the high-heat computing control plane and token execution plane arranged close to the packaging substrate to optimize heat dissipation performance.

8. A method for accelerating inference in large-scale models using three-dimensional hybrid bonding and reconfigurable dataflow, characterized in that: The architecture described in any one of claims 1-7 is characterized by comprising a Prefill step and a Decode step: Prefill step: The RISC-V cluster divides the model weights into blocks, loads the weights into the TGU's local SRAM through UCIe interconnect, configures the TGU in systolic array mode, reads the input data through WoW interconnect to complete batch matrix operations, and generates KV Cache which is stored in the top-level DRAM. Decode steps: RISC-V and SIMT process the current token position encoding, switch TGU to broadcast reduction mode, TGU randomly reads the historical KV cache through WoW interconnect, performs autoregressive operation to generate the output token and update the KV cache.

9. The method for accelerating inference of large models with three-dimensional hybrid bonding and reconfigurable data flow according to claim 8, characterized in that, The Prefill step employs an SRAM double-buffering mechanism. When the TGU performs the current block operation, it preloads the weight of the next block in parallel, and the calculation and data transmission overlap in parallel.

10. The method for accelerating inference of large models with three-dimensional hybrid bonding and reconfigurable data flow according to claim 8, characterized in that, Hardware data flow is managed by the Xtgu instruction set. The compiler converts deep learning operators into a sequence of instructions consisting of cfg_tgu, set_tgu_shape, tgu_dma, and wait_tgu.

Images