A matrix multiplication calculation method based on a three-dimensional stacked chip sub-block structure

Abstract

The application discloses a matrix multiplication calculation method based on a three-dimensional calculation chip and a three-dimensional high-bandwidth memory. The system comprises a three-dimensional stacked calculation chip, a three-dimensional stacked high-bandwidth memory (HBM) and an interposer. The three-dimensional stacked calculation chip is vertically interconnected by a plurality of layers of calculation slices (Slices) through through-silicon vias (TSVs) and is divided into a plurality of three-dimensional calculation blocks in the vertical direction and the plane direction; the three-dimensional stacked high-bandwidth memory is vertically interconnected by a plurality of layers of dynamic random access memories (DRAMs) through TSVs and is also divided into a plurality of three-dimensional storage blocks. The plurality of three-dimensional calculation blocks and the plurality of three-dimensional storage blocks establish a corresponding mapping relationship in the three-dimensional space, so that the calculation and the storage are correspondingly close in space, and the data transmission path is shortened. The output tensor of matrix multiplication or batch matrix multiplication is blocked along three dimensions of Batch, M and N, a plurality of three-dimensional output sub-blocks are generated and are respectively distributed to corresponding three-dimensional calculation blocks, K dimensions are processed in turns as reduction dimensions, and each calculation block accumulates partial sums in an on-chip buffer. A two-dimensional processing element (PE) array is arranged in each three-dimensional calculation block, and each layer of calculation slices is vertically parallel operated; different storage groups (Bank Groups) are used in each three-dimensional storage block to separately store different types of data, and a single storage of a shared input sub-block is realized through an interposer broadcast interconnection structure, and a plurality of blocks are multiplexed; and a double-buffering mechanism is used to overlap the data loading and the multiplication and addition calculation.

Description

Technical Field

[0001] This invention relates to the fields of 3D stacked computing chips, 3D stacked memory, 3D packaging, and artificial intelligence computing acceleration chips, specifically to a matrix multiplication calculation method based on 3D stacked computing blocks, 3D stacked high-bandwidth memory blocks, and intermediate layer broadcast interconnection. The computing chip described herein is also referred to as a logic chip in some literature. Background Technology

[0002] Matrix multiplication and batch matrix multiplication (BMM) are core operations in artificial intelligence inference and training, accounting for a high proportion of the total computational cost of deep neural networks. As model sizes continue to increase, the demands on computing chips for bandwidth, computing power, and power consumption grow accordingly. Existing architectures face the following major bottlenecks:

[0003] First, the separation of computation and storage leads to bandwidth bottlenecks. Most existing AI acceleration chips adopt a two-dimensional planar structure, with computation units and storage units physically separated. Data needs to be transported over long distances between on-chip buffers and off-chip storage. Even with the introduction of High Bandwidth Memory (HBM), data still needs to be transmitted through multiple layers, including the underlying layer, intermediate layer, and chip boundary, creating a bandwidth bottleneck.

[0004] Second, the three-dimensional structure of HBM is not fully utilized. High-bandwidth memory adopts a vertically stacked structure of multi-layer dynamic random access memory (DRAM), which has natural three-dimensional spatial characteristics. However, in the existing system architecture, it is still abstracted as several two-dimensional channels. Three-dimensional block storage and three-dimensional data scheduling are not performed at the architecture level, resulting in low utilization of the internal bandwidth and spatial structure of HBM.

[0005] Third, three-dimensional stacked computing chips have not formed a three-dimensional architecture. Although some existing computing chips have achieved vertical stacking of the computing layer at the physical layer, such as using multi-layer silicon wafers with TSV interconnects, they still use two-dimensional partitioning, two-dimensional data paths, and two-dimensional scheduling models at the architecture level, failing to transform the physical advantages of vertical stacking into three-dimensional computing capabilities at the architecture level.

[0006] Fourth, computation scheduling and storage scheduling are not designed in a coordinated manner. In existing systems, computation block models and storage block models are often designed independently, resulting in asymmetric data paths between computation units and storage units, low data sharing efficiency, frequent waiting for storage data by computation units, and low utilization of processing elements (PEs).

[0007] Fifth, in existing technologies, high-bandwidth memories are usually organized for computation in units of channels. The multi-layer storage structure of high-bandwidth memories is not isomorphically divided into three-dimensional blocks and spatially mapped with the multi-layer computing structure of computing chips, resulting in asymmetric data paths between computing blocks and storage blocks. Existing broadcast mechanisms usually occur at the on-chip network or bus level, lacking a single-memory multi-block reuse mechanism that uses three-dimensional storage blocks as the source and adjacent three-dimensional computing blocks as the target.

[0008] Patent application CN119342842A, published on June 17, 2025, discloses a three-dimensional stacked integrated circuit architecture. This solution relies on an interposer layer to achieve a three-dimensional stacked layout of various chips, including memory chips, computing chips, and CPU chips. Through vertical stacking and short-distance interconnection between computing and storage chips, it reduces chip communication latency, improves chip computing power and storage capacity, and enables AI and machine learning related computational functions. However, this solution differs from the one described above. The latter only discloses a general three-dimensional chip stacking hardware architecture and does not design a dedicated three-dimensional computing block and three-dimensional storage block spatial mapping mechanism for matrix multiplication calculations. It also does not implement core matrix multiplication optimization strategies such as three-dimensional block scheduling of matrix tensors, storage group isolation, and double-buffered data overlap execution.

[0009] Patent application CN111291859A, published on June 16, 2020, discloses a general-purpose matrix-matrix multiplication dataflow accelerator semiconductor circuit based on a three-dimensional stacked DRAM architecture. This solution relies on a three-dimensional stacked DRAM structure, replacing traditional multiplication operations with a built-in lookup table in the memory, and combining a pulsating dataflow architecture and a hierarchical lookup mechanism to accelerate matrix multiplication, alleviating the memory wall problem in deep learning tensor computation. This solution differs from ours in its implementation method. The pulsating propagation-based matrix multiplication acceleration relies on lookup table operations and pulsating propagation, and it does not construct a three-dimensional computation architecture that integrates three-dimensional computation, three-dimensional storage, and three-dimensional matrix multiplication block collaboration. It lacks a spatial mapping relationship between three-dimensional computation blocks and three-dimensional storage blocks, and it does not implement dedicated optimization mechanisms for batch, M, N, K multi-dimensional matrix block scheduling, on-chip partial and cumulative operations, and single-copy storage of shared data for multiple-block reuse.

[0010] In summary, existing technologies fail to construct a fully integrated, three-dimensional collaborative computing architecture that combines three-dimensional computational blocks, three-dimensional high-bandwidth storage blocks, three-dimensional matrix multiplication blocks, three-dimensional data paths, three-dimensional broadcast scheduling, and three-dimensional double-buffered pipelines, making it difficult to fully leverage the matrix operation acceleration performance of three-dimensional stacked hardware. Therefore, the industry urgently needs a method to achieve collaborative mapping of three-dimensional computation, storage, and matrix computation blocks for matrix multiplication calculations. This method could shorten data transmission paths, improve matrix parallel computing capabilities, and maximize computing power utilization through data reuse and a three-dimensional double-buffering mechanism. Summary of the Invention

[0011] The purpose of this invention is to provide a method based on three-dimensional stacked computation blocks, three-dimensional stacked high-bandwidth storage blocks, intermediate layer three-dimensional broadcasting, and three-dimensional matrix multiplication parallel scheduling, so that computation and storage establish a corresponding mapping relationship in three-dimensional space, data flows along a shorter path, and realizes three-dimensional matrix multiplication block computation, shared input data, reduced redundant storage, broadcast reuse, and double-buffered pipeline.

[0012] To achieve the above objectives, this invention proposes a matrix multiplication calculation method based on a three-dimensional stacked chip block structure. The system includes a three-dimensional stacked computing chip, a three-dimensional stacked high-bandwidth memory, and an interconnect structure of an intermediate layer between the two. The three-dimensional stacked computing chip is divided into multiple three-dimensional computing blocks in both the vertical and planar directions, and the three-dimensional stacked high-bandwidth memory is divided into multiple three-dimensional storage blocks in both the vertical and planar directions. A one-to-one correspondence or proximity mapping relationship is established between the multiple three-dimensional computing blocks and the multiple three-dimensional storage blocks in three-dimensional space, thereby shortening the data transport path and improving data supply efficiency.

[0013] During matrix multiplication or batch matrix multiplication, the output tensor is divided into three-dimensional blocks along the Batch, M, and N dimensions, and each output sub-block is mapped to a corresponding three-dimensional computation block for execution. The K-dimensional dimension is used as the reduction dimension to divide the input into multiple reduction sub-blocks, which are processed in rounds. Each three-dimensional computation block performs on-chip accumulation of partial products. For input sub-blocks that satisfy sharing relationships, a one-to-many broadcast structure in the intermediate layer enables single-copy storage and multi-block reuse, reducing redundant storage and data transfer. Furthermore, a double-buffered structure is set within each three-dimensional computation block, allowing the loading of the current reduction sub-block to overlap with the multiplication and addition calculation of the previous reduction sub-block, thus forming a pipelined processing mechanism.

[0014] The three-dimensional stacked computing chip can be composed of multiple computing slices, with a two-dimensional PE array set in a local area of ​​each slice, and the multiple slices can perform parallel computing in the vertical direction. The three-dimensional stacked high-bandwidth memory can be composed of multiple DRAM stacks, and multiple storage groups or bank groups can be configured within the block to support the grouped storage and parallel access of input data, output data and prefetch data.

[0015] The invention will be further described below in conjunction with the system structure and method steps:

[0016] A matrix multiplication calculation method based on a three-dimensional stacked chip block structure, characterized in that:

[0017] The three-dimensional stacked computing chip comprises multi-layer computing slices stacked vertically and interconnected via TSVs, and is divided into several three-dimensional computing blocks in both the vertical and planar dimensions. Each three-dimensional computing block is equipped with a two-dimensional PE pulsation array and a dual-buffered storage structure built from two on-chip buffers. It is also equipped with a computing block-level control unit to coordinate the synchronous operation of each layer of computing slices, buffer switching, and data loading scheduling. Each PE unit in the PE pulsation array integrates a multiplier, an adder, and a register, and is compatible with two data flow operation modes: weight-resident WS and output-resident OS.

[0018] The three-dimensional stacked high-bandwidth memory is composed of multiple DRAM storage layers stacked vertically, interconnected by TSVs. The whole is divided into multiple three-dimensional HBM blocks in both vertical and planar directions. Each three-dimensional HBM block contains three BankGroups, which are used to store the data of input sub-block A, input sub-block B, and output sub-block C, respectively.

[0019] The intermediary layer is arranged between the 3D stacked computing chip and the 3D stacked high-bandwidth memory. It integrates high-density metal interconnects and one-to-many broadcast buffer nodes. Multiple 3D computing blocks and multiple 3D HBM blocks establish a one-to-one mapping or proximity mapping relationship in 3D space. Each 3D computing block and its corresponding 3D HBM block are directly opposite each other or adjacent to each other in the planar projection position of the intermediary layer.

[0020] The matrix multiplication calculation method includes the following steps:

[0021] Step 1: In HBM, the input tensor A (Batch,M,K), input tensor B (Batch,K,N) and output tensor C (Batch,M,N) are divided into three-dimensional blocks along the Batch, M and N directions to form multiple three-dimensional sub-blocks, and each three-dimensional sub-block is mapped to a three-dimensional computation block.

[0022] Step 2: Divide the K direction into multiple reduction sub-blocks. Each 3D computation block reads the sub-blocks of input tensor A and input tensor B in turn for the 3D output sub-block it is responsible for, performs multiply-accumulate calculations, and accumulates a partial sum in the on-chip buffer.

[0023] Step 3: Within each 3D computation block, different batch sub-ranges are processed vertically and in parallel through multi-layer computation slices; in the PE pulsation array within each computation slice, input data is transmitted along the row direction and weight data is transmitted along the column direction. Each PE performs multiply-accumulate operations in the same computation cycle and accumulates the results to the partial sum register until multiple computation cycles in the K direction are completed.

[0024] Step 4: For multiple 3D computation blocks with the same Batch block index and the same M block index but different N block indexes, share a sub-block of input tensor A; for multiple 3D computation blocks with the same Batch block index and the same N block index but different M block indexes, share a sub-block of input tensor B; the shared input sub-block is stored as a single copy in the source 3D HBM block and distributed to multiple target 3D computation blocks through the one-to-many broadcast buffer node of the intermediate layer;

[0025] Step 5: Implement double-buffered pipeline between the two on-chip buffers within each 3D computation block; while one buffer supplies the current reduced subblock data to the PE pulsating array, the other buffer loads the next reduced subblock data from the local or adjacent 3D HBM block. The two buffers work alternately and write the accumulated 3D output subblock back to the corresponding 3D HBM block.

[0026] The beneficial effects of this invention include: constructing a three-dimensional block computing structure; realizing the correspondence between three-dimensional storage and three-dimensional computing space; realizing three-dimensional matrix multiplication in a three-dimensional parallel manner; reducing the repeated storage of input data and reducing storage overhead through broadcast reuse; realizing a three-dimensional storage group structure with read-write separation; realizing intermediate layer broadcast and double-buffered pipeline; improving the utilization rate of high-bandwidth storage, the utilization rate of computing blocks and the parallelism of processing units, and reducing data transport distance. Attached Figure Description

[0027] Figure 1 This is a side cross-sectional view of the overall structure of the three-dimensional stacked chip according to an embodiment of the present invention.

[0028] Figure 2 is a schematic diagram of the three-dimensional matrix multiplication block calculation and the step-by-step accumulation along the K-dimensional axis according to an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the PE pulsating array architecture according to an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the timing of the double-buffered pipeline in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the overall data flow path in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will be described below with reference to the embodiments. The number of layers, blocks, processing unit size and memory specifications in the embodiments are for illustration only and are not intended to limit the scope of protection of the present invention.

[0033] The three-dimensional matrix multiplication referred to in this paper refers to a computational method in which the output tensor of matrix multiplication or batch matrix multiplication is divided into blocks along the three dimensions of Batch, M, and N, and these blocks are mapped to multiple three-dimensional computational blocks for parallel execution. The K dimension is used as the reduction dimension and processed in rounds. The scheme described in this paper does not change the mathematical definition of matrix multiplication itself.

[0034] The term "3D computation block" as used in this paper refers to a collection of computational units formed by vertically stacking the same planar partitions within a predetermined computational slice layer group, with local regions of each layer vertically interconnected via TSVs. The term "3D storage block" as used in this paper refers to a collection of storage units formed by vertically stacking the same planar partitions within a predetermined dynamic random access memory layer group, belonging to the same storage access domain through address mapping. The 3D storage block can be a physical layout block, a computational address mapping block, or a combination of both. Each 3D computation block contains a computation block-level control unit for coordinating the synchronous computation startup, dual-buffer switching, data loading scheduling, and partial summation control of the multiple computational slices within that block. The computation block-level control unit communicates with the layer-level control units within each computational slice through vertical TSVs and exchanges synchronization signals with the control units of other 3D computation blocks through an intermediary layer, achieving unified scheduling of the entire chip's computation.

[0035] Reference Figure 1 The system of this invention includes a 3D stacked computing chip, a 3D stacked high-bandwidth memory, an interposer, a bottom layer, and BGA solder balls. The 3D stacked computing chip and the 3D stacked high-bandwidth memory are placed side-by-side above the interposer and electrically connected to it via microbumps. The interposer contains high-density metal interconnects and one-to-many buffer nodes to enable high-bandwidth short-path communication between the 3D stacked computing chip and the 3D stacked high-bandwidth memory. A bottom layer is located below the interposer, and BGA solder balls are located below the bottom layer for connection to an external system board. The interposer is made of silicon or an organic interposer, supporting high-density signal connections between the 3D stacked computing chip and the 3D stacked high-bandwidth memory. The 3D stacked computing chip and the 3D stacked high-bandwidth memory are placed adjacent to each other on the interposer plane, with each 3D computing block directly opposite or adjacent to its corresponding 3D memory block in its planar projection, resulting in a short interconnect path between them.

[0036] The 3D stacked computing chip consists of eight silicon wafers, from Slice0 to Slice7, interconnected vertically by TSVs. Each slice is divided into four planar regions in a 2×2 configuration, with an 8×8 processing unit array in each region. Each processing unit supports FP16 multiplication and FP32 accumulation. The chip as a whole is divided into eight 3D computing blocks in a 2×2×2 configuration: vertically, it is divided into two levels: the lower half (Slices0 to Slice3) and the upper half (Slices4 to Slice7), and planarly, it is uniformly divided into four regions. The eight 3D computing blocks correspond to SB0 to SB7, where SB0 to SB3 are the lower half computing blocks, corresponding to the four planar regions of the lower slice, and SB4 to SB7 are the upper half computing blocks, corresponding to the four planar regions of the upper slice. Each 3D computing block consists of the homogeneous planar regions of the four computing slices within its corresponding vertical level. The four computing slices are interconnected via vertical TSVs, supporting vertical data transfer, control signal synchronization, and clock distribution.

[0037] Within each 3D computation block, buffers Buffer[0] and Buffer[1] are set in the local regions of each computation slice. The local buffers of each layer together constitute the double-buffered storage structure of the 3D computation block, and the computation is uniformly scheduled by the computation block level control. Each buffer partition can at least hold the sub-block of input tensor A, the sub-block of input tensor B, and part and data of the corresponding output sub-block required for the current reduction sub-block round.

[0038] In this embodiment, the three-dimensional stacked high-bandwidth memory conforms to the HBM2 specification; in other embodiments, stacked high-bandwidth memory conforming to HBM, HBM2, HBM3, or other subsequent compatible specifications can be used. The three-dimensional stacked high-bandwidth memory includes eight layers of memory chips from DRAM Layer 0 to DRAM Layer 7, with vertical interconnects between layers via TSVs. The bottom layer carries a base die responsible for external interfaces. The overall memory architecture is divided into eight three-dimensional memory blocks in a 2×2×2 pattern: vertically, it is divided into two parts: the lower half of DRAM Layer 0 to DRAM Layer 3 and the upper half of DRAM Layer 4 to DRAM Layer 7. Each single-layer plane is uniformly divided into four regions: left front 00, right front 01, left rear 10, and right rear 11. This results in eight three-dimensional memory blocks from HB0 to HB7, which are matched one-to-one with the three-dimensional computing blocks SB0 to SB7. Each memory block corresponds to an independent channel CH0 to CH7. Each 3D storage block contains three independent storage groups: Bank Group 0 stores sub-blocks of input tensor A and supports sequential read access; Bank Group 1 stores sub-blocks of input tensor B and supports sequential read access; Bank Group 2 stores the write-back positions of output sub-blocks and partial sum data and supports normal write and cumulative write.

[0039] By implementing memory group-level read / write separation, sub-block reads of input tensor A, sub-block reads of input tensor B, sub-block write-backs of output tensor C, and prefetching for the next stage can be performed in parallel or partially in parallel on different memory groups, reducing the probability of read / write conflicts and improving memory access efficiency. A corresponding mapping relationship is established between the eight 3D computation blocks SB0 to SB7 and the eight 3D storage blocks HB0 to HB7 in 3D space. SB0 HB0 CH0 Left front 00 Second half SB1 HB1 CH1 Right front 01 Second half SB2 HB2 CH2 Left rear 10 Second half SB3 HB3 CH3 Right rear 11 Second half SB4 HB4 CH4 Left front 00 First half SB5 HB5 CH5 Right front 01 First half SB6 HB6 CH6 Left rear 10 First half SB7 HB7 CH7 Right rear 11 First half

[0040] Each 3D computation block primarily accesses its local data within its corresponding 3D storage block. This local data is transmitted via a short path through the intermediary layer, resulting in low transmission latency and power consumption. Input sub-blocks that meet the sharing conditions obtain data from their corresponding or adjacent 3D storage blocks through a broadcast structure within the intermediary layer.

[0041] Referring to Figure 2, the execution process of three-dimensional block matrix multiplication is illustrated using a batch matrix multiplication with Batch=16, M=16, N=16, and K=16 as an example. It should be noted that... Figure 2b The block numbers shown (A0, A1, A2, A3, etc.) are schematic diagrams of numbering according to three-dimensional coordinate positions, and the numbers reflect the corresponding positions of the tensors in three-dimensional space.

[0042] Step 1: 3D Blocking. The input tensor A, input tensor B, and output tensor C are bisected along the Batch, M, and N dimensions respectively, resulting in eight 2×2×2 3D output sub-blocks: In the Batch dimension, batch 0 to 7 is the lower half, and batch 8 to 15 is the upper half; in the M dimension, m 0 to 7 is the first half, and m 8 to 15 is the second half; in the N dimension, n 0 to 7 is the left half, and n 8 to 15 is the right half. Each of the eight 3D computation blocks calculates a corresponding 3D output sub-block, with the following correspondence: SB0 C[0 to 7][0 to 7][0 to 7] SB1 C[0 to 7][0 to 7][8 to 15] SB2 C[0 to 7][8 to 15][0 to 7] SB3 C[0 to 7][8 to 15][8 to 15] SB4 C[8 to 15][0 to 7][0 to 7] SB5 C[8 to 15][0 to 7][8 to 15] SB6 C[8 to 15][8 to 15][0 to 7] SB7 C[8 to 15][8 to 15][8 to 15]

[0043] Step 2: K-direction reduction. Divide the K-direction (K=16) into two parts: the 0th round corresponds to k=0 to 7, and the 1st round corresponds to k=8 to 15.

[0044] For any computation block SB, compute the output sub-block it is responsible for. The process is as follows:

[0045]

[0046] Executed in two rounds:

[0047]

[0048]

[0049]

[0050] Step 3: Vertical Parallelism. A single 3D computation block contains four computation slices, with each slice handling the computation of two batches. Taking SB0 as an example, Slice0 is responsible for batches 0 to 1, Slice1 for batches 2 to 3, Slice2 for batches 4 to 5, and Slice3 for batches 6 to 7. The four computation slices run synchronously, forming vertical parallelism. The processing unit arrays of each layer complete 2D matrix multiplication and addition, and the multi-layer collaborative operation constructs a 3D matrix multiplication and addition computation path.

[0051] Reference Figure 3 For ease of illustration, this embodiment uses... Figure 3 The calculation process is illustrated using a 4×4 processing unit array as an example; in actual hardware, the processing unit array size in each planar region is 8×8, and its data flow principle is the same as... Figure 3 The 4×4 array shown is identical, only scaled proportionally in row and column size and number of beats. The processing unit array consists of multiple processing units numbered PE(i,j), where i and j are the row and column numbers, respectively, and are counted starting from 0; each processing unit integrates a multiplier, an adder, a partial sum register initialized to 0, an input data register, and a weight register.

[0052] Preferably, the PE pulsating array of the present invention can support either Weight Stationary (WS) or Output Stationary (OS) data flow modes: In WS mode, the weights are preloaded into the internal weight register of the PE and remain unchanged across multiple computation cycles; input activations are passed forward to the next PE row by row, and partial sums are passed down and accumulated column by column; In OS mode, partial sums reside in the internal accumulator of the PE and do not move; weights and input activations are simultaneously input from outside the array to each PE and multiply-accumulate. The computation block-level control unit can flexibly switch between the WS and OS data flow modes according to different matrix shapes, batch sizes, and data reuse characteristics.

[0053] At the t-th computation cycle, corresponding to the k_t-th reduction position in the K-dimensional matrix: A[0][k_t], A[1][k_t], A[2][k_t], and A[3][k_t] from the sub-block matrix of A are respectively sent to rows 0 through 3, and transmitted along their respective row directions to all processing units in that row; B[k_t][0], B[k_t][1], B[k_t][2], and B[k_t][3] from the sub-block matrix of B are respectively sent to columns 0 through 3, and transmitted along their respective column directions to all processing units in that column; The execution part and accumulation of each processing unit PE(i,j) are as follows: .

[0054] After all K-direction beats are completed, each processing unit outputs the corresponding result element C[i][j]. After all K-direction beats corresponding to the current reduction sub-block are completed, each processing unit in the array obtains a partial sum of the corresponding output element. For Figure 3 The 4×4 array shown can output 16 result elements of a 4×4 output block in parallel with 16 processing units; for the actual 8×8 array in hardware, 64 processing units can output 64 result elements of an 8×8 output block in parallel. The 4×4 array expands the data flow by row and column: a set of first input data is transmitted along the row and a set of second input data is transmitted along the column in a single cycle. The 16 processing units perform multiplication and addition synchronously. After passing through multiple computation cycles corresponding to the length of the current reduction sub-block along the K direction, all result elements of the two-dimensional output block complete one round of partial sum accumulation; multi-layer computation slices process different batch ranges in parallel in the vertical direction, thereby expanding the computation of a single-layer two-dimensional processing unit array into a three-dimensional computation block for solid matrix multiplication.

[0055] Each 3D HBM block employs a Bank Group separation strategy to store different types of data. Each Bank Group supports independent access, and access operations do not interfere with each other. The storage group partitioning method shown in the aforementioned Bank Group allocation table allows the reading of sub-blocks of input tensor A, the reading of sub-blocks of input tensor B, the writing back of output sub-blocks, and the prefetching of the next stage to be performed in parallel or partially in parallel on different Bank Groups. The specific capacity of each Bank Group can be flexibly configured according to the actual matrix sub-block size and the number of reduced sub-blocks.

[0056] In 3D block matrix multiplication, multiple 3D computation blocks with the same Batch block index and the same M block index, but different N block indices, require the same sub-blocks of input tensor A; similarly, multiple 3D computation blocks with the same Batch block index and the same N block index, but different M block indices, require the same sub-blocks of input tensor B. For input sub-blocks that satisfy the sharing condition, a primary memory copy is retained in the source 3D storage block, and the interconnect structure is broadcast through the intermediate layer for reuse by multiple 3D computation blocks, reducing redundant storage.

[0057] Taking SB0 and SB1 as examples, the intermediate layer broadcast mechanism is introduced: SB0 is responsible for calculating C[0 to 7][0 to 7][0 to 7], and the required data is A[0 to 7][0 to 7][0 to K-1] and B[0 to 7][0 to K-1][0 to 7]; SB1 is responsible for calculating C[0 to 7][0 to 7][8 to 15], and the required data is A[0 to 7][0 to 7][0 to K-1] and B[0 to 7][0 to K-1][8 to 15]; the two have the same Batch block index and the same M block index, but different N block indexes, and share the sub-block A[0 to 7][0 to 7][0 to K-1] of the input tensor A.

[0058] The intermediate layer broadcast mechanism is implemented as follows: Sub-blocks of shared input tensor A are stored only in Bank Group 0 of HB0 as primary storage copies; after HB0 reads the shared data, it transmits it to the one-to-many buffer nodes inside the intermediate layer. The buffer nodes copy the data twice and send it to SB0 and SB1 respectively; SB0 reads the sub-blocks of input tensor A and B[0 to 7][0 to K-1][0 to 7] from its local HB0, and SB1 reads B[0 to 7][0 to K-1][8 to 15] from its local HB1, while simultaneously obtaining the sub-blocks of shared input tensor A in HB0 through the broadcast path. Similarly, SB0 and SB2, which are adjacent in dimension M, have the same batch block index and the same N block index, but different M block indices. Sub-blocks of shared input tensor B are also broadcast through the intermediate layer, and the primary storage copy is reused by the two computation blocks.

[0059] Reference Figure 4 Each three-dimensional computation block has a double-buffered storage structure containing two on-chip buffer partitions, Buffer[0] and Buffer[1]. In this embodiment, Buffer[0] is used to carry the input data of the 0th round reduction sub-block (K=0) in the K direction, and Buffer[1] is used to carry the input data of the 1st round reduction sub-block (K=1) in the K direction. The two work alternately so that the loading of HBM data and the multiply-accumulate calculation of PE array can be performed in time overlapping.

[0060] Figure 4 Taking a 3D computation block processing two batches (Batch0 and Batch1), with each batch divided into two reduction sub-blocks (K=0 and K=1) along the K direction as an example, the timing process of a double-buffered pipeline is described. The required input data is read from two storage sub-blocks in the 3D HBM block: HBM storage sub-block 0 provides the data for batch K=0, and HBM storage sub-block 1 provides the data for batch K=1. The entire pipeline process includes the following stages:

[0061] Phase P0 (Initial Prefetch): Reads batch K=0 data from Batch0 from HBM storage sub-block 0 and writes it to the computing chip sub-block Buffer[0] area; the PE array has not yet been started in this phase and is in the initialization state.

[0062] Phase P1 (Batch0 first round calculation + read and write Batch0 second round data): The calculation chip sub-block Buffer[0] provides the K=0 batch data of Batch0 to the PE array and starts multiply-accumulate calculation, and accumulates the result to the local part and register; at the same time, it reads the K=1 batch data of Batch0 from HBM storage sub-block 1 and writes it to the calculation chip sub-block Buffer[1] area.

[0063] Phase P2 (Batch 0 final round calculation + read and write Batch 1 first round data + write back the result after Batch 0 calculation): The calculation chip sub-block Buffer[1] provides Batch 0 K=1 batch data to the PE array, participates in the calculation, and is partially added with the previous round. After completing the final accumulation result of Batch 0 output sub-block C, it is written back to the BankGroup 2 of the corresponding three-dimensional HBM block through the intermediary layer; at the same time, Batch 1 K=0 batch data is read from HBM storage sub-block 0 and written to the calculation chip sub-block Buffer[0] area to prepare for the calculation of the next batch.

[0064] Phase P3 (Batch 1 first round of calculation + reading and writing Batch 1 second round of data): The calculation chip sub-block Buffer[0] supplies Batch 1 K=0 batch data to the PE array and starts the calculation; at the same time, it reads Batch 1 K=1 batch data from HBM storage sub-block 1 and writes it into the calculation chip sub-block Buffer[1] area.

[0065] Stage P4 (Batch 1 final round calculation + Batch 1 calculation results written back): The calculation chip sub-block Buffer[1] supplies Batch 1 K=1 batch data to the PE array for calculation, and adds it with the previous round partial sum. After the final accumulation of Batch 1 output sub-block C is completed, it is written back to the Bank Group 2 of the corresponding three-dimensional HBM block through the intermediary layer. This pipeline section is completed.

[0066] Through the above alternating scheduling, Buffer[0] and Buffer[1] are bound to the data channels of K=0 and K=1 respectively, and are prefetched by the two HBM storage sub-blocks respectively, so as to realize the time overlap of data loading and multiply-accumulate calculation to hide the HBM access delay; when the on-chip capacity is sufficient, some are stored on the chip and written back after the batch is accumulated; when the capacity is insufficient, they are written back to Bank Group 2 in stages. This process can be naturally extended to more batches and more K-direction reduction sub-blocks. The two buffers always work alternately in the manner of "one supply and one prefetch" until all the output sub-blocks responsible for this three-dimensional calculation block are calculated.

[0067] Reference Figure 5Taking the sub-block of input tensor A read from HB0 by SB0 as an example, the complete chip data flow transmission and calculation path is sorted out. Bank Group 0 of HB0 is used to store the data of the shared input sub-block A. The data is first read vertically through the internal TSV of HBM, and then transmitted to the input end of the intermediate layer through the output driver of HBM substrate and microbump. Then, it is sent to the broadcast buffer node through the metal trace inside the intermediate layer. The node performs one-to-many copying and splitting of the data, one broadcast to the corresponding calculation area of ​​SB0, and the other broadcast to the corresponding calculation area of ​​SB1. The target data flowing to SB0 is accessed to the bottom layer of the computing chip through the microbump, and vertically distributed to the target computing slice through the internal TSV of the computing chip and stored in the slice's built-in on-chip double buffer Buffer[0] or Buffer[1] to complete the on-chip loading of the input data. After the data is loaded, the buffer broadcasts the matrix A and matrix B data along the row and column directions respectively. The data is distributed to each PE unit in the array through the row and column broadcast bus of the PE pulsating array. Each PE(i,j) synchronously performs multiplication and addition operations. The multi-beat part and accumulation are completed along the K dimension, and the final matrix calculation result C[i][j] is output. The result data is then written back along the link in reverse, passing through the internal TSV, microbump, intermediate layer trace, and HBM interconnect path of the computing chip in sequence, and then written into the Bank Group 2 storage space of HB0. The entire link fully realizes the entire process of three-dimensional storage data reading, on-chip pulsating matrix multiplication operation, and reverse storage of calculation results. The entire process relies on the chip's three-dimensional vertical stacking architecture to shorten the data transmission path. Combined with broadcast data multiplexing and on-chip double buffer pipeline mechanism, the data handling overhead is greatly reduced, and the overall operation efficiency of matrix multiplication of three-dimensional stacked chips is effectively improved. Other embodiments

[0068] The architecture described in this invention is not limited to the specific configuration of the 8-layer slice and 2×2×2 block. In other embodiments, the number of layers in the 3D stacked computing chip can be 4, 8, 16, or other numbers; the planar blocks can be symmetrical or asymmetrical forms such as 2×2, 1×4, and 4×4; and a one-to-one, one-to-many, or many-to-one mapping relationship can be adopted between the 3D computing blocks and the 3D HBM blocks. Accordingly, the number of blocks of the output tensor along the Batch, M, and N directions can be flexibly configured according to the actual matrix shape to adapt to inference and training scenarios of different scales, including edge inference, large-batch training, large-vocabulary language model output projection, computationally intensive or bandwidth-intensive matrix multiplication, and other application scenarios. All the above modifications are within the protection scope of this invention.

Claims

1. A matrix multiplication calculation method based on a three-dimensional stacked chip block structure, characterized in that: The system includes a three-dimensional stacked computing chip, a three-dimensional stacked high-bandwidth memory, and an intermediary layer; The three-dimensional stacked computing chip comprises multi-layer computing slices stacked vertically and interconnected via TSVs, and is divided into several three-dimensional computing blocks in both the vertical and planar dimensions. Each three-dimensional computing block is equipped with a two-dimensional PE pulsation array and a dual-buffered storage structure built from two on-chip buffers. It is also equipped with a computing block-level control unit to coordinate the synchronous operation of each layer of computing slices, buffer switching, and data loading scheduling. Each PE unit in the PE pulsation array integrates a multiplier, an adder, and a register, and is compatible with two data flow operation modes: weight-resident WS and output-resident OS. The three-dimensional stacked high-bandwidth memory is composed of multiple DRAM storage layers stacked vertically, interconnected by TSVs. The whole is divided into multiple three-dimensional HBM blocks in both vertical and planar directions. Each three-dimensional HBM block has three Bank Groups, which are used to store data of input sub-block A, input sub-block B, and output sub-block C, respectively. The intermediary layer is arranged between the 3D stacked computing chip and the 3D stacked high-bandwidth memory. It integrates high-density metal interconnects and one-to-many broadcast buffer nodes. Multiple 3D computing blocks and multiple 3D HBM blocks establish a one-to-one mapping or proximity mapping relationship in 3D space. Each 3D computing block and its corresponding 3D HBM block are directly opposite each other or adjacent to each other in the planar projection position of the intermediary layer. The matrix multiplication calculation method includes the following steps: Step 1: In HBM, the input tensor A (Batch,M,K), input tensor B (Batch,K,N) and output tensor C (Batch,M,N) are divided into three-dimensional blocks along the Batch, M and N directions to form multiple three-dimensional sub-blocks, and each three-dimensional sub-block is mapped to a three-dimensional computation block. Step 2: Divide the K direction into multiple reduction sub-blocks. Each 3D computation block reads the sub-blocks of input tensor A and input tensor B in turn for the 3D output sub-block it is responsible for, performs multiply-accumulate calculations, and accumulates a partial sum in the on-chip buffer. Step 3: Within each 3D computation block, different batch sub-ranges are processed vertically and in parallel through multi-layer computation slices; in the PE pulsation array within each computation slice, input data is transmitted along the row direction and weight data is transmitted along the column direction. Each PE performs multiply-accumulate operations in the same computation cycle and accumulates the results to the partial sum register until multiple computation cycles in the K direction are completed. Step 4: For multiple 3D computation blocks with the same Batch block index and the same M block index but different N block indexes, share a sub-block of input tensor A; for multiple 3D computation blocks with the same Batch block index and the same N block index but different M block indexes, share a sub-block of input tensor B; the shared input sub-block is stored as a single copy in the source 3D HBM block and distributed to multiple target 3D computation blocks through the one-to-many broadcast buffer node of the intermediate layer; Step 5: Implement double-buffered pipeline between the two on-chip buffers within each 3D computation block; while one buffer supplies the current reduced subblock data to the PE pulsating array, the other buffer loads the next reduced subblock data from the local or adjacent 3D HBM block. The two buffers work alternately and write the accumulated 3D output subblock back to the corresponding 3D HBM block.

Images