System and operation method of three-dimensional accelerator and multiplication and accumulation device

Abstract

The invention discloses a system and an operation method of a three-dimensional accelerator and a multiplication and accumulation device. In one aspect, a system includes a plurality of memory layers, each including a group of memory groups. The system may include a multiply-accumulate layer including a multiply-accumulate array having a plurality of multiply-accumulate devices. Each of the multiply-accumulate devices may be coupled to a respective memory group of the group of memory groups through at least one via structure. The group of memories and the multiply-accumulate devices can be respectively configured in a predetermined number of rows and columns.

Description

Technical Field

[0001] One embodiment of this disclosure provides a system and method for operating a three-dimensional accelerator for generative artificial intelligence computation. Background Technology

[0002] In addition to various interconnections between circuit devices, integrated circuits (ICs) can contain a variety of hardware circuit devices or logic types, including FPGAs, application-specific integrated circuits (ASICs), logic gates, registers, or transistors. ICs can be manufactured using or constructed from semiconductor materials, for example, as part of electronic devices such as computers, portable devices, smartphones, and Internet of Things (IoT) devices. The development and increasing complexity of ICs have driven the growing demand for higher computing efficiency and speed. More specifically, ICs can be configurable and / or programmable to perform calculations in the sequence or variations required by manufacturers, developers, technicians, or programmers. Summary of the Invention

[0003] A three-dimensional accelerator system includes multiple memory layers, each containing a set of memory groups. The system includes a multiply-accumulate layer containing a multiply-accumulate array having multiple multiply-accumulate devices. Each multiply-accumulate device is coupled to a specific memory group within the set of memory groups via at least one via structure.

[0004] A multiply-accumulate device includes a multiply-accumulate array comprising a plurality of multiply-accumulate devices defined on a first semiconductor die. Each multiply-accumulate device includes an input buffer for storing at least one input vector. Each multiply-accumulate device includes a plurality of interconnect structures, each corresponding to a separate row of the multiply-accumulate device. Each interconnect structure includes a semiconductor via coupled to at least one second semiconductor die. The multiply-accumulate array is used to receive the input vector from the input buffer. The multiply-accumulate array is used to receive a plurality of data values ​​from the second semiconductor die via the interconnect structures. The multiply-accumulate array is used to generate a set of partial sums using the input vector and the data values.

[0005] An operating method for a 3D accelerator includes the following steps: storing a set of weight values ​​in a memory layer of a 3D accelerator circuit. The operating method may include the following steps: receiving input operands for multiply-accumulate operations from an input buffer. The operating method may include the following steps: providing the set of weight values ​​from the memory layer to a set of multiply-accumulate tiles in a multiply-accumulate layer of the 3D accelerator circuit. The set of weight values ​​is provided using a set of via structures coupling the memory layer to the multiply-accumulate layer. The operating method may include the following steps: generating an output vector using the multiply-accumulate layer based on the set of weight values ​​and the input operands. Attached Figure Description

[0006] The various features of this disclosure can be best understood by reading the following detailed description in conjunction with the accompanying drawings. It should be noted that, in accordance with industry standard practice, the features are not drawn to scale. In fact, the dimensions of the features may be arbitrarily increased or decreased for clarity of explanation.

[0007] Figure 1 A perspective block diagram illustrating an example of a three-dimensional (3D) accelerator circuit implemented according to some embodiments of this disclosure to process generative accelerated artificial intelligence (AI) computations.

[0008] Figure 2 The description of some embodiments according to this disclosure may include: Figure 1 A perspective block diagram of the instance multiply-accumulate (MAC) array layer in a 3D accelerator circuit;

[0009] Figure 3 Explanation of some embodiments of this disclosure: A cross-sectional block diagram illustrating the interconnection between the memory layer and the MAC layer of the 3D accelerator circuit described herein;

[0010] Figure 4 The block diagram illustrating some embodiments of this disclosure shows an instance mapping of a set of weight values ​​to memory components in the memory layer of the 3D accelerator circuit described herein;

[0011] Figure 5 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 4 The diagram shown contains a block diagram of other data values ​​stored in the mapping.

[0012] Figure 6 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 4 and Figure 5The diagram shown is a block diagram storing further data values.

[0013] Figure 7 This illustration demonstrates how the MAC device of the MAC array of the 3D accelerator circuit described herein can process data according to some embodiments of this disclosure. Figure 4 , Figure 5 and Figure 6 The mapping shown provides a block diagram of data values;

[0014] Figure 8 The illustration shows a block diagram illustrating a set of weight values ​​mapped to another instance of memory components in the memory layer of the 3D accelerator circuitry described herein, according to some embodiments of this disclosure.

[0015] Figure 9 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 8 The diagram shown contains a block diagram of other data values ​​stored in the mapping.

[0016] Figure 10 A block diagram illustrating how the memory layer is aligned on the MAC device of the MAC array in the 3D accelerator circuit described herein, according to some embodiments of this disclosure;

[0017] Figure 11 A flowchart illustrating an example operation method for operating the disclosed circuit described herein, according to some embodiments of this disclosure.

[0018] [Symbol Explanation]

[0019] 100: 3D Accelerator Circuit

[0020] 102, 302: Multiplication-Accumulation Layer / MAC Layer / MAC Array Layer

[0021] 104, 304, 1002, 304A~304D: Memory layer / MAC layer

[0022] 106: Through-hole connection

[0023] 108: Input buffer

[0024] 110: Global accumulator circuit

[0025] 112: Nonlinear function circuit / nonlinear activation circuit

[0026] 114: Incision

[0027] 200: Perspective block diagram / illustration

[0028] 202: Multiply-Accumulate Array Layer / MAC Array Layer

[0029] 204, 700, 1004: Multiply-accumulate array / MAC array

[0030] 206, 702, 1008: Multiply-accumulate device / MAC device

[0031] 208, 408, 508, 608, 706, 808, 908, 1010: Interconnection Structure

[0032] 210, 306: Through-hole structure

[0033] 300: Cross-sectional view

[0034] 308: Hybrid Bonding

[0035] 400, 500, 800, 900, 1000: Block diagram / attached diagram

[0036] 402, 802: Data tiles / tiles

[0037] 404, 504, 604, 804, 904: Data values

[0038] 406, 506, 606, 806, 906: Memory groups

[0039] 410, 510, 610, 810, 910: Memory components

[0040] 502, 602, 802, 902: Tiles / Zones

[0041] 600: Attached image

[0042] 704: Multiply-accumulate circuit / MAC circuit

[0043] 800, 900: Block diagram / attached diagram

[0044] 1006: Memory Partitioning

[0045] 1100: Operating Instructions

[0046] 1102, 1104, 1106, 1108: Operations Detailed Implementation

[0047] The following disclosure provides numerous different embodiments or instances for implementing various features of the provided subject matter. Specific examples of elements and configurations are described below to simplify this disclosure. Of course, these are merely examples and are not intended to be limiting. For example, in the following description, forming a first feature above or on a second feature may include embodiments where the first and second features are formed in direct contact, and may also include embodiments where an additional feature may be formed between the first and second features so that the first and second features are not in direct contact. Furthermore, reference numerals and / or letters may be repeated in various instances in this disclosure. This repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0048] Furthermore, for ease of description, spatial relative terms such as “beneath,” “below,” “lower,” “above,” “upper,” “top,” “bottom,” and similar terms are used herein to describe the relationship between one component or feature and another illustrated in the figures. In addition to the orientations depicted in the figures, spatial relative terms are also intended to cover different orientations of the device in use or operation. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein will be interpreted accordingly.

[0049] Generative artificial intelligence (AI) computations (such as MAC operations) are typically limited by memory bandwidth due to the amount of information that needs to be transmitted through the circuitry responsible for performing the computation. This memory bandwidth limitation results in latency where the computational circuitry is idle while data to be processed (e.g., weight data from the AI ​​model) is accessed, retrieved, and loaded into the appropriate temporary registers / memory components. These memory access latency significantly degrades the performance of familiar AI accelerator circuits.

[0050] More specifically, AI operations implemented by AI circuitry may include: a pre-filling phase, in which input cues with a number of symbols are propagated via a network to produce a first output symbol, involving matrix-matrix multiplication (MMM); and a generation phase, in which the next symbol is generated iteratively, with each iteration producing the latest generated symbol as new input to a transformer model, involving matrix-vector multiplication (MVM). Weight updates for the AI ​​circuitry are performed during the generation phase, where weights in the MAC circuitry can be updated between MVM operations. Because conventional AI circuitry is constrained by a pipelined architecture when performing weight updates, such circuitry is significantly limited by memory bandwidth, thus reducing device performance. Extending existing 2D circuitry to perform weight updates in fewer cycles results in impractical large circuit routing complexity or area usage.

[0051] To address these and other issues, the techniques described herein provide a three-dimensional (3D) accelerator circuit, comprising multiple layers of memory components and at least one layer of processing components, the processing components using position-aware weight mapping to significantly improve memory access performance. The 3D accelerator described herein uses 3D circuit interconnect structures (such as through-silicon vias, TSVs) to implement a multilayer memory device coupled to at least one processing layer of computing elements (e.g., MAC units, etc.). The memory layers and MAC array of the 3D accelerator can be connected such that the weight values ​​of generative AI models can be written to the MAC array in parallel and within a single cycle to improve device performance.

[0052] The 3D accelerator circuit described in this paper implements position-aware weight mapping, allowing weight values ​​of generative AI models (e.g., transformer-based models, such as generative pre-trained transformer (GPT) models) to be stored in memory groups / devices located above the computational circuitry that processes those weight values. Multiple memory groups / devices can be defined above the same computational circuitry in additional memory layers of the 3D accelerator circuitry. Storing specific weight values ​​in memory groups / devices above the computational circuitry that accesses those weight values ​​eliminates the wiring and circuitry required for laterally moving / shifting weight values ​​between components on the same layer, thereby significantly reducing memory access energy.

[0053] Figure 1This illustration shows a perspective block diagram of an example 3D accelerator circuit 100 implemented according to some embodiments of the present disclosure to accelerate generative artificial intelligence (AI) computation. The 3D accelerator circuit 100 is shown as including: at least one MAC layer 102 having a MAC array (combined with...) Figure 2 (Describing its details); and one or more memory layers 104, stacked on top of at least one MAC layer 102. In some embodiments, and as shown in this example, the 3D accelerator circuit 100 may further include an input buffer 108, a global accelerator circuit 110, and a nonlinear function circuit 112. In some embodiments, the 3D accelerator circuit 100 may include a controller for the memory layer 104. In this example, the notch 114 represents a notch corresponding to... Figure 3 The cross-section shown in Figure 300 is a cross-section.

[0054] The 3D accelerator circuit 100 may include one or more logic gates and sub-circuits, each of which may be constructed from one or more logic gates. A logic gate is an electronic device that performs logical operations on one or more input signals to generate a single output signal. Various embodiments of the circuitry and logic gates implementing the 3D accelerator circuit 100 may include various transistors. The transistors described herein may have a certain type (n-type or p-type), but the embodiments are not limited thereto. A transistor can be any suitable type of transistor, including but not limited to metal oxide semiconductor field effect transistors (MOSFETs), complementary metal oxide semiconductors (CMOS) transistors, P-channel metal-oxide semiconductors (PMOS), N-channel metal-oxide semiconductors (NMOS), bipolar junction transistors (BJTs), high-voltage transistors, high-frequency transistors, P-channel and / or N-channel field effect transistors (PFETs / NFETs), FinFETs, planar MOS transistors with convex source / drain electrodes, nanosheet FETs, nanowire FETs, or the like.

[0055] The 3D tiling accelerator circuit 100 is shown to include at least one input buffer 108, which may be a buffer circuit storing input data for one or more artificial intelligence operations. In some embodiments, the 3D tiling accelerator circuit 100 may include multiple input buffers 108. Although shown separately from the MAC layer 102 in this example, it should be understood that this representation is provided for visual clarity, and the input buffers 108 may be included in the circuitry of the MAC layer 102. In some embodiments, the input buffers may be defined in a layer separate from the MAC layer 102 and electrically coupled to the MAC layer using one or more interconnect structures.

[0056] Input buffer 108 may include any number of memory components, which may include dynamic random-access memory (DRAM) cells, static random-access memory (SRAM) cells, flash memory cells, eFuse memory cells, or any other type of memory cell capable of electronically storing information. Input buffer 108 may store input vectors for generative artificial intelligence operations, such as input vectors for vector-matrix multiplication performed by the MAC array of MAC layer 102. Input buffer 108 may provide input operands / vectors to at least one MAC array of MAC layer 102. In some embodiments, input buffer 108 may also receive and be modified by nonlinear activation circuitry 112. For example, when performing generative artificial intelligence operations (such as autoregressive text generation), the output of the first iteration may be used as input for subsequent iterations. Interconnection and logic circuitry may enable nonlinear activation circuitry 112 to provide its output for storing input buffer 108 for subsequent iterations.

[0057] Input buffer 108 may store information received from one or more external circuits (such as other memory circuits or processing circuits). Input buffer 108 may include memory components that store binary information of any suitable format, including floating-point data of various precisions, integer data of various precisions, or other types of electronic information. One or more control circuits may communicate with input buffer 108 to coordinate read operations (e.g., from one or more elements of MAC layer 102) and / or write operations (e.g., from nonlinear activation circuit 112, other external circuits / elements, etc.).

[0058] The 3D accelerator circuit 100 is shown as including at least one global accumulator circuit 110. The output of operations performed by the MAC circuitry of the MAC array of MAC layer 102 can be provided as input to the global accumulator circuit 110. Although shown separately from MAC layer 102 in this example, it should be understood that this representation is for visual clarity, and the global accumulator circuit 110 may be included in the circuitry of MAC layer 102. In some embodiments, the global accumulator circuit 110 may be defined in a layer separate from MAC layer 102 and electrically coupled to MAC layer 102 using one or more interconnect structures.

[0059] For example, the global accumulator circuit 110 can combine partial sums generated by the adder tree of MAC layer 102 to produce periodic / iterative outputs for generative artificial intelligence operations. For example, the MAC array can produce one or more partial sums for a vector-matrix MAC operation between a set of weight values ​​stored in the MAC array and a vector of input data in input buffer 108. Partial sums can be combined using one or more adder circuits included in the global accumulator circuit 110. In some implementations, the global accumulator circuit 110 can provide multiple parallel outputs, depending on the type of operation being performed. The operation to be performed using the global accumulator circuit 110 can be configurable based on instructions received from other circuitry communicating with the 3D accelerator circuit 100. In one example, the global accumulator circuit 110 can provide an output vector generated by a vector-matrix multiplication performed using MAC layer 102.

[0060] The 3D accelerator circuit 100 is shown to include at least one nonlinear activation circuit 112. The nonlinear activation circuit 112 may include logic gates, circuit elements, or other logic circuitry performing one or more activation function operations and / or pooling operations. Although shown separately from the MAC layer 102 in this example, it should be understood that this representation is for visual clarity, and the nonlinear activation function 112 may be included in the circuitry of the MAC layer 102. In some embodiments, the nonlinear activation circuit 112 may be defined in a layer separate from the MAC layer 102 and electrically coupled to the MAC layer 102 using one or more interconnect structures.

[0061] In some implementations, the output data generated by the global accumulator circuit 110 may be provided as input to the nonlinear activation circuit 112. The nonlinear activation circuit 112 may be an electronic circuit including various logic gates, transistors, or other logic elements or devices that can process the received data according to one or more activation functions and / or pooling functions. The activation function may be a nonlinear operation applied to each output generated by the global accumulator circuit 110. The activation function can be used to introduce nonlinearity into the data processed by the artificial intelligence model implemented by the 3D accelerator circuit 100. Pooling can be used to map downsampled output values ​​generated by the MAC operation described herein, thereby reducing the spatial size of the output in the aggregation while retaining information important to the artificial intelligence operation.

[0062] In some embodiments, the nonlinear activation circuit 112 can be used to perform max pooling, average pooling, or global pooling operations (e.g., global average pooling, global max pooling, etc.). In some embodiments, the output of the nonlinear activation circuit 112 can be stored in the input buffer 108 for further processing via the MAC layer 102. For example, after processing a set of input data stored in the input buffer 108 to produce a set of output data, different weight values / parameters stored in the memory layer 104 can be used in one or more MAC operations performed by the MAC layer 102. The output data / vectors provided by the nonlinear activation circuit 112 for storage in the input buffer 108 can then be used as input data for further processing using the weight / parameter values ​​of the artificial intelligence model according to the techniques described herein. In some embodiments, this process can be repeated until the output of the artificial intelligence model is produced.

[0063] The 3D accelerator circuit 100 is shown as including one or more memory layers 104. Although four memory layers 104 are shown in this example, it should be understood that any number of memory layers may be included in the 3D accelerator circuit 100. Memory layers 104 may include an array of memory groups, each of which may be coupled to one or more individual via connections 106. The via connections 106 in the 3D accelerator circuit 100 may be interconnect structures that facilitate communication between memory layers 104 and MAC layers 102. The via connections 106 may include any type of interconnect structure that facilitates the transmission of electronic signals between layers in a 3D semiconductor device architecture, including but not limited to hybrid bonding (HB) connections and / or through-silicon vias (TSVs). The via connections 106 may include vertical interconnects penetrating the semiconductor substrate of the memory layers 104, thereby enabling data transfer between different layers of the circuit.

[0064] As shown in the figure, via connections 106 are defined in parallel rows to allow the contents of memory groups / devices defined on memory layer 104 to be efficiently transferred to the MAC circuitry of MAC layer 102. For example, the parallel rows of via connections may be coupled to the corresponding MAC circuitry of MAC layer 102. In some embodiments, the via connections 106 of the first memory layer 104 may be coupled to the corresponding via connections 106 of the second memory layer 104 in the stack, which allows data to be shifted via memory groups / devices of different memory components. Figures 4 to 6 Further details describe the memory group / device and its corresponding through-hole connection 106.

[0065] The via connections 106 of memory layer 104 can couple memory groups / devices of the memory layer to corresponding MAC circuitry of MAC layer 102. In some embodiments, memory layer 104 may include electrical wiring, registers, buffers, or other circuitry to enable different memory devices to transfer data using via connections 106. Electrically coupling memory groups / devices using via connections 106 allows for the selective transfer of data, such as weight values, from memory layer 104 to the MAC circuitry of MAC layer 102. This reduces memory access power and latency, and improves the overall performance of various artificial intelligence operations conventionally limited by memory bandwidth / latency and access power. In some embodiments, via connections 106 can transmit data in extremely high throughput to improve overall device performance.

[0066] When performing various artificial intelligence operations, via connection 106 can be activated to transfer weight values ​​and / or other data from memory layer 104 to MAC circuitry in MAC layer 102. In some embodiments, via connection 106 can transfer data in parallel, thereby allowing simultaneous access to multiple data elements from different memory groups / devices and providing them to the corresponding MAC circuitry in MAC layer 102. In some embodiments, memory layer 104 may include wiring, components, or logic circuitry supporting multiple data transfer protocols and capable of dynamically adjusting the data transfer rate between memory layer 104 and MAC circuitry in MAC layer 102. Data transfer from memory layer 104 can be coordinated by one or more control circuits electrically coupled to memory layer 104. In some embodiments, control circuitry may be provided on MAC layer 102.

[0067] Each memory layer 104 may include a set of memory groups, each of which may include a corresponding set of memory devices. Each memory layer 104 may include any number of memory components, which may include DRAM memory cells, SRAM cells, flash memory cells, eFuse memory cells, or any other type of memory cell capable of storing information electronically. The memory components of memory layer 104 may be modified by one or more control circuits that write and / or read data to and from the memory components of memory layer 104. In some embodiments, the memory components of memory layer 104 may store weight values ​​or other parameters of an artificial intelligence model (such as a converter-based model). Memory layer 104 may provide one or more of these parameters to MAC layer 102 for processing. In some embodiments, the memory layers 104 of a 3D accelerator may collectively store all weight values ​​of a generative artificial intelligence model. In some implementations, memory layer 104 may include circuitry that enables data to be written rapidly and in parallel to memory groups / components of each memory layer 104, thereby significantly reducing latency in performing memory-binding operations. Figures 4 to 6 Further details describe the structure of the memory device and its corresponding mapping to the MAC circuit.

[0068] Figure 2 The description of some embodiments according to this disclosure may include: Figure 1 A perspective block diagram 200 shows an example MAC array layer 202 in a 3D accelerator circuit. The MAC array layer 202 is shown as including a MAC array 204. Figure 200 also shows a scaled view of an example MAC device 206 of the MAC array 204, which may be coupled to and communicate with interconnect structure 208. The MAC array layer 202 may be similar to and include... Figure 1 Any of the structures and functions of MAC layer 102.

[0069] MAC array layer 202 may include one or more logic gates and sub-circuits, each of which may be constructed from one or more logic gates. A logic gate is an electronic device that performs logical operations on one or more input signals to produce a single output signal. Various embodiments of the circuitry and logic gates implementing 3D accelerator circuit 100 may include various transistors. The transistors described herein may have a certain type (n-type or p-type), but the embodiments are not limited thereto. Transistors may be any suitable type of transistor, including but not limited to MOSFETs, CMOS transistors, PMOS, NMOS, BJTs, high-voltage transistors, high-frequency transistors, PFETs / NFETs, FinFETs, planar MOS transistors with raised source / drain electrodes, nanosheet FETs, nanowire FETs, or the like.

[0070] MAC layer 202 is shown as including MAC array 204. MAC array 204 may include one or more MAC devices 206. Each MAC device 206 in MAC array 204 may include binary multiplication circuitry, adder circuitry, and further circuitry facilitating data transfer, buffering, and other operations. In some embodiments, the multiplication circuitry may be any suitable circuitry capable of performing binary multiplication on integer or floating-point values, or both. The multiplier circuitry may multiply two values ​​(such as the value of input data and the weight / parameter values ​​of an artificial intelligence model) to produce a product. The adder circuitry of MAC device 206 may be used to accumulate products from multiple iterative and / or multiplication circuits to produce one or more sums or partial sums generated by MAC operations.

[0071] The adder circuit of MAC device 206 may be any suitable adder circuit that accumulates the product generated by the multiplier circuit. Any suitable adder circuit may include a full adder and a carry-lookahead circuit or the like. In some embodiments, MAC array 204 may include one or more adder trees that can sum and accumulate values ​​generated by the multiplication circuit of MAC device 206. In some embodiments, the adder circuit of MAC device 206 may include one or more registers or memory components to store the output of MAC device 206 over multiple processing cycles. For example, the adder circuit may include one or more registers that receive and accumulate the output of MAC device 206 of MAC array 204 to perform artificial intelligence operations. The registers may receive and accumulate the output of multiplier circuit of MAC device 206 until a sufficient number of multiplications have been performed to produce a partial sum for MAC operations. The output of each adder circuit of MAC device 206 may be provided as an output to a global accumulator circuit (e.g., Figure 1 The global accumulator circuit 110).

[0072] MAC layer 202 and / or MAC array 204 may include one or more input registers, interconnect circuit systems, or logic circuit systems (e.g., registers, buffers, multiplexers, etc.) to receive data from input data buffers (e.g., ...). Figure 1The input buffer 108 of the MAC layer 202 receives input data. In some embodiments, the MAC array 204 and / or each MAC device 206 may include an input register. The input register may receive input data from the input buffer of the MAC layer 202 (e.g., input buffer 108). The input register may include circuitry for writing to and reading from one or more memory components of the input buffer of the MAC layer 202. In some embodiments, the input register of the MAC device 206 may be pipelined by storing subsequent input data for the next iteration of the MAC operation in the input buffer after the input data is provided to the arithmetic circuitry / components of the MAC device 206.

[0073] MAC device 206 is shown coupled to and / or includes interconnect structure 208. The interconnect structure may include circuitry and logic components (e.g., logic gates, multiplexers, etc.) to route data to and / or from MAC device 206 using one or more via structures 210. Interconnect structure 208 may include circuitry for selectively providing data to one or more layers of the 3D accelerator circuitry described herein. In some embodiments, control circuitry communicating with interconnect structure 208 and / or MAC device 206 may provide control signals for controlling input and / or output data of MAC device 206. As described herein, MAC device 206 may receive data from memory layers (e.g., memory layers). Figure 1 The data and input buffers (e.g., memory layer 104) of the memory layer 104 Figure 1 The input buffer 108 is used to perform one or more MAC operations. Other circuitry in the interconnect structure 208 and / or MAC layer 202 may include logic components to route data into and out of the MAC device 206 to perform the MAC operations described herein.

[0074] Interconnect structure 208 is shown as including one or more via structures 210. In this example, eight via structures 210 are shown. However, it should be understood that any number of via structures 210 can be provided in conjunction with a corresponding MAC device. In some embodiments, the number of via structures 210 may correspond to the precision (e.g., number of bits, etc.) of the data to be processed by the MAC device 206. For example, eight via structures 210 may be provided to transmit 8-bit floating-point values. As described herein, interconnect structure 208 can be used to access and / or receive values ​​(e.g., weight values) from one or more memory layers (e.g., memory layer 104) of the 3D accelerator circuitry. The via structures 210 may include TSVs and may implement hybrid bonding between layers of the 3D accelerator circuitry to facilitate communication between multiple layers. Figure 3 Further details are described regarding the interconnect structure 208 and the via structure 210.

[0075] In this example, MAC array 204 is shown as comprising twelve MAC devices 206 configured in three columns and four rows. However, it should be understood that any suitable number of MAC devices 206 may be included in MAC array 204 to perform the various techniques described herein. In some embodiments, each MAC device 206 may operate in parallel, such that each MAC device 206 produces a corresponding partial sum for the artificial intelligence iteration during the same period. As described herein, the partial sums may be provided to a global accumulator circuit to produce the output of the MAC operation computed using MAC array 204.

[0076] Figure 3 This description illustrates a cross-sectional view 300 of the interconnect between the memory layers and the MAC layer of a 3D accelerator circuit described herein, based on some embodiments of this disclosure. In cross-sectional view 300, an example 3D accelerator is depicted, showing multiple memory layers 304A–304D (sometimes commonly referred to as “memory layer 304”) stacked on top of the MAC layer 302. Each memory layer 304 may be similar to… Figure 1 The memory layer 104 includes any of its structure and function. The MAC array layer 302 may be similar to... Figure 1 MAC layer 102 or Figure 2 The MAC layer 202, and includes Figure 1 MAC layer 102 or Figure 2 The MAC layer 302 can be defined on a first semiconductor die, and each memory layer 304 can be defined on an additional semiconductor die.

[0077] The memory layers 304A-304D of the 3D accelerator circuit are shown stacked on top of each other to form a multilayer semiconductor die structure. Each memory layer 304A-304D may include an array of memory groups / devices storing weight values ​​and other data required for operations performed by the MAC layer 302. As shown, each of the memory layers 304 includes a corresponding set of via structures 306 that connect the memory layer 304 to another layer in the 3D accelerator stack (e.g., to another memory layer and / or the MAC layer 302) using at least one hybrid bonding 308. Stacking the memory layers 304A-304D on top of the MAC layer 302 improves memory density and reduces wiring complexity, which would further significantly reduce memory bandwidth in conventional 2D methods.

[0078] Data to be processed by the MAC device / circuit of MAC layer 302 can be stored in memory groups / components of each memory layer 304 coupled to the via structure 306 that most closely approximates the MAC device / circuit. This minimizes the amount of wiring and buffering required to provide data stored in the memory groups / components of the 3D accelerator circuitry to the MAC circuitry / device of MAC layer 302. In this example, the memory groups / devices of MAC layer 304 share the via structure 306, which reduces the overall device area and wiring complexity within each memory layer. In some embodiments, some memory layers 304 may not need to share the via structure 306 and may be directly coupled to the corresponding structure of MAC layer 302 without being electrically coupled to the via structures 306 of other memory layers.

[0079] In some implementations, data can be subdivided among memory devices / groups within each memory layer 304, thereby avoiding simultaneous access to memory devices sharing the via structure 306. This method reduces the total area and number of via structures 306 within the 3D accelerator circuitry without sacrificing memory bandwidth. In instances where the 3D accelerator circuitry stores weight values ​​for an artificial intelligence model, each memory layer 304 may store weight values ​​for one or more layers of the artificial intelligence model, thereby minimizing or completely eliminating simultaneous access to memory components across different memory layers 304.

[0080] As described herein, the via structure 306 may be a TSV or other vertical interconnect that penetrates the substrate of each memory layer 304A-304D and MAC layer 302. The via structure 306 may be defined as parallel rows (sometimes referred to herein as “channels”), each row providing data to a row of MAC circuitry in MAC layer 302, as further described herein. The via structure 306 may be formed using any suitable face-to-back (F2B) process.

[0081] In this example, the via structure 306 of each memory layer 304 and MAC layer 302 is shown as coupled to one or more hybrid bonds 308. The hybrid bonds 308 can be used to electrically connect the memory layers 304 to each other and mechanically connect them to the MAC layer 302. Forming the hybrid bonds 308 may include forming metal-to-metal and / or dielectric-to-dielectric bonding techniques. The hybrid bonds 308 provide a low-resistance path for data transfer between the memory layers 304 and the MAC layer 302, thereby reducing memory access power and latency, and improving the overall performance of artificial intelligence computing. The hybrid bonds 308 can provide mechanical stability between the stacked memory layers 304 and the MAC layer 302.

[0082] In some implementations, due to the MAC device of MAC layer 302 (e.g., Figure 2The via structure 306 of the MAC layer 302 can couple to additional interconnect structures within the MAC layer 302 to compensate for the size differences between the MAC device 206 and memory groups / components in different memory layers 304. These additional interconnect structures can compensate for the MAC array (e.g., Figure 2 The difference in footprint size between the MAC array 204 and the memory groups / components of the memory layer 304. In one example, the MAC array of the MAC layer may have a smaller footprint than the memory components of the stacked memory layer 304. In such embodiments, the MAC layer 302 may include additional wiring metal, circuit elements, or logic components to route data from the via structure 306 to the corresponding MAC device of the MAC layer.

[0083] Figure 4 , Figure 5 and Figure 6 The description shows the weight values ​​of one or more layers of the artificial intelligence model, and the memory groups / components of different memory layers (e.g., Figure 1 An example diagram illustrating the mapping between the memory layer 104 and the MAC device (e.g., MAC device 206) of the MAC layer of the 3D accelerator circuit described herein. This mapping is provided for illustrative purposes, and it should be understood that data that can be stored in the memory layer and processed by the MAC layer of the 3D accelerator circuit described herein is configurable.

[0084] Figure 4 This illustration, based on some embodiments of the present disclosure, shows a block diagram 400 illustrating an example mapping of a set of weight values ​​to memory components in a memory layer of the 3D accelerator circuitry described herein. In this example, a set of data values ​​404 is subdivided into data tiles 402 for storage and processing by the 3D accelerator circuitry described herein. In one example, the data value 404 may be a single weight of a neural network or generative artificial intelligence model. In this example, the data value 404 is provided as a 24x16 matrix. To efficiently process the data values ​​using the 3D accelerator circuitry described herein, the data value 404 is subdivided into a number of data tiles 402, the number of data tiles 402 corresponding to the memory layer (e.g., ...). Figure 1 Memory layer 104 Figure 3 The number of memory components 410 in each memory group 406 (such as memory layer 304, etc.).

[0085] In this example, each memory layer of the 3D accelerator circuit includes 32 memory groups 406. The number of memory groups 406 within each memory layer corresponds to the number of MAC devices provided in the MAC array of the MAC layer of the 3D accelerator circuit. In this example, 32 memory groups 406 are provided, each corresponding to a combination of... Figure 7The diagram illustrates and describes 32 MAC devices in a MAC array. In this example, each memory group 406 is shown as comprising 12 memory units 410, each of which can store a specific data value 404. To store the data value 404 in the memory unit 410 of each memory group 406, the data value 404 is subdivided into a number of tiles 402, the number of tiles 402 corresponding to the number of memory units 410 in each memory group 406. In this example, the data value 404 is subdivided into 12 data tiles 402.

[0086] In the illustrated example, the data value 404 of each tile 402 is shown stored in memory group 406 according to the individual coordinates of the data value 404 within each tile 402. In this example, the top-left data value (row-column) of each tile 402 is stored in the memory unit 410 of the top-left memory group 406. In this example, since each of the data value tiles 402 comprises 32 data values ​​404, each data value 404 of each partition can be stored in the individual memory unit 410 of the corresponding memory group 406 corresponding to its row-column coordinates in the data value tile 402.

[0087] Figures 4 to 6 The mapping in the illustrated example is indicated by shading of tiles 402 and different data values ​​404 in each memory group 406. Specifically, each memory unit 410 within the memory group 406 is colored the same color to indicate the row-column coordinates of the respective data value 404 of its corresponding tile 402. In some embodiments, all data values ​​404 may be provided / stored in parallel in the memory units 410 of each memory group 406. In another example, all data values ​​may be pipelined such that all data values ​​404 may be written to the memory units 410 of the memory group 406 in a predetermined number of cycles. Although only a single memory layer is shown here, it should be understood that a greater number of data values ​​404 can be handled by storing data values ​​in memory groups 406 across multiple memory layers, each of which can be updated / written using pipelined write operations. Data values ​​can be written to each of the memory layers using corresponding memory control circuits, which can be provided in one or more of the memory layers and / or the MAC layers of the 3D accelerator circuitry.

[0088] Each memory group 406 may include logic for implementing parallel access to or provision of at least one data value 404 in at least one memory component 410 using interconnect structure 408. Interconnect structure 408 may be similar to... Figure 1The via connection 106 may include via structures or other interconnects to couple memory groups of the memory layer to one or more corresponding MAC devices. In this example, each memory group 406 is shown as providing data values ​​404 stored in its upper-left (row-to-column) memory component 410 to the MAC device of the MAC layer using an interconnect structure. Figure 5 and Figure 6 In further detail, this process can be repeated to efficiently process each weight value of a given tile in parallel.

[0089] Figure 5 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 4 The diagram 500 shows the block diagram storing other weight values. In diagram 500, the subsequent processing iterations of data value 504 are illustrated; data value 504 has been subdivided into similar... Figure 4 The data value is 404 for tile 402 and 502 for tile 502. In this example, Figure 500 can represent... Figure 4 The subsequent processing iterations of data value 404. As shown in the figure, a similar mapping between data value 504 and memory group 506 has been performed, such that the memory component 510 of each memory group 506 stores data value 504 with the same row-column coordinates in each individual tile 502.

[0090] As described herein, each memory group 506 may include memory peripheral logic that enables parallel access to or provision of at least one data value 504 in at least one memory component 510 using interconnect structure 508. In this example, the next data value 504 has been provided in parallel to the interconnect structure 508 of each memory group 506, representing the processing of tile 502 of the data value 504, which is the second from the left in the top row of tiles 502, as shown. In this example, each memory group 506 is shown as providing the data value 504 stored in the memory component 510 located in the second-to-last row from the left of the memory group 506. In this example, a single memory component 510 is shown as providing the data component to the interconnect structure 508. However, it should be understood that this is an example representation, and in some embodiments, multiple memory components 510 of the memory group 506 may provide the data value 504 to the interconnect structure 508 for parallel processing by the MAC array of the MAC layer.

[0091] Figure 6 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 4 and Figure 5The diagram shows a block diagram of the further data value 604 stored in the mapping. In Figure 600, the subsequent processing iterations of data value 604 are illustrated; data value 504 has been subdivided into similar... Figure 4 and Figure 5 The data values ​​are 404 and 504 for tile 402 and 502 for tile 602. In this example, Figure 600 can represent... Figure 4 The final processing iteration of data value 404. As shown in the figure, a similar mapping between data value 604 and memory group 606 has been performed, such that the memory component 610 of each memory group 606 stores data value 604 with the same row-column coordinates in its respective partition 602.

[0092] As described herein, each memory group 606 may include memory peripheral logic that enables parallel access to or provision of at least one data value 604 in at least one memory unit 610 using interconnect structure 608. In this example, the last data value 604 has been provided in parallel to the interconnect structure 608 of each memory group 606, representing the processing of the lower right partition 602 of the data value 604 in the process. In this example, each memory group 606 is shown as providing the data value 604 stored in the lower right memory unit 610 of the memory group 606. In this example, a single memory unit 610 is shown as providing the data unit to the interconnect structure 608. However, it should be understood that this is an example representation, and in some embodiments, multiple memory units 610 of the memory group 606 may provide the data value 604 to the interconnect structure 608 for parallel processing by the MAC array of the MAC layer. Figure 4 , Figure 5 and Figure 6 The example shown represents the first, second, and final iterations in the process of performing MAC operations on the entire set of data values ​​404, where some intermediate processing iterations of tiles 602 are omitted for visual clarity and simplicity.

[0093] Figure 7 This illustration demonstrates, according to some embodiments of the present disclosure, how the MAC device 702 of the MAC array 700 of the 3D accelerator circuit described herein can process... Figure 4 , Figure 5 and Figure 6 The map shown provides a block diagram of data values. The MAC array 700 can be included in the MAC layer of a 3D accelerator circuit, such as in combination with... Figure 1 , Figure 2 and Figure 3 As described. In this example, each MAC device 702 is shown as including MAC circuitry 704 communicating with interconnect structure 706. MAC circuitry 704 may be similar to... Figure 2 The described MAC device 206, and the interconnect structure 706 can be similarly combined Figure 2 The interconnection structure 208 is described.

[0094] In this example, from according to Figure 4 , Figure 5 and Figure 6 The data values ​​provided by the mapped memory components shown can be processed in parallel by each MAC device. As shown, data values ​​can be accessed or provided from interconnect structure 608, which couples each MAC device 702 to a corresponding memory group (e.g., ...). Figure 4 , Figure 5 and Figure 6 Memory groups 406, 506, 606, etc.). The mapping between MAC device 702 and its corresponding memory component and data value is shown by the shading of each MAC device 702. In this example, each MAC device can process one data value at a time. Each row of MAC device 702 can be obtained from the input buffer (e.g., Figure 1 The input buffer 108) additionally receives all or part of the input vector or input data structure.

[0095] In one example, each row of the MAC device 702 can perform input vector and weight matrix (by...) in the row direction. Figure 4 , Figure 5 and Figure 6 The partial sum is a multiplication operation of at least a portion of the data value representation. The partial sum can be calculated by summing the values ​​generated by the MAC device 702 in the column direction. The partial sum generated by the MAC array 700 can be provided to the global accumulator circuit (e.g., Figure 1 A global accumulator circuit 110 is used to generate an output value. In this example, the MAC array 700 can be used to generate the output of a vector-matrix multiplication operation in a single cycle, which can be performed as part of an artificial intelligence model computation. Each MAC device 702 can access and process data values ​​in parallel. (As combined...) Figure 4 , Figure 5 and Figure 6 As described, data values ​​from each memory bank can be provided and processed in parallel iteratively by each MAC device 702. In some embodiments, the MAC device 702 may include pipelined logic elements that process data values ​​in each cycle or in a predetermined number of cycles.

[0096] Figure 8 The illustration, according to some embodiments of this disclosure, shows block diagram 800 mapping a set of weight values ​​to another example of memory components in the memory layer of the 3D accelerator circuit described herein. Figure 800 illustrates the combination of... Figures 4 to 6 The described alternative mapping. In Figure 800, illustrating the first processing iteration of the instance of data value 804, data value 804 has been subdivided into similar... Figure 4The data value is 404 for tile 402 and 802 for tile 802. Similar to a combination. Figures 4 to 6 The described configuration subdivides data value 804 into a number of data tiles 802, the number of data tiles 802 corresponding to memory layers (e.g., Figure 1 Memory layer 104 Figure 3 The number of memory components 810 in each memory group 806 (such as memory layer 304, etc.).

[0097] In this example, interconnect structure 808 is used to provide the top four memory components 810 of each memory group 806 in the leftmost column of memory group 806 to the corresponding MAC devices in the MAC layer. Data values ​​804 provided from the four memory components 810 can be (e.g., via corresponding logic circuitry) mapped to the corresponding rows of four MAC devices (e.g., MAC devices 702) vertically aligned with memory group 806. Implementing this type of mapping can reduce the total number of row activations and peripheral overhead (e.g., mapping corresponding weight values ​​from each memory group 806 to multiple MAC devices within the corresponding row) at the cost of increased data movement within each memory layer.

[0098] Figure 9 The illustration shows how the 3D accelerator circuit described herein can be used to process data according to some embodiments of this disclosure. Figure 8 The diagram 900 shows other data values ​​stored in the mapping. In diagram 900, the subsequent processing iterations of data value 904 are illustrated; data value 904 has been subdivided into similar... Figure 8 Tile 802 and tile 902. In this example, Figure 900 can represent... Figure 8 The subsequent processing iterations of data value 804. As shown in the figure, a similar mapping between data value 904 and memory group 906 has been performed, such that the memory component 010 of each memory group 906 stores data value 904 with the same row-column coordinates in each individual tile 902.

[0099] In this example, the top row of four data values ​​904 in each memory group 906 of the next column (e.g., the second from the left) has been provided in parallel to the interconnect structure 908 of each memory group 906. (As combined with...) Figure 8 As described, the data value 804 provided from the four memory components 810 can be (e.g., via a corresponding logic circuit system) mapped to the corresponding row of four MAC devices (e.g., MAC device 702) that are vertically aligned with each memory group 806 providing the data value 804 via the interconnect structure 908.

[0100] Data values ​​can be provided iteratively in subsequent iterations, where each iteration provides the data value from the next column of the memory group in the memory array. After providing the top row of four data values ​​from the last (e.g., rightmost) column, the next (e.g., middle row) data value can be provided from the first (e.g., leftmost) column of the memory group. This process can be repeated until all data values ​​have been provided to the MAC array (e.g., combined with...). Figure 7 The corresponding MAC device of the described MAC array is processed by it.

[0101] Figure 10 Block diagram 1000 illustrates how memory layers are aligned on the MAC device of a MAC array in a 3D accelerator circuit described herein, according to some embodiments of this disclosure. Block diagram 1000 illustrates how memory layers 1002 (e.g., memory arrays) may occupy the same two-dimensional region as the corresponding MAC array 1004 of the 3D accelerator circuit described herein. Although a single memory layer 1002 is shown in this example, it should be understood that the 3D accelerator circuit may include any number of memory layers 1002. In embodiments where the 3D accelerator circuit includes multiple memory layers 1002, the interconnect structure 1010 coupling the memory layers 1002 to the MAC array 1004 may be shared among the memory layers 1002. In such embodiments, data values ​​can be mapped to the memory components of the memory layers 1002 in a manner that minimizes parallel access to the multiple memory layers 1002.

[0102] As shown in the figure, the MAC array 1004 includes a plurality of MAC devices 1008 vertically aligned with corresponding memory partitions 1006 of the memory layer 1002. In some embodiments, the MAC devices 1008 may occupy the same or similar areas as the memory partitions 1006. Maintaining similar areas between the MAC array 1004 and the memory layer 1002 minimizes energy consumption during memory access, thereby improving the overall power consumption of the 3D accelerator circuitry described herein.

[0103] refer to Figure 11 This document describes a flowchart illustrating an example operation method 1100 for operating the disclosed circuit described herein, according to some embodiments of this disclosure. Operation method 1100 can be used to perform MAC operations or other artificial intelligence operations. Operation method 1100 can be performed in conjunction with any of the systems, devices, circuits, or elements described herein. It should be understood that... Figure 11 Additional operations are provided before, during, and after operation method 1100, and some other operations may only be briefly described in this document.

[0104] In summary, operation method 1100 begins with operation 1102, which includes storing a set of weight values ​​(e.g., data values ​​404, 504, 604) in a memory layer of a 3D accelerator circuit (e.g., 3D accelerator circuit 100, etc.). Operation method 1100 proceeds to operation 1104, which includes receiving input operands (e.g., input vectors) from an input buffer (e.g., input buffer 108) for use in a MAC operation. Operation method 1100 proceeds to operation 1106, which includes providing the set of weight values ​​from the memory layer to a set of MAC tiles in the MAC layer of the 3D accelerator circuit. Operation method 1100 proceeds to operation 1108, which includes using the MAC layer to generate an output vector based on the set of weight values ​​and the input operands.

[0105] Referring to operation 1102, a set of weight values ​​can be stored in a memory layer of a 3D accelerator circuit. This set of weight values ​​can be provided using interconnect structures (e.g., via connection 106, via structure 306, hybrid bond 308, etc.) for storage in one or more memory components (e.g., memory components 410, 510, 610) of corresponding memory groups (e.g., memory groups 406, 506, 606) of one or more memory layers of the 3D accelerator circuit. The weight values ​​can be stored in a memory group directly connected to a corresponding MAC device corresponding to the weight values ​​to be processed (e.g., for artificial intelligence computation). The memory group can be coupled to the MAC device using a corresponding interconnect structure. The interconnect structure may include one or more TSVs and / or hybrid bonds to couple the memory layer to the MAC layer. In some embodiments, each memory layer may store a subset of the weight matrix of the artificial intelligence model. In some embodiments, each memory layer may store one or more weight matrices corresponding to one or more layers of the artificial intelligence model.

[0106] Referring to operation 1104, input operands (e.g., input vectors) can be received from an input buffer (e.g., input buffer 108, etc.). The input vector may be an input vector to be multiplied by weight tiles written into the MAC array. In some embodiments, the input vector is then to be multiplied by a plurality of weight tiles written sequentially into the MAC array. The input vector may include vectors used for artificial intelligence operations, such as vectors of digits, embeddings, or other digital data stored in generative artificial intelligence operations. The input vector may be provided as the output of a previous iteration of the 3D accelerator circuit described herein. In some embodiments, the input buffer may provide input operands to each row of MAC devices in the MAC array of the MAC layer, such that each MAC device can perform at least a portion of a vector-matrix MAC operation.

[0107] Referring to operation 1106, the set of weight values ​​can be provided from the memory layer to the corresponding MAC device in the MAC layer of the 3D accelerator for MAC operations. One or more TSVs and / or hybrid bonds coupling the memory layer to the MAC layer can be used to provide the weight values ​​to the MAC layer. As described in this document... Figures 4 to 9 As described, weight values ​​can be stored in a memory bank perpendicularly aligned with the MAC device to which those weight values ​​are to be processed. Through-hole structures can be used to provide weight values ​​in parallel to each of the MAC devices. In some implementations, weight values ​​can be provided to the MAC device at the MAC layer in a single cycle.

[0108] Referring to operation 1108, an output is generated using a MAC array defined on the MAC layer based on provided weight values ​​and input operands. Each MAC device in the MAC array can process at least a portion of the MAC layer using corresponding weight values ​​provided from one or more memory layers to generate a set of partial sums. The partial sums generated using the MAC devices can be accumulated into an output vector. For example, in some embodiments, the MAC layer may be coupled to or may include a global accumulator circuit. In such embodiments, the partial sums generated by the MAC devices (or columns of MAC devices) of the MAC array can be provided to the global accumulator circuit to generate an output vector. The output vector may be the result of a MAC operation between an input vector and one or more weight matrices stored in a first memory layer and a second memory layer.

[0109] In instances where the MAC operation is part of an artificial intelligence operation, the output vector can be provided to a nonlinear activation circuit (e.g., nonlinear activation circuit 112). The nonlinear activation circuit can use the output vector generated by the global accumulator circuit to perform one or more activation functions and / or pooling functions. In some embodiments, the output of the nonlinear activation circuit can be provided as a second output vector. In some embodiments, the second output vector of the nonlinear activation circuit can be stored in an input buffer for subsequent artificial intelligence operations. To perform multiple iterations of the MAC operation, the MAC array can iteratively retrieve (or provide) corresponding additional sets of weight values ​​from one or more memory layers to perform further MAC calculations.

[0110] In one embodiment of this disclosure, a system is disclosed. The system includes: a plurality of memory layers, each containing a group of memory units. The system also includes: a MAC layer containing a MAC array having a plurality of MAC devices. Each of the plurality of MAC devices is coupled to a specific memory unit within the group of memory units via at least one via structure.

[0111] According to some embodiments, the memory bank and the multiplication and accumulation device are each arranged in a predetermined number of rows and columns.

[0112] According to some embodiments, each of the multiply-accumulate devices is coupled to a single memory group within each memory layer of the memory layer.

[0113] According to some embodiments, it further includes input buffering circuitry that provides at least a portion of the input vector to at least one row of the multiply-accumulate device of the multiply-accumulate array.

[0114] According to some embodiments, the multiplication accumulation layer is defined on a first semiconductor die, and the memory layer is defined on a plurality of second semiconductor dies stacked on top of the first semiconductor die.

[0115] According to some embodiments, the columns of the multiply-accumulate devices in the multiply-accumulate array are used to generate partial sums.

[0116] According to some embodiments, it further includes a global accumulator circuit that communicates with the multiply-accumulate array, the global accumulator circuit being used to receive individual partial sums from each column of the multiply-accumulate device of the multiply-accumulate array.

[0117] According to some embodiments, the group of memory cells of each of the memory layers is coupled to a multiply-accumulate array using a shared interconnect structure.

[0118] According to some embodiments, the memory group includes a predetermined number of memory components.

[0119] According to some embodiments, the memory layer is coupled to the multiplication accumulation layer using a face-to-back stack with multiple hybrid bonds and multiple TSVs.

[0120] In another embodiment of this disclosure, a multiply-accumulate device is disclosed. The multiply-accumulate device includes: a MAC array comprising a plurality of MAC devices defined on a first semiconductor die; an input buffer for storing at least one input vector; and a plurality of interconnect structures, each corresponding to a separate row of the plurality of MAC devices. The plurality of interconnect structures includes: a semiconductor via coupled to at least one second semiconductor die. The MAC array is used to receive the at least one input vector from the input buffer. The MAC array is used to receive a plurality of data values ​​from the at least one second semiconductor die via the plurality of interconnect structures. The MAC array is used to generate a set of partial sums using the at least one input vector and the plurality of data values.

[0121] According to some embodiments, it further includes: a global accumulator circuit for: receiving the set of partial sums from the multiplication accumulator array; and generating an output vector corresponding to a vector-matrix multiplication operation between at least one input vector and a data value.

[0122] According to some embodiments, the nonlinear activation circuit is further used to provide the output vector to the input buffer for subsequent iterations of the artificial intelligence computation.

[0123] According to some embodiments, the multiplication and accumulation device is used to receive data values ​​from at least one second semiconductor die in a single cycle.

[0124] According to some embodiments, each of the multiplication and accumulation devices is used to iteratively: receive at least one data value of the weight matrix of the artificial intelligence model; and generate an output product based on at least a portion of at least one input vector and the data value.

[0125] According to some embodiments, the multiply-accumulate array is further used to generate individual partial sums in the set of partial sums by summing the output products of subsets of the multiply-accumulate devices in individual columns of the multiply-accumulate array.

[0126] According to some embodiments, the first semiconductor die is coupled to the second semiconductor die using a plurality of hybrid bonding junctions and a plurality of TSVs.

[0127] In another embodiment of this disclosure, an operation method is disclosed. The operation method may include the following steps: storing a set of weight values ​​in a memory layer of a 3D accelerator circuit. The operation method may include the following steps: receiving an input operand from an input buffer for a MAC operation. The operation method may include the following steps: providing the set of weight values ​​from the memory layer to a set of MAC tiles in a MAC layer of the 3D accelerator circuit. The set of weight values ​​is provided using a set of via structures coupling the memory layer to the MAC layer. The operation method may include the following steps: using the MAC layer to generate an output vector based on the set of weight values ​​and the input operand.

[0128] According to some embodiments, it further includes the step of generating an output based on a plurality of partial sums generated by the multiply-accumulate array via a global accumulator circuit of a multiply-accumulate circuit.

[0129] According to some embodiments, the method further includes the step of providing the set of weight values ​​from the memory layer to the set of multiplicative tiles in a single cycle.

[0130] As used herein, the terms “about” and “approximately” generally mean plus or minus 10% of the stated value. For example, about 0.5 would include 0.45 and 0.55, about 10 would include 9 to 11, and about 1000 would include 900 to 1100.

[0131] The foregoing summary outlines the features of several embodiments to enable those skilled in the art to better understand the nature of this disclosure. Those skilled in the art should understand that they can readily use this disclosure as a basis for designing or modifying other processes and structures to achieve the same purposes and / or advantages of the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made herein without departing from the spirit and scope of this disclosure.

Claims

1. A three-dimensional accelerator system, characterized in that, Include: Multiple memory layers, each containing a set of memory blocks; and A multiplication accumulation layer includes a multiplication accumulation array having multiple multiplication accumulation devices. Each of the plurality of multiply-accumulate devices is coupled to a separate memory group within the set of memory groups via at least one through-hole structure.

2. The system as described in claim 1, characterized in that, The memory group and the multiple multiply-accumulate devices are each arranged in a predetermined number of rows and columns.

3. The system as described in claim 1, characterized in that, It further includes an input buffer circuit that provides at least a portion of an input vector to at least one row of the plurality of multiply-accumulate devices of the multiply-accumulate array.

4. The system as described in claim 1, characterized in that, The multiplication accumulation layer is defined on a first semiconductor die, and the plurality of memory layers are defined on a plurality of second semiconductor dies stacked on top of the first semiconductor die.

5. The system as described in claim 1, characterized in that, One column of the plurality of multiply-accumulate devices in the multiply-accumulate array is used to generate a partial sum.

6. The system as described in claim 1, characterized in that, Each of the plurality of memory layers has its memory group coupled to the multiply-accumulate array using a shared interconnect structure.

7. The system as described in claim 1, characterized in that, The memory group contains a predetermined number of memory components.

8. The system as described in claim 1, characterized in that, The plurality of memory layers are coupled to the multiplication accumulation layer using a face-to-back stack with a plurality of hybrid bonds and a plurality of silicon vias.

9. A multiplication and accumulation device, characterized in that, Include: A multiply-accumulate array comprising a plurality of multiply-accumulate devices defined on a first semiconductor die; An input buffer is used to store at least one input vector; and Multiple interconnect structures, each corresponding to a separate row of the multiple multiply-accumulate devices, each interconnect structure including a semiconductor via coupled to at least one second semiconductor die, wherein the multiply-accumulate array is used for: Receive the at least one input vector from the input buffer. Multiple data values ​​are received from at least one second semiconductor die via the plurality of interconnect structures, and A set of partial sums is generated using at least one input vector and the plurality of data values.

10. A method for operating a three-dimensional accelerator, characterized in that, Includes the following steps: A set of weight values ​​is stored in a memory layer of a three-dimensional accelerator circuit; Receive an input operand from an input buffer for use in a multiplication-accumulation operation; The set of weight values ​​from the memory layer is provided to a set of multiply-accumulate tiles of a multiply-accumulate layer of the three-dimensional accelerator circuit, using a set of via structures that couple the memory layer to the multiply-accumulate layer to provide the set of weight values; and The multiplication accumulation layer is used to generate an output vector based on the set of weights and the input operands.

Images