Die-to-die dense package for deterministic streaming processors

Abstract

Embodiments relate to an integrated circuit having a plurality of dies connected in a die-to-die (D2D) configuration. An integrated circuit may include a first die and a second die connected to the first die in a D2D configuration via a D2D interface circuit to form a D2D structure with the first die. The D2D interface may connect the first plurality of hyper-channels of the first die with the second plurality of hyper-channels of the second die for streaming data between the first die and the second die in a first direction or a second direction orthogonal to the first direction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 63 / 312,781, filed on February 22, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to processor architectures having multiple dies and, more particularly, to die-to-die dense packaging of deterministic streaming processors. Background Art

[0004] Over the past decade, data center operators have installed multi-core processor systems in warehouse-scale computers. These systems have tens to thousands of processing cores that vary widely in form and function, used in powerful graphics processing units (GPUs), tensor processing units (TPUs), field programmable gate arrays (FPGAs), many of which use data network controllers for efficient remote memory access. Such systems are often used to accelerate the performance of deep neural network training and inference (application of trained neural networks) for a wide range of uses in business, such as convolutional neural networks (CNNs) for recommendation algorithms, computer vision, and image classification for product search; and recurrent neural networks (RNNs) for natural language processing in user interfaces. The increased computational requirements of these models, many of which involve very large vector and matrix calculations, have been a catalyst for a resurgence of architectural innovation as they are used by millions of consumers and billions of transactions.

[0005] In a conventional chip multiprocessor (CMP), the processing cores are interconnected using an on-chip network to exchange data between all the processing cores. In this simple load-store model, a set of general-purpose data registers are used as intermediate storage between the main memory system and the processor cores, which may include an arithmetic logic unit (ALU) that operates on the data. Instructions are dispatched to each core and executed by local integer or floating-point processing modules, while intermediate results are stored in general-purpose registers. This load-store architecture moves data (also called "operands") and calculation results between registers and main memory. Instruction execution is typically performed in several stages: 1) instruction fetch, 2) instruction decode, 3) execution on the ALU, 4) memory read, 5) memory write to update the result in the register.

[0006] However, CMP workloads continue to grow in both size and complexity, placing severe scalability, performance, and availability demands on traditional CMP architectures. Managing this demand effectively requires using a large number of on-chip ALUs for vector and matrix calculations at near-peak performance levels throughout program execution. Unfortunately, the hardware complexity caused by the added ALUs and other components greatly increases the scheduling difficulty of many of these architectures. This in turn hinders system performance because it is difficult to prevent runtime stalls. Furthermore, while architectural enhancements such as caches, branch predictors, and prefetchers go a long way in improving performance, they do not constrain worst-case performance. Summary of the invention

[0007] Embodiments of the present disclosure relate to an integrated circuit having one or more deterministic processors (e.g., tensor stream processors (TSPs) or artificial intelligence processors), each of which has a functional slice architecture. In some embodiments, each deterministic processor is configured to process a machine learning model. Each deterministic processor is divided into a plurality of functional units, which are organized into a plurality of functional slices. Each functional slice is configured to perform a specific function within the deterministic processor, which may include a memory functional slice (MEM) for storing operand data, an arithmetic functional slice for performing operations (e.g., vector processing, matrix manipulation) on received operand data, etc. The functional unit of the deterministic processor is configured to stream operand data across a first dimension (e.g., a time dimension) in a direction indicated in a corresponding instruction, and receive instructions across a second dimension (e.g., a spatial dimension). The compiler of the deterministic processor knows the hardware configuration of the processor, and configures the timing of the data stream and the instruction stream so that the corresponding data and instructions cross at each computing element at a predetermined time. Each functional slice of the deterministic processor can operate on a data channel set in a single instruction multiple data (SIMD) manner. This set of data channels may be referred to herein as a "hyperchannel" and represents a cross-section of all functional slices on the processor chip.

[0008] Embodiments of the present disclosure relate to an integrated circuit. The integrated circuit includes a first die and a second die, the second die being connected to the first die to form a tile structure. The first die is offset by a first offset along a first dimension relative to the second die, and is offset by a second offset along a second dimension orthogonal to the first dimension. The tile structure is configured as a single-core processor operating as a model parallelism for the first die and the second die across the tile structure.

[0009] Embodiments of the present disclosure also relate to an integrated circuit comprising an array of block structures. Each block structure in the array comprises a first die and a second die connected to the first die in a face-to-face (F2F) configuration. The first die is offset by a first offset relative to the second die along a first dimension, and is offset by a second offset along a second dimension orthogonal to the first dimension, thereby forming an offset alignment between the first die and the second die. In some configurations, the first offset can be up to 50% of the first size of the first die (or the second die) along the first dimension, and / or the second offset can be up to 50% of the second size of the first die (or the second die) along the first dimension. The array of block structures is configured to operate as a single-core processor for model parallelism across block structures.

[0010] Embodiments of the present disclosure also relate to an integrated circuit including a stacked block structure. The integrated circuit includes a first plurality of block structures coupled to a first side of a block-to-block (T2T) bridge, and a second plurality of block structures coupled to a second side of the T2T bridge opposite to the first side. Each of the first plurality of block structures and the second plurality of block structures includes a first die and a second die connected in an F2F configuration. The first die is offset by a first offset along a first dimension relative to the second die, and is offset by a second offset along a second dimension orthogonal to the first dimension. The integrated circuit also includes a first stack of a high bandwidth memory (HBM) coupled to the first side, a second stack of HBM coupled to the second side, and a heat sink coupled to the outer surface of the first plurality of block structures and the first stack of HBM. In some embodiments, the integrated circuit includes a cuboid structure of a block structure. The block structure of the cuboid structure is configured to operate as a single-core processor for model parallelism across the block structure.

[0011] Embodiments of the present disclosure also relate to a process (or method) for performing computations using one or more deterministic streaming processors (e.g., tensor streaming processors) of an integrated circuit. The process includes: initiating the issuance of instructions for execution by processing units (e.g., computing elements of one or more functional slices) of multiple dies across one or more block structures of the integrated circuit; initiating the streaming of data through the processing units of multiple dies across one or more block structures to execute instructions; and returning the resulting data to one or more memory slices of one or more block structures.

[0012] Embodiments of the present disclosure also relate to a non-transitory computer-readable storage medium including computer-executable instructions stored thereon that, when executed by a compiler running on at least one computer processor, cause the at least one computer processor to: initiate issuing instructions for execution by processing units of multiple dies (e.g., computing elements of one or more functional slices) across one or more block structures of an integrated circuit, initiate streaming of data through the processing units of the multiple dies across the one or more block structures for instruction execution, and initiate returning resulting data to one or more memory slices of the one or more block structures.

[0013] Embodiments of the present disclosure also relate to a die-to-die (D2D) densely packaged integrated circuit implemented as a deterministic streaming processor (e.g., a tensor streaming processor). The integrated circuit includes a plurality of dies connected in a D2D configuration, and each die includes a corresponding deterministic streaming processor. The integrated circuit may include a first die and a second die, the second die being connected to the first die in a D2D configuration via a D2D interface circuit to form a D2D structure with the first die. The D2D interface may connect a first plurality of super channels (i.e., a group of first streaming data channels) of the first die to a second plurality of super channels (i.e., a group of second streaming data channels) of the second die for streaming data between the first die and the second die in a first direction or in a second direction orthogonal to the first direction. The D2D structure is configured to be used as a single-core processor for model parallelism of the first die and the second die across the D2D structure.

[0014] Embodiments of the present disclosure also relate to a process (or method) for computing using one or more deterministic streaming processors (e.g., tensor streaming processors) of an integrated circuit. The process (e.g., performed by a compiler) includes: initiating the issuance of multiple instructions for execution by multiple processing units (e.g., computing elements of one or more functional slices) across a first die and a second die, the second die being connected to the first die in a D2D configuration via a D2D interface circuit to form a D2D structure with the first die; and initiating streaming of data between a first plurality of super channels of the first die and a second plurality of super channels of the second die via a D2D interface circuit along a first direction or a second direction orthogonal to the first direction to execute multiple instructions.

[0015] Embodiments of the present disclosure also relate to a non-transitory computer-readable storage medium, which includes computer-executable instructions stored thereon, which, when executed by a compiler running on at least one computer processor, cause the at least one computer processor to: initiate issuance of multiple instructions for execution by multiple processing units (e.g., computing elements of one or more functional slices) across a first die and a second die, the second die being connected to the first die in a D2D configuration via a D2D interface circuit to form a D2D structure with the first die; and initiate streaming of data between a first plurality of super channels of the first die and a second plurality of super channels of the second die via the D2D interface circuit along a first direction or a second direction orthogonal to the first direction to execute the multiple instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A An arrangement of functional slices in a tensor streaming processor (TSP) is shown according to some embodiments.

[0017] Figure 1B An example TSP architecture is shown in accordance with some implementations.

[0018] Figure 1C The organization and data flow within a row of a TSP according to some embodiments is shown.

[0019] Figure 2A An example tile structure is shown in accordance with some implementations.

[0020] Figure 2B According to some embodiments, Figure 2A An example die of the block structure in FIG.

[0021] Figure 2C An example block structure with a block-to-block (T2T) bridge is shown in accordance with some implementations.

[0022] Figure 2D An example block structure with high bandwidth memory (HBM) is shown in accordance with some embodiments.

[0023] Figure 3 An example data flow within a block structure is shown according to some implementations.

[0024] Figure 4 An example of a two-dimensional array of block structures for implementing various multi-die processor architectures according to some embodiments is shown.

[0025] Figure 5A An example of a pair of dies connected in a block configuration is shown in accordance with some implementations.

[0026] Figure 5B According to some embodiments, a Figure 5A An integrated circuit in which multiple dies are interconnected in a block structure configuration.

[0027] Fig. 6A Another example of a pair of dies connected in a block configuration is shown in accordance with some embodiments.

[0028] Figure 6B According to some embodiments, a Fig. 6A An example integrated circuit of multiple dies interconnected in a block structure configuration.

[0029] Fig. 7A Example top and bottom views of an integrated circuit including multiple block structures interconnected via T2T bridges are shown according to some embodiments.

[0030] Figure 7B Another example top and bottom views of an integrated circuit including multiple block structures interconnected via T2T bridges are shown in accordance with some embodiments.

[0031] Fig. 8A An example integrated circuit including a stack of block structures and HBMs interconnected via T2T bridges is shown in accordance with some embodiments.

[0032] Figure 8B An example integrated circuit including a stack of block structures and HBMs interconnected via T2T bridges and a heat sink is shown in accordance with some embodiments.

[0033] Fig. 9 An example integrated circuit of a cuboid structure implemented as a block structure is shown in accordance with some implementations.

[0034] Fig.10 is a flow chart illustrating a method of using an integrated circuit for data processing with model parallelism across multiple dies of one or more block structures, according to some embodiments.

[0035] Fig.11 An example die-to-die (D2D) architecture is shown having two deterministic stream processors (or dies) connected in a D2D configuration in accordance with some embodiments.

[0036] Fig.12 An example D2D structure with a superlane extending across multiple dies is shown in accordance with some embodiments.

[0037] Fig.13A An example D2D structure is shown with three dies connected in a D2D configuration in accordance with some embodiments.

[0038] Fig. 13B An example D2D structure with dies connected in a D2D folded mesh configuration is shown in accordance with some embodiments.

[0039] Fig. 13C An example D2D structure with dies connected in a D2D ring configuration is shown in accordance with some embodiments.

[0040] Fig.14 An example D2D mapping structure is shown for D2D mapping of a super channel between a pair of dies in accordance with some embodiments.

[0041] Fig.15A An example die is shown having a first number of super channels and D2D interfaces in accordance with some embodiments.

[0042] Fig. 15B An example die is shown having a second number of super channels and a D2D interface in accordance with some embodiments.

[0043] Fig. 15C An example die with a third number of super channels and a D2D interface is shown in accordance with some embodiments.

[0044] Fig.16 is a flow chart illustrating a method of using an integrated circuit to perform data processing with model parallelism across multiple dies connected in a die-to-die structure according to some embodiments.

[0045] Fig.17A is an abstract diagram of an example computer system suitable for enabling implementations of the claimed disclosure to be used commercially, according to some implementations.

[0046] Fig. 17B is another abstract diagram of an example computer system suitable for enabling implementations of the claimed disclosure to be used commercially, in accordance with some implementations.

[0047] Fig.18 Additional example computing machines for use in commerce are shown in accordance with some embodiments.

[0048] The accompanying drawings depict embodiments of the present disclosure for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles of the present disclosure described herein or the benefits indicated. DETAILED DESCRIPTION

[0049] The drawings and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will readily be considered as viable alternatives that can be employed without departing from the principles claimed.

[0050] Reference will now be made in detail to a number of embodiments, examples of which are shown in the accompanying drawings. Note that, where feasible, similar or like reference numerals may be used in the drawings and may indicate similar or like functions. The drawings depict embodiments of the disclosed systems (or methods) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0051] Introductory Overview

[0052] Disclosed is a configuration of an integrated circuit including one or more deterministic processors (e.g., a tensor streaming processor (TSP) or an artificial intelligence processor). Each deterministic processor may have a functional slice architecture. In some embodiments, each deterministic processor is configured to process a machine learning model. Each deterministic processor is divided into a plurality of functional units. These functional units are organized into a plurality of functional slices. Each functional slice is configured to perform a specific function within the deterministic processor. The deterministic processor may include a memory functional slice (MEM) for storing operand data, an arithmetic functional slice for performing operations (e.g., vector processing, matrix manipulation) on received operand data, and the like. The functional units of the deterministic processor are configured to stream operand data across a first (e.g., time) dimension and receive instructions across a second (e.g., space) dimension in the direction indicated in the corresponding instruction. The compiler of the deterministic processor knows the hardware configuration of the processor and configures the timing of the data flow and the instruction flow so that the corresponding data and instructions cross at each computing element at a predetermined time. Each functional slice of the deterministic processor may operate on a data lane set in a single instruction multiple data (SIMD) manner. This set of data channels may be referred to herein as a "hyperchannel" and represents a cross-section of all functional slices on the processor chip.

[0053] The disclosed embodiments relate to a deterministic streaming processor with a functional slicing architecture. In some embodiments, the deterministic streaming processor may include a tensor streaming processor (TSP) with a functional slicing architecture, which may be used for hardware accelerated machine learning (ML) applications.

[0054] A deterministic stream processor (e.g., TSP) includes multiple "computing elements", each of which corresponds to a functional unit within the processor. The on-chip memory and on-chip network (NoC) of the processor architecture are fused to provide storage for both operands and results, and can act as a pipeline for transferring operands and / or result data to / from the functional units of the processor. The computing elements of the deterministic stream processor are divided into different functions (e.g., memory, arithmetic operations, etc.) and are organized as functional slices that operate on multidimensional data (e.g., tensors). For example, each functional slice consists of computing elements that are horizontally and vertically adjacent to each other (or adjacent) to form a functional slice. The number of computing elements and the computational granularity of each computing element can be selected to take advantage of the underlying technology on which the computing element is built. The number of computing elements (N) and the word granularity (M) of the memory (e.g., static random access memory (SRAM)) together generate the vector length (VL) of the machine.

[0055] In some embodiments, each functional slice of a deterministic stream processor operates independently and receives instructions from an instruction control unit (ICU). The ICU can pass instructions to a first computing element of a functional slice, which is then propagated along the functional slice to the remaining computing elements of the functional slice on a first time dimension of the processor. On the other hand, data operands for storage and / or processing can be transferred between different functional slices of a deterministic stream processor on a second spatial dimension of the processor perpendicular to the first time dimension. Therefore, the data stream and instruction stream of a deterministic stream processor are separated from each other.

[0056] In some embodiments, the compiler of the deterministic stream processor knows the hardware configuration of the deterministic stream processor, and synchronizes the timing of the data stream and the instruction stream so that the corresponding data and instructions are received at each computing element with a predetermined time relationship (e.g., during the same clock cycle, separated by a predetermined delay, etc.). In some embodiments, the predetermined time relationship can be based on the hardware of the deterministic stream processor, the instruction type, etc. Since the time relationship between the data and the instruction is known to the compiler, the operand data received by the computing element does not include any metadata indicating the purpose of the data. Instead, each computing element receives an instruction and executes the instruction on the corresponding data based on a predetermined timing. This allows data and instructions to flow through the deterministic stream processor more efficiently.

[0057] Embodiments of the present disclosure relate to an integrated circuit. The integrated circuit includes a first die forming a block structure and a second die connected to the first die. The first die is offset by a first offset along a first dimension relative to the second die, and is offset by a second offset along a second dimension orthogonal to the first dimension. The block structure is configured to operate as a single-core processor for model parallelism across the first die and the second die of the block structure.

[0058] Embodiments of the present disclosure also relate to an integrated circuit comprising an array of block structures. Each block structure in the array comprises a first die and a second die, the second die being connected to the first die in a face-to-face (F2F) configuration. The first die is offset by a first offset relative to the second die along a first dimension, and is offset by a second offset along a second dimension orthogonal to the first dimension, thereby forming an offset alignment between the first die and the second die. The array of block structures is configured to operate as a single-core processor for model parallelism across block structures.

[0059] Embodiments of the present disclosure also relate to an integrated circuit including a stacked block structure. The integrated circuit includes a first plurality of block structures coupled to a first side of a block-to-block (T2T) bridge and a second plurality of block structures coupled to a second side of the T2T bridge opposite to the first side. Each of the first plurality of block structures and the second plurality of block structures includes a first die and a second die connected in an F2F configuration. The first die is offset by a first offset along a first dimension relative to the second die, and is offset by a second offset along a second dimension orthogonal to the first dimension. The integrated circuit also includes a first stack of a high bandwidth memory (HBM) coupled to the first side, a second stack of the HBM coupled to the second side, and a heat sink coupled to the outer surface of the first plurality of block structures and the first stack of the HBM. The stacked block structure (e.g., stacked in a cuboid structure) is configured to operate as a single-core processor for model parallelism across the block structure.

[0060] Architectural overview of the Tensor Streaming Processor

[0061] According to an embodiment of the present disclosure, the processor plane includes a TSP, for example, commercially available from GROQ, Inc. of Mountain View, California. It should be understood that although many of the embodiments described herein use the TSP as the preferred processor, other deterministic processors may also be used in commercial applications. The accompanying drawing (Figure) 1A shows the arrangement of functional slices in a TSP according to some embodiments.

[0062] Certain core architecture elements set up the TSP separately from the GPU and accelerators. In a conventional chip multiprocessor (CMP), each "compute element" is an independent core that is interconnected using an on-chip network to exchange data between cores. Instruction execution is performed in the following stages: (i) instruction fetch (IF), (ii) instruction decode (ID), (iii) execute (EX) on the arithmetic logic unit (ALU), (iv) memory access (MEM), and (v) write back (WB) to update the result in the general purpose register (GPR).

[0063] Compared to conventional multicores (where each computational element is a heterogeneous collection of functional units but globally homogeneous), TSP inverts this to have local functional homogeneity but chip-wide (global) heterogeneity. More specifically, TSP reorganizes the homogeneous two-dimensional grid of cores into Figure 1A The microarchitecture of functional slices is shown in Figure 1. In this approach, each computing element implements a specific function and is stacked vertically into specific "functional slices" in the Y dimension of the two-dimensional on-chip grid. TSP decomposes the basic elements of a conventional multi-core according to their respective functions as follows: instruction control and dispatch (e.g., via the instruction control unit (ICU)), memory (MEM), integer (INT) arithmetic, floating point unit (FPU) arithmetic, and network (NET) interface, as described in Figure 1A Each row of the 2D on-chip grid contains cross sections of all functional slices.

[0064] In this organization, each functional slice is independently controlled by an instruction sequence specific to its on-chip role. For example, the MEM functional slice supports read and write, but not necessarily addition or multiplication, which are usually performed in arithmetic functional slices (e.g., vector execution module (VXM) and matrix execution module (MXM) functional slices) for some typical machine learning (ML) algorithms such as linear regression algorithms.

[0065] The compute elements of all functional slices execute the same instruction stream - single instruction multiple data (SIMD) instructions. Therefore, the common instruction decode and dispatch logic can be broken out into its own compute element (e.g., ICU), and the normal instruction execution pipeline is broken into two areas: (i) instruction fetch, decode, and pack; and (ii) operand read, execute, and write back. This approach decouples the memory subsystem from the functional units that retrieve their operands and store their results.

[0066] In some embodiments, each functional slice implements, for example, a 20-stage vector pipeline that spans the computing elements of each functional slice, wherein each computing element produces 16 elements of a maximum vector length of 320 elements. When the data stream passes through different functional types, this organization naturally decomposes the instruction stream in the vertical dimension and decomposes the data stream in the horizontal dimension. With this processor organization, instruction execution is performed by different computing elements: when the (vertically flowing) dispatched instruction intersects with the (horizontally flowing) operand data on which the dispatched instruction is operating, the instruction fetching and decoding in the ICU, and the operand decoding, execution and write-back at each computing element of the functional slice. It should be understood that references to "vertical" and "horizontal" or "north", "south", "east" and "west" are used in conjunction with the diagrams shown in the accompanying drawings, are abstractions intended only to help the reader, and should not be inferred as technical limitations.

[0067] Figure 1B An example TSP 100 according to some embodiments is shown. The TSP 100 may include memory and arithmetic units that are optimized for multiplying and adding input data with a set of (e.g., trained or being trained) weights for a machine learning application (e.g., training or reasoning). For example, the TSP 100 includes a VXM 110 for performing operations on vectors (i.e., one-dimensional arrays of values). The other elements of the system are symmetrically arranged on either side of the VXM 110 to optimize processing speed. For example, the VXM 110 is adjacent to the MEMs 111 to 112, the SXMs 113 to 114 to control data routing, data domains and presentation controllers (or numerical interpretation modules (NIMs)) 115 to 116 and the MXMs 117 to 118. For example, the ICU 120 controls the flow of data across blocks 110 to 118 and the execution of operations. The TSP 100 may further include communication circuits such as chip-to-chip (C2C) circuits 123 to 124 and external communication circuits (e.g., PCIe) 121. For example, the TSP 100 may further include a chip control unit (CCU) 122 to control boot operations, clock resets, and other low-level setup operations.

[0068] Figure 1C 1 shows the organization and data flow within a row of a TSP 100 according to some embodiments. Figure 1CAs shown, each row of the two-dimensional on-chip grid of the TSP 100 contains a cross-section of all functional slices, such as MXM (e.g., N=320) configured for NxN arrays of both integer (INT) and floating point (FP) values ​​(e.g., INT8 and FP16), S MEM functional slices (S=44), VXM functional slices with V vector ALUs per channel (e.g., V=16), and SXM functional slices. In this organization, each functional slice is independently controlled by an instruction sequence specific to its on-chip role taken from a corresponding ICU array (e.g., I=144 ICUs in total). Conceptually, the functional slices are fixed, and data 130 flows across the computational elements of the functional slices. As data flows through a particular functional slice, each functional slice can optionally intercept data operands and compute results (e.g., in the case of MXM and VXM), or move data between data transmission channels on the network (e.g., in the case of SXM and MEM). Instructions flow northward from the ICU to the functional slices, while data (operands and results) primarily flows east and west between functional slices. Any inter-lane data movement within a vector uses the on-chip network to the functional slices.

[0069] Note that the "East-West-North-South" directionality is provided herein for ease of discussion and relativity. Furthermore, the "East-West-North-South" directionality is used as a reference for explaining the process flows described herein and is not intended to be limited to labels of specific directions. For example, North-South may be redirected as East-West, and the principles currently described with respect to East-West may apply to the redirected North-South. In another example of a directionality that is not intended to be limited to the description according to the noted reference, the directionality may be referenced such that North-South is up-down and East-West is right-left, and the principles will apply accordingly.

[0070] In one embodiment, 320 channels are superimposed on the TSP 100, where each computing element in the on-chip grid operates on, for example, 16 channels in a SIMD manner. The 16-channel unit may be referred to herein as a "super channel" and represents a cross-section of all functional slices on the chip. Therefore, a super channel may represent the minimum vector length (minVL) of an architecture of, for example, 16 elements. Similarly, a vertical combination of 20 blocks forming a functional slice may produce, for example, a maximum vector length (maxVL) of 20x16=320 functional units. Each of the 144 independent on-chip ICUs may issue one or more instructions per clock cycle. The compiler has explicit control over the program order in each instruction queue, for example, by generating an assembler 140 to be executed by the ICU and the functional slice. Each channel has 64 logical flows for moving operands or results on the chip with, for example, 32 flows to the east and 32 flows to the west. 220MB of globally shared SRAM may deliver 32 bytes of stream bandwidth per channel and low latency access to model parameters. For example, in less than 30 clock cycles (including SRAM and on-chip network transmission delays), the MEM can read and the MXM can install more than, for example, 100,000 weights into a 320x320 array (i.e., 320 channels x 320 functional units).

[0071] like Figure 1B and Figure 1C As shown in , the on-chip network is implemented as an X-dimensional grid and a Y-dimensional grid of computing elements with XYX dimensional sequential routing. Each instruction specifies the first hop direction (east or west), so the memory instruction semantics have both address and data flow directions (see Figure 1C ). The stream is routed in the X dimension through MEM 111 / 112 and in the Y dimension using the SXM's 113 / 114 permuters and channel shifters to move data elements vertically. The SXM's 113 / 114 permuters implement a permutation function, which is a mathematical technique for determining the number of possible arrangements in a set when the order of the arrangements is important. Common mathematical problems involve selecting only a few items from a collection of items in a certain order.

[0072] MEM 111 / 112 and SXM 113 / 114 provide deterministic routing of streaming data as it flows in the X and Y dimensions, respectively. Using the TSP architecture 100, functional slices interact with streams of data in a producer-consumer fashion. That is, functional slices consume operands from a stream and produce results onto a (possibly different) stream, like assembly line operators (functional slices) and conveyor belts (streams).

[0073] Conceptually, the functional slices are fixed, and data flows across the computing elements, e.g. Figure 1CAs data flows through the functional slice, each computational element may optionally intercept data operands and compute results (if the computational element includes an arithmetic logic unit (ALU)), or move data between channels on the network if the computational element includes a switching element.

[0074] Streams provide a programming abstraction and are the conduits through which data flows between functional slices. Unlike GPRs, functional slices operate on parallel streams of data that flow east or west (horizontally) across the chip. Horizontally flowing streams carrying operands intercept vertically (northward) flowing instructions (see Figure 1C ) to perform computations at the compute elements of a functional slice. The compiler maintains the architectural state of the chip precisely and uses that knowledge to ensure that instructions correctly intercept their stream operands.

[0075] Streams are implemented in hardware through a chip-wide stream register file. Streams are architecturally visible and transfer operands and results between functional slices. A common software pattern involves reading operand data from one or more MEM functional slices, which is then subsequently consumed and operated on by downstream arithmetic functional slices. The results of the operation are then produced on another stream so that they can be written back to memory or passed to subsequent computational elements. For example, a Z=X+Y operation may require four instructions: read S1,X and read S2,Y are executed on two MEM functional slices, and point inward to the ALU functional slice to perform addition S1, S2, S3. Finally, the result is stored back to memory by writing S3,Z. A stream represents a set of N elements, which is operated on in a SIMD manner by each functional slice.

[0076] For example, the TSP architecture makes some intentional tradeoffs at the hardware-software interface, pushing the complexity associated with scheduling into the compiler. Specifically, the compiler needs to accurately schedule instructions to use the hardware correctly and efficiently. Sometimes this may involve choosing one of several ways that an algorithm or meta-operation can be implemented in hardware. Eliminating the control complexity of dynamic instruction scheduling for multi-issue execution units enables the ICU to be relatively small, for example, less than 3% of the chip area.

[0077] The compiler accesses, for example, a 320-lane programming abstraction overlaid on the TSP architecture, where each computational element in the on-chip grid operates on 16 lanes in a SIMD fashion. A unit of 16 lanes may be referred to as a "super-channel," which is the smallest granularity of cross-section and computation of all functional slices on the chip. Thus, a super-channel represents the minimum vector length minVL of the architecture of 16 elements. Likewise, the vertical combination of the 20 blocks forming the functional slice yields a maximum vector length maxVL of 20x16=320 elements.

[0078] The compiler accesses, for example, 144 independent on-chip instruction queues (i.e., ICUs): (a) 6 MXMs for westbound instruction queues including two independent two-dimensional MAC (multiply-accumulate) arrays; (b) 14 SXMs for westbound instruction queues for switching within and between superchannels by rearranging elements of vectors; (c) 44 MEMs for westbound instruction queues including 44 parallel functional slices of static random access memory (SRAM); (d) 16 VXMs for eastbound instruction queues including 16 vector ALUs per channel; (e) 44 MEMs for eastbound instruction queues including 44 parallel functional slices of SRAM; (f) 14 SXMs for eastbound instruction queues; and (g) 6 MXMs for eastbound instruction queues including two independent two-dimensional MAC arrays, and each instruction queue can issue one or more instructions per cycle, and the compiler has explicit control over the program order in each instruction queue.

[0079] The compiler accesses, for example, 64 logical streams per channel. For example, 32 logical streams are required to operate on 16minVL per channel for moving operands or results on the chip with 32 streams to the east and 32 streams to the west.

[0080] The compiler accesses, for example, 220 megabytes of globally shared SRAM, which delivers 32 bytes of streaming bandwidth per channel and low latency access to model parameters. For example, in less than 40 operation cycles (including SRAM and on-chip network transmission delays), the MEM can read and the MXM can install 400K weights into all 4 320x320 arrays.

[0081] Flows are specified by both an identifier (0, ..., 31) and a direction. For example, in(28) specifies flow 28 inward, and out(24) specifies flow 24 toward the outer edge of the chip. The direction of the flow can be specified as inward (toward the chip bisection) or outward (toward the outer edge of the chip), or the direction can be specified as east or west, as in Figure 1C as shown in .

[0082] Spatially organize the components of the superchannel, such as Figure 1C The instruction set architecture (ISA) of TSP defines instructions that span different functional areas. The partitioned global address space (PGAS) presented by the MEM functional slice provides memory semantics for vectors that are addressed from SRAM and loaded into an architecturally visible stream in the direction of data flow towards the functional slice that intends to operate on the vector.

[0083] The first functional region (i.e., ICU) provides inter-slice synchronization using explicit instruction fetching with the IFetch instruction, and chip-wide barrier synchronization between participating functional slices using the Sync and Notify instructions. Repeated NOP (no-op) instructions allow precise cycle-by-cycle control of inter-instruction delays. For example, the compiler has cycle-accurate control when scheduling two operations A and B with an intermediate NOP so that N cycles separate them (e.g., OpA NOP (N) OpB).

[0084] The second functional area, VXM, consists of a 4x4 grid of ALUs in each channel for point-by-point arithmetic operations.

[0085] The third functional area (ie, MXM) consists of four independent two-dimensional MAC arrays that operate on, for example, INT8 or FP16 data types.

[0086] On-chip data movement uses the fourth functional area, SXM, for intra-superchannel and inter-channel switching by rearranging the elements of the vector. SXM is similar to Figure 1A NET interface for communication between cores. MEM and SXM work together to form the XY dimension of the on-chip network.

[0087] The fifth functional area (i.e., the east and west hemispheres of the on-chip MEM module) includes 44 parallel MEM functional slices of SRAM and provides the memory access concurrency required to fully utilize 32 streams in each east or west direction. Each functional slice provides 13-bit physical addressing of 16-byte memory words, and for a total of 220 megabytes of on-chip SRAM, each byte is mapped to a channel.

[0088] The additional sixth functional area includes a C2C module configured to provide send and receive primitives for exchanging 320-byte vectors between a pair of TSP chips. A possible TSP implementation has, for example, a total of 16x 4 links operating at 30 Gbps, each for off-chip pin bandwidth, 16x 4x 30 Gbps x 2 directions = 3.84 Tb / s (terabytes per second) of total off-chip bandwidth, which can be flexibly divided to support the high-radix interconnect network of TSP for large-scale systems. The host interface of Peripheral Component Interconnect Express (PCIe) Gen4 can also be processed in this module. The host interface provides a lightweight direct memory access (DMA) engine to place the model on the TSP memory and provides an entry point for starting model execution. The host interface also provides a general mechanism for delivering interrupts to the host, which may be necessary, for example, if a multi-bit memory error is observed.

[0089] Table I provides a summary of example instructions for each functional slice according to some embodiments.

[0090] Table I

[0091] Summary of instructions for each functional slice

[0092]

[0093]

[0094] Sequences of instructions executed on different functional slices can be chained to create more complex actions without the need to write intermediate results back to memory. This allows efficient processing of streams at full bandwidth and minimal latency.

[0095] Machine learning algorithms typically operate on vectors of coefficients with specified data types (e.g., INT8, FP16, etc.). These vectors can be interpreted as an abstraction over the underlying data, whose elements can be processed in a SIMD manner by the same operations. TSP operates on vectors that are sometimes organized into rank-2 tensors, and relies on a graph-lowering compiler to transform higher-rank tensors into rank-2 tensors.

[0096] The programming model of TSP is a producer-consumer model in which each functional slice acts as a consumer and producer of one or more streams. When a vector is read from main memory, it is given a stream identifier (0,...,31) and a direction: east or west. Once a vector is read into a stream register, it is a stream and "flows" in the following sense: given a vector at coordinates x0,x 1, The spatially adjacent functional slices at x2 (where the spatial coordinates increase in the flow direction) are then i At t, the vector representing stream s1 at functional slice x1 can be accessed as an operand by the functional slice. Similarly, functional slices at x0 and x2 will access different stream values ​​of the same stream register. In the subsequent cycle t i+1 In this case, the value s1 is either propagated to the function slice at x2 or overwritten with the result r1 produced by the function slice at x1 at cycle t. Similarly, there exists a i The stream value s0 to be consumed by the function slice at coordinate x0 will be consumed in the next cycle t i+1 can be used for function slicing at x1 (without x0 overwriting time t i Stream operands are directed towards functional slices that consume them and produce streams of results. Streams flow continuously across the chip, serving as a way for functional slices to communicate with each other.

[0097] In the TSP programming model, at a given compiler scheduling time t, instructions are issued on a functional slice, and the operand vectors supplied by the stream (e.g., operand vectors supplied by the stream of up to 320 elements) are SIMD operations performed to generate vectors of the same length on the result stream. For example, at the microarchitecture level, the vertical stacking of the computing elements in the 320-element SIMD instructions across the functional slice is pipelined. That is, at the scheduling time t, the instruction will be issued to the bottommost computing element in the functional slice (e.g., corresponding to the first 16-element superchannel of the operand / result vector). In the subsequent operation cycle, the instruction will be propagated to the next computing element to the north in the functional slice, which in turn executes the instruction on the next 16-element superchannel of the operand vector. The process continues cycle by cycle until the process traverses all 20 computing elements in, for example, the functional slice. The combination of the above-mentioned vertical instruction pipelines and the need for the operands and instructions to coincide at precise times produce the spatial "interleaving" of SIMD operands and result data.

[0098] Block structure of multi-die processors

[0099] Embodiments of the present disclosure relate to a tile structure for implementing a multi-die device. The tile structure may include a first array of processing units on a first die, which is connected to a second die with a second array of processing units in a face-to-face (F2F) configuration. The tile structure may also include a specific interface that allows connection with one or more other tile structures (i.e., other dies) to implement various multi-die devices with multiple processors or a multi-die single processor (e.g., a TSP commercially available from GROQ) in some embodiments. The tile structure presented herein allows efficient coupling of multiple blocks without using any interface based on an interposer (e.g., a silicon-based interposer). In some other embodiments, the tile structure presented herein includes a pair of artificial intelligence (AI) processors connected in an F2F configuration, a face-to-back (F2B) configuration, or a back-to-back (B2B) configuration.

[0100] Figure 2A 2 shows an example block structure 200 according to some embodiments. The block structure 200 includes a die 202 and a die 204 connected to the die 202 to form the block structure 200. Figure 2AAs shown, die 202 and 204 in block structure 200 are positioned in a stacked offset configuration. Die 204 is a bottom die having a bottom extension that extends outward from below block structure 200 and may be referred to as a "frame," while die 202 is a top die having a top extension that extends outward from above one edge of die 202 and may be referred to as a "flange." Die 202 is connected to die 204 in an F2F configuration forming block structure 200. Details on the F2F configuration forming block structure 200 will be described below in conjunction with Figure 3 Give a description.

[0101] The die 202 (i.e., the portion of the die 202 where the flange is formed) can be offset relative to the die 204 (i.e., the portion of the die 204 where the frame is formed) by a first offset along a first dimension (e.g., an x-dimension or horizontal dimension) and offset by a second offset along a second dimension (e.g., a y-dimension or vertical dimension) orthogonal to the first dimension. The first offset can be equal to or different from the second offset. The die 202 can include a first TSP having a first plurality of functional units (i.e., functional slices), and the die 204 can include one of the following: a second TSP having a second plurality of functional units (i.e., functional slices), a memory device (e.g., an HBM), an interface chip, some other chip, or some combination thereof. In a preferred embodiment, the block structure 200 forms a single streaming processor (i.e., a single-core TSP) having a processing unit (i.e., a functional slice) on each die 202, 204. In other embodiments, the block structure 200 can include multiple multi-core devices or other circuits. Figure 2A The block structure 200 and flange / frame configuration shown in enables additional block structures to be electrically connected to the block structure 200 .

[0102] Figure 2B An example die 202 of a block structure 200 according to some embodiments is shown. Die 202 may include a processing unit 206 and an interface circuit, such as a processing unit and an interface circuit of a TSP (e.g., TSP 100). Processing unit 206 may include functional slices of multiple TSPs. In one or more embodiments, die 204 of block structure 200 has the same structure as die 202 and forms a single-core TSP with die 202. In other words, block structure 200 may include a single-core TSP. In one or more other embodiments, die 204 includes an HBM, an interface chip, some other chip, or some combination thereof.

[0103] The interface circuit of the die 202 may include a first set of die-to-die (D2D) pins 208, a second set of D2D pins 210, a first set of block-to-block (T2T) pins 212, and a second set of T2T pins 214. The first D2D interconnect region including the first set of D2D pins 208 and the second D2D interconnect region including the second set of D2D pins 210 represent an interface region of the die 202 for interconnecting the die 202 in an F2F configuration (e.g., Figure 3 2D interconnect region 208 and 210 are connected to the die 204 to form the block structure 200. In one or more embodiments, the first D2D interconnect region and the second D2D interconnect region are coupled to the die 204 via at least a subset of the D2D pins 208, 210 ultrasonically bonded to corresponding connection circuits (e.g., D2D pins) of the die 204. In one or more other embodiments, the first D2D interconnect region and the second D2D interconnect region are coupled to the die 204 by forming an electrical connection in which at least a subset of the D2D pins 208, 210 are placed in physical contact with corresponding connection circuits (e.g., D2D pins) of the die 204.

[0104] The flange area of ​​die 202 (and similarly, the shelf area of ​​die 204) may include a first set of T2T pins 212 and a second set of T2T pins 214 for connecting block structure 200 to one or more other block structures. The area around the T2T pins 212, 214 may form a flange area (or similarly, the shelf area of ​​die 204, which is the same structure as die 202) to enable block structure 200 and another block structure to be interconnected to form an electrical connection where their corresponding T2T pins are aligned. The T2T pins 212 and 214 may be sufficiently spaced apart to align with corresponding T2T pins of adjacent block structures and any desired die-to-die spacing (e.g., on the order of 0.1 mm to 0.5 mm). The flange area of ​​die 202 (and similarly the shelf area of ​​die 204) is divided into two regions, such that one region is along the "side" edge of die 202, and the other region is along the "top" edge of die 202 (or the "bottom" edge of die 204). Thus, block structure 200 can be coupled to a pair of block structures using the two flange areas of die 202 to form a tightly coupled three-structure device. Similarly, die 204 of block structure 200 can be coupled to another pair of block structures using corresponding T2T pins in the shelf area of ​​die 204.

[0105] Figure 2CAn example block structure 220 with a T2T bridge 225 is shown in accordance with some embodiments. The block structure 220 has the same configuration as the block structure 200. In one or more embodiments, the flange and / or rack die of the block structure 220 form a direct T2T connection to a shared memory (e.g., HBM) via the T2T bridge 225, which is used to store data passed from an adjacent block structure. The T2T bridge 225 can be positioned along one side of the block structure 220, for example, on the available space on the top of the rack (i.e., the bottom die) of the block structure 220. Alternatively ( Figure 2C 2 ), the T2T bridge 225 may be located along some other side of the block structure 220 (e.g., along the bottom side of the block structure 220). The T2T bridge 225 may include a T2T pin 230 for connecting to an adjacent block structure. The T2T bridge 225 may include connection pads with the T2T pin 230 on one or both sides of the T2T bridge 225 for connecting to one or both dies of an adjacent block structure.

[0106] In some embodiments, a T2T bridge 225 is used to deliver power from one die of a block structure 220 to another die of an adjacent block structure. In this case, mirror imaging may be required between the flange die and the rack die of the block structure 220, for example, to achieve power supply VDD and VSS connections. The T2T pins 230 of the T2T bridge 225 are preferably positioned in an area near the edge of the T2T bridge die. For example, an area along the edge of the T2T bridge die may include a first subset of the T2T pins 230, and an area along the top or bottom of the T2T bridge die may include a second subset of the T2T pins 230. The T2T pins 230 can form an electrical connection when connected to the pins of another block structure or die, and the electrical connection enables high-speed data transmission due to the low-ohmic connection formed thereby.

[0107] In one or more embodiments, Figure 2C As shown in FIG. 1 , at least one input-output (IO) die 235 is placed along one or more sides of the block structure 220 to facilitate connection to one or more external devices, such as one or more host computers, one or more sensors (e.g., cameras or other imaging devices), memory devices (e.g., HBM structures), or some other external devices. The IO die 235 may include an interconnect pin area in which interconnect pins 240 are placed to mate with corresponding T2T pins of the block structure 220 ( Figure 2C In some cases, due to limitations imposed by external devices, the IO die 235 may not be directly coupled to pins located in a shelf or flange of the block structure 220. In this case, a T2T bridge may be used to couple the IO die 235 to the block structure 220.

[0108] It should be noted that closely positioning the dies of adjacent block structures can provide improved thermal conductivity to remove heat from the top (and / or bottom) side of the connected block structures. The F2F block configuration presented herein represents a preferred method of ensuring that there is little to no gap between the dies of adjacent block structures. Having good uniform thermal conductivity facilitates efficient cooling and heat removal using, for example, a cooling system at the top side of the block structure. Figure 8B Provides more details on the cooling of the block structure.

[0109] Figure 2D An example block structure 250 coupled to a memory device 270 is shown according to some embodiments. Block structure 250 may include die 255 (e.g., a top die or flange) and die 260 (e.g., a bottom die or shelf). Block structure 250 may be an embodiment of block structure 200, i.e., die 255 may be connected to die 260 in a F2F configuration.

[0110] In some embodiments, at least a subset of the T2T pins of die 255 (and / or die 260) include one or more through silicon via (TSV) connectors 265. TSV connectors 265 may allow block structure 250 to be configured in an F2F configuration, an F2B configuration, or a B2B configuration (e.g., except Figure 4 In one or more embodiments, the other block structures coupled to the block structure (in a F2F configuration, a F2B configuration, or a B2B configuration) may include a TSP, a memory device (e.g., an HBM device), an interface device, some other device, or a combination thereof.

[0111] In some embodiments, Figure 2D As shown, a memory device 270 (eg, an HBM device) is placed on top of the block structure 250 and connected to the die 255 (ie, the flange) via a TSV connector 265. In other embodiments ( Figure 2D (not shown), the bridge die can provide connections from the pins on the die 255 (i.e., flange) or die 260 (i.e., shelf) positioned on the block structure 250 to an adjacent block structure or another device (e.g., HBM device) stacked on top of the block structure. FIG. 7A to FIG. 7B and FIG. 8A to FIG. 8B More details about stacked block structures and HBM devices in integrated circuits using bridge dies are described.

[0112] Figure 32 shows an example data flow within a block structure 300 according to some embodiments. The block structure 300 may be an embodiment of the block structure 200. The block structure 300 includes a pair of die 305A, 305B connected in a F2F configuration. An instruction control unit (ICU) integrated in each die 305A, 305B (e.g., as part of a corresponding D2D pin 310A, 310B, such as Figure 3 305A, 305B) can issue instructions for execution by one or more processing units (e.g., computing elements of one or more functional slices) in each die 305A, 305B. When an instruction is issued by the ICU, data flow is initiated through the processing units in each die 305A, 305B, and the resulting data is returned to the corresponding D2D pins 315A, 315B ( Figure 3 As shown in the side view 330 of the block structure 300 , the resulting data can be passed from one die to another die within the block structure 300 via the D2D pins 310A, 310B, thereby providing high-speed data communication between a pair of dies 305A, 305B of the block structure 300 .

[0113] Figure 4 An example of a two-dimensional array of block structures for implementing various multi-die processor architectures according to some embodiments is shown. In one embodiment, a deterministic streaming processor is represented in integrated circuit 405, which consists of a pair of dies connected in a F2F configuration to form a block structure (e.g., an embodiment of block structure 200). Various multi-die processors can be implemented by horizontally connecting two or more block structures.

[0114] like Figure 4 As shown, integrated circuit 410 includes a pair of block structures connected horizontally in a 1x2 block structure array. In this case, the top die of the first block structure can be connected to the bottom die of the second block structure (e.g., via corresponding T2T pins), and the bottom die of the first block structure can be connected to the top die of the second block structure (e.g., via corresponding T2T pins). This can be achieved by offsetting the top die and the bottom die relative to each other in each block structure so that the second block structure can fit into the available die area of ​​the first block structure. Figure 4 As also shown in FIG. 1 , the same process can be repeated multiple times in both the x-dimension and the y-dimension to implement other multi-die processors. Figure 4As shown, four block structures can be horizontally connected into a 2x2 array of block structures to form an integrated circuit 415; six block structures can be horizontally connected into a 2x3 array of block structures to form an integrated circuit 420; and eight block structures can be horizontally connected into a 2x4 array of block structures to form an integrated circuit 425. Each array of block structures (i.e., each integrated circuit 405, 410, 415, 420, 425) can be configured to function as a single core processor for model parallelism across die of block structures.

[0115] Figure 4 The purpose is to illustrate a scalable deterministic stream processor architecture, for example, for a TSP device. In other embodiments, the deterministic stream processor architecture may include multiple cores, and Figure 4 Various other variations for connecting block structures not shown in the figure are also possible, such as vertical connection of block structures on top of each other, a combination of vertical and horizontal connection of block structures, etc. In some embodiments, integrated circuit 405 can be referred to as a "single-core" integrated circuit. Similarly, integrated circuits 410, 415, 420, 425 can be referred to as "dual-core" integrated circuits, "quad-core" integrated circuits, "six-core" integrated circuits, and "octa-core" integrated circuits, respectively. It should be noted that the use of the term "core" is not limited to one or more processor cores. Instead, the use of "core" can simply represent the number of block structures of the same configuration (e.g., the configuration of block structure 200) included in a deterministic streaming processor.

[0116] Figure 5A A pair of dies 505A, 505B are shown as an example connected in a block structure 500 according to some embodiments. Die 505A can be offset by a first offset along a first dimension (e.g., x dimension) relative to die 505B, and offset by a second offset along a second dimension (e.g., y dimension) orthogonal to the first dimension, thereby forming an offset alignment between die 505A and die 505B. Block structure 500 can be configured to operate as a single-core processor for model parallelism across dies 505A, 505B. Block structure 500 can be an embodiment of block structure 200.

[0117] Each die 505A and 505B in the block structure 500 may include a TSP having an array of computing elements on a substrate (e.g., as part of a functional slice / memory slice). Alternatively, in place of the TSP, one of the dies 505A, 505B may include a memory device (e.g., an HBM), an interface device (e.g., a bridge die), or some other device. A plurality of D2D pins 510A, 510B may be positioned on each die 505A, 505B. It may be observed that each die 505A, 505B may be divided into four quadrants, and each quadrant in each die 505A, 505B may include a portion of D2D pins 510A, 510B for direct high-speed connection to a corresponding pair of D2D pins 510A, 510B in an adjacent die. As Figure 5A As shown, the lower right quadrant of die 505A is connected to the upper left quadrant of die 505B, for example, by positioning portions of D2D pins 510B of die 505A on top of corresponding portions of D2D pins 510A of die 505A (or vice versa), thereby providing high-speed data communication between die 505A and die 505B.

[0118] Figure 5A The direct connection of a pair of dies 505A, 505B shown in FIG. 5 can be extended to implement F2F connections between more than two dies. Figure 5B An integrated circuit 520 is shown having multiple dies (eg, 13 dies) interconnected using the configuration of the block structure 500 in accordance with some embodiments. Figure 5B As shown, each quadrant of die A can be connected via a corresponding portion of the D2D pin (such as Figure 5A 525(1), 525(2), 525(4), and 525(5). Thus, high-speed data communication via the D2D pins can be established between die 530A and each of die 525(1), 525(2), 525(4), and 525(5). Similarly, each quadrant of die 530B can be connected via a corresponding portion of the D2D pins (as shown). Figure 5A 525(2), 525(3), 525(5), and 525(6). Thus, high-speed data communication via the D2D pins can be established between die 530B and each of die 525(2), 525(3), 525(5), and 525(6). In addition, each quadrant of die 530C can be connected via a corresponding portion of the D2D pins (as shown). Figure 5A525(4), 525(5), 525(7), and 525(8). Thus, high-speed data communication via the D2D pins is established between die 530C and each of dies 525(4), 525(5), 525(7), and 525(8). Finally, each quadrant of die 530D is connected via a corresponding portion of the D2D pins (as shown in FIG. Figure 5A As shown in FIG. 1 , the die 530D is connected to the corresponding quadrants of the die 525(5), the die 525(6), the die 525(8), and the die 525(9). Therefore, high-speed data communication via the D2D pins is established between the die 530D and each of the die 525(5), the die 525(6), the die 525(8), and the die 525(9). Therefore, Figure 5B The integrated circuit 520 in the example includes 13 tube cores connected to each other via corresponding D2D pins for high-speed data communication. Each tube core 530A, 530B, 530C, 530D can be a TSP, a memory device (e.g., an HBM device), an interface device (e.g., a bridge tube core), or some other device. It should be noted that each tube core 530A, 530B, 530C, 530D, when implemented as an interface device (e.g., a bridge tube core), can be used to interface the integrated circuit 520 (e.g., via one or more quadrants of the corresponding tube core 530A, 530B, 530C, 530D) with a host computer, an HBM device, or some other device.

[0119] Fig. 6A An example pair of dies connected in a block structure 600 is shown according to some embodiments. Die 605A can be offset by a first offset along a first dimension (e.g., x dimension) relative to die 605B, and offset by a second offset along a second dimension (e.g., y dimension) orthogonal to the first dimension, thereby forming an offset alignment between die 605A and die 605B. Block structure 600 can be configured to operate as a single-core processor for model parallelism across dies 605A, 605B. Block structure 600 can be an embodiment of block structure 200.

[0120] Each die 605A and 605B in the block structure 600 may include a TSP having an array of computing elements on a substrate (e.g., as part of a functional slice / memory slice). Alternatively, instead of a TSP, one of the dies 605A, 605B may include a memory device (e.g., an HBM), an interface device (e.g., a bridge die), or some other device. A plurality of D2D pins 610A, 610B may be located on each die 605A, 605B. Similarly, as in Figure 5AIn the case of block structure 500 in block structure 600, a direct connection between die 605A and die 605B is achieved, for example, by positioning a portion of D2D pin 610B of die 605B on top of a corresponding portion of D2D pin 610A of die 605A (or vice versa).

[0121] Return to reference Figure 5A , it can be observed that Figure 5A Each die 505A, 505B in the embodiment can be divided into four quadrants of equal size, and each quadrant in each die 505A, 505B utilizes a corresponding portion of D2D pins 510A, 510B of uniform size for direct connection with a corresponding portion of pins 510A, 510B in an adjacent die. Figure 5A The block structure in 500 is different, Fig. 6A The block structure 600 in FIG. 5 is formed by connecting a pair of dies via portions of D2D pins 610A, 610B that are longer than in the case of the block structure 500. Fig. 6A As shown, the direct connection between the lower right quadrant of die 605A and the upper left quadrant of die 610A via D2D pins 610A, 610B is longer than block structure 500. Fig. 6A As can be observed in FIG. 6 , the rest of the D2D pin area in each die 605A, 605B is smaller, which means that fewer possible connections can be made and each die 605A, 605B is closer to the corresponding adjacent die ( Fig. 6A The connections between these portions of the region via D2D pins 610A, 610B (not shown) are smaller than the block structure 500. Due to the fewer D2D pins in these regions, each pin can be multiplexed so that multiple signals can be routed through each pin connection.

[0122] Figure 6B An example integrated circuit 620 is shown having multiple dies (eg, five dies) interconnected using the configuration of the block structure 600. Figure 6B As shown, die 630 is directly connected to die 625(2) and 625(4) via corresponding D2D pins in a corresponding larger area. Figure 6BAs shown, die 630 is directly connected to die 625(1) and 625(3) via corresponding D2D pins in a smaller area. Therefore, integrated circuit 620 can provide direct high-speed data communication between die 630 and each of die 625(1), 625(2), 625(3) and 625(4). The communication bandwidth between each pair of dies is not uniform, but the communication bandwidth between die 630 and die 625(2) (or die 625(4)) is greater than the communication bandwidth between die 630 and die 625(1) (or die 625(3)).

[0123] When a higher data communication bandwidth is required between two pairs of dies (e.g., between die 630 and die 625(2) and between die 630 and die 625(4)), it is used to implement Figure 6B The integrated circuit 620 Fig. 6A The block structure 600 in may be suitable. In one or more embodiments, the tube core 625 (3) (or Figure 6B Some other dies in Figure 6B The other die in the embodiment, namely die 625(3), may be a computer device (e.g., a printed circuit board) having a circuit board for connecting integrated circuit 620 and Figure 5B The die 630 may be a TSP, a memory device (e.g., an HBM device), an interface device (e.g., a bridge die), or some other device. It should be noted that the die 630, when implemented as an interface device (e.g., a bridge die), may be used to interface the integrated circuit 620 (e.g., via one or more quadrants) with a host computer, an HBM device, or some other device.

[0124] Fig. 7AAn example top view and bottom view of an integrated circuit 700 according to some embodiments are shown, the integrated circuit 700 including a plurality of block structures (e.g., eight block structures) interconnected via T2T bridges. The integrated circuit 700 includes an array of block structures 710A to 710H spanning a first dimension (e.g., x dimension), a second dimension orthogonal to the first dimension (e.g., y dimension), and a third dimension orthogonal to the first and second dimensions (e.g., z dimension). The block structures in the array are interconnected into the integrated circuit 700 via T2T bridges 715. The block structures 710A, 710B, 710C, 710D are adjacent to each other on both sides and are placed in the same 2D plane (e.g., spanning the x and y dimensions), and the T2T bridges 715 are superimposed in the third dimension (e.g., z dimension). Similarly, the block structures 710E, 710F, 710G, 710H are adjacent to each other on two sides and are placed in the same 2D plane (e.g., across the x-dimension and the y-dimension), and the T2T bridge 715 is superimposed in the third dimension (e.g., the z-dimension). At least a subset of the block structures 710A to 710H can be configured to function as a single core processor for model parallelism across multiple dies of the block structures 710A to 710H.

[0125] Each block structure 710A to 710H can have the same configuration as the block structure 200. Thus, each block structure 710A to 710H can include a first die and a second die connected to the first die in an F2F configuration. The first die can be offset by a first offset along a first dimension relative to the second die, and offset by a second offset along a second dimension, thereby forming an offset alignment between the first die and the second die. The first die in each block structure 710A to 710H can include a TSP, and the second die in each block structure 710A to 710H can include another TSP, a memory device (e.g., an HBM device), an interface device, some other device, or some combination thereof.

[0126] The top view 705A of the integrated circuit 700 shows the block structures 710A, 710B, 710C, 710D interconnected via a first side (e.g., top side) of the T2T bridge 715. Similarly, the bottom view 705B of the integrated circuit 700 shows the block structures 710E, 710F, 710G, 710H interconnected via a second side (e.g., bottom side) of the T2T bridge 715 opposite the first side. The T2T bridge is implemented as a bridge die with T2T pins (e.g., interconnect pads) on both sides of the bridge die. At least a subset of the T2T pins on the first side of the T2T bridge 715 are aligned with corresponding T2T pins of the block structures 710A, 710B, 710C, 710D. Similarly, at least a subset of the T2T pins on the second side of the T2T bridge 715 are aligned with corresponding T2T pins of the block structures 710E, 710F, 710G, 710H. Thus, the integrated circuit 700 effectively includes two horizontal layers of block structures - a first horizontal layer of block structures 710A, 710B, 710C, 710D and a second horizontal layer of block structures 710E, 710F, 710G, 710H, and the two horizontal layers of block structures are interconnected along a vertical dimension (e.g., z dimension) via a T2T bridge 715.

[0127] Figure 7B An example top view and bottom view of an integrated circuit 720 according to some embodiments are shown, the integrated circuit 720 including a plurality of block structures (e.g., 32 block structures) interconnected via T2T bridges. The integrated circuit 720 includes an array of block structures TS1 to TS32 spanning a first dimension (e.g., an x ​​dimension), a second dimension orthogonal to the first dimension (e.g., a y dimension), and a third dimension orthogonal to the first and second dimensions (e.g., a z dimension). The block structures in the array are interconnected into the integrated circuit 720 via a T2T bridge 730. At least some of the block structures in the array of block structures TS1 to TS32 can be configured to function as a single core processor for model parallelism across multiple dies of the block structures TS1 to TS32.

[0128] Each block structure TS1 to TS32 may have the same configuration as the block structure 200. Thus, each block structure TS1 to TS32 may include a first die and a second die connected to the first die in an F2F configuration. The first die may be offset by a first offset along a first dimension relative to the second die, and offset by a second offset along a second dimension, thereby forming an offset alignment between the first die and the second die. The first die in each block structure TS1 to TS32 may include a TSP, and the second die in each block structure TS1 to TS32 may include another TSP, a memory device (e.g., an HBM device), an interface device, some other device, or some combination thereof.

[0129] A top view 725A of the integrated circuit 720 shows block structures TS1 to TS16 interconnected via a first side (e.g., top side) of a T2T bridge 730. Similarly, a bottom view 725B of the integrated circuit 720 shows block structures TS17 to TS32 interconnected via a second side (e.g., bottom side) of the T2T bridge 730 opposite the first side. The T2T bridge 730 is implemented as a bridge die with T2T pins (e.g., T2T interconnect pads) on both sides of the bridge die. At least a subset of the T2T pins on the first side of the T2T bridge 730 are aligned with corresponding T2T pins of the block structures TS1 to TS16. Similarly, at least a subset of the T2T pins on the second side of the T2T bridge 730 are aligned with corresponding T2T pins of the block structures TS17 to TS32. Thus, integrated circuit 720 effectively includes two horizontal layers of block structures—a first horizontal layer of block structures TS1 to TS16 and a second horizontal layer of block structures TS17 to TS32—interconnected along a vertical dimension (eg, z dimension) via T2T bridge 730 .

[0130] Fig. 8A 800 includes a plurality of block structures and an HBM stack connected to each other via a T2T bridge. The integrated circuit 800 includes a first side (e.g., top side) (e.g., with a T2T bridge 810) connected to each other via a T2T bridge 810. Fig. 7A 705A in the top view) interconnected block structures 805A, 805B (and Fig. 8A The integrated circuit 800 further includes a second side (eg, bottom side) of the T2T bridge 810 (eg, with Fig. 7A 705B in the bottom view) interconnected block structures 805C, 805D (and Fig. 8A The T2T bridge 810 may be an implementation of the T2T bridge 715 or the T2T bridge 730 .

[0131] Each block structure in the integrated circuit 800 may have the same configuration as the block structure 200. The first die in each block structure of the integrated circuit 800 may include a TSP, and the second die in each block structure of the integrated circuit 800 may include another TSP, a memory device (e.g., an HBM device), an interface device, some other device, or some combination thereof. The number of block structures connected via the top side of the T2T bridge 810 is 4N, the number of block structures connected via the bottom side of the T2T bridge 810 is 4N, and N is an integer. Therefore, the total number of block structures in the integrated circuit 800 may be 8N (e.g., N=1, 2, 3, or 4).

[0132] The integrated circuit 800 also includes a first stack 815A of memory devices (e.g., a stack of HBM devices), which is placed on a first side of the T2T bridge 810, for example, spatially located between the block structure 805A and the block structure 805B. The HBM stack 815A may include one or more HBMs stacked on top of each other in an F2B configuration or in a B2B configuration (e.g., via a TSV connector). The integrated circuit 800 also includes a second stack 815B of memory devices (e.g., a stack of HBM devices), which is placed on a second side of the T2T bridge 810, for example, spatially located between the block structure 805C and the block structure 805D. The HBM stack 815B may include one or more HBMs stacked on top of each other in an F2B configuration or in a B2B configuration (e.g., via a TSV connector). The block structure in the integrated circuit 800 and the HBM stacks 815A, 815B can be used as a single-core processor for model parallelism of multiple dies and HBMs across the block structure.

[0133] Figure 8B 8 shows an example side view of an integrated circuit 820 including a plurality of block structures and an HBM stack and a heat sink connected to each other via a T2T bridge according to some embodiments. The integrated circuit 820 includes a first side (e.g., top side) (e.g., with a T2T bridge 830) connected to each other. Fig. 7A 705A in the top view) interconnected block structures 825A, 825B (and Figure 8B The integrated circuit 820 also includes a second side (eg, bottom side) of the T2T bridge 830 (eg, with Fig. 7A 705B in the bottom view) interconnected block structures 825C, 825D (and Figure 8B The integrated circuit 820 may include at least two more block structures not shown in the figure). The T2T bridge 830 may be an implementation of the T2T bridge 715 or the T2T bridge 730. Each block structure in the integrated circuit 820 may have the same configuration as the block structure 200. The first die in each block structure of the integrated circuit 800 may include a TSP, and the second die in each block structure of the integrated circuit 800 may include another TSP, a memory device (e.g., an HBM device), an interface device, some other device, or some combination thereof.

[0134] The integrated circuit 820 also includes a first stack 835A of memory devices (e.g., a stack of HBM devices), which is placed on the first side of the T2T bridge 830, for example, spatially located between the block structure 825A and the block structure 825B. The HBM stack 835A may include one or more HBMs stacked on top of each other in an F2B configuration or in a B2B configuration (e.g., via a TSV connector). The integrated circuit 820 also includes a second stack 835B of memory devices (e.g., a stack of HBM devices), which is placed on the second side of the T2T bridge 830, for example, spatially located between the block structure 825C and the block structure 825D. The HBM stack 835B may include one or more HBMs stacked on top of each other in an F2B configuration or in a B2B configuration (e.g., via a TSV connector). The block structure in the integrated circuit 820 and the HBM stacks 835A, 835B can be used as a single-core processor for model parallelism of multiple dies and HBMs across the block structure.

[0135] The integrated circuit 820 also includes a heat sink 840 coupled to the outer surface of the block structures 825A, 825B (and any additional block structures coupled to the first side of the T2T bridge 830) and the HBM stack 835A. The heat sink 840 can be configured to dissipate heat from the block structures 825A to 825D (and any additional block structures coupled to the first side and the second side of the T2T bridge 830) and the HBM stack 835A, 835B. In one or more embodiments, the heat sink 840 is implemented as a heat sink die, which includes, for example, a metal layer formed on top of a substrate. In one or more other embodiments, the heat sink 840 can be implemented as a thermal filler that fills the gap between the block structures. For example, the thermal filler of the heat sink 840 can fill the gap between the block structure 825A and the block structure 825B (and any additional block structures coupled to the first side of the T2T bridge 830). The thermal filler of the heat sink 840 may be, for example, a silicon (graphene-based) filler or a graphene tube placed on a copper cold plate in contact with the silicon.

[0136] The integrated circuit 820 may also include a power supply layer 845, for example, coupled to the outer surface of the block structures 825C, 825D (and any additional block structures coupled to the second side of the T2T bridge 830) and the HBM stack 835B. The power supply layer 845 may include an array of C4 bumps to provide a power supply network for the lower silicon layers in the integrated circuit 820 (e.g., the lower die of the HBM stack 835B, the lower die of the block structures 825C, 825D, and any additional block structures coupled to the second side of the T2T bridge 830). Other dies (i.e., upper silicon layers) in the integrated circuit 820 may include power delivery via TSV connectors.

[0137] Fig. 9 An example integrated circuit of a cuboid structure 900 implemented as a block structure according to some embodiments is shown. Cuboid structure 900 represents a three-dimensional array of interconnected block structures, i.e., an array of NxMxK ​​block structures spanning three spatial dimensions (e.g., x-dimension, y-dimension, and z-dimension, respectively), where N, M, and K are integers (e.g., less than 5). Each block structure in cuboid structure 900 can have the configuration of block structure 200. The block structures in cuboid structure 900 can be configured to operate as a single-core processor for model parallelism across block structures.

[0138] like Fig. 9 As shown, the cuboid structure 900 includes multiple horizontal layers of block structures, and the multiple horizontal layers are interconnected vertically (e.g., along the z dimension). Each horizontal layer of the block structure includes a two-dimensional array (e.g., an NxM array) of block structures spanning a first dimension (e.g., an x ​​dimension) and a second dimension (e.g., a y dimension). A pair of adjacent block structures in each horizontal layer can be interconnected via an offset alignment formed between a pair of tube cores in each block structure (e.g., as shown in FIG. Figure 4 A pair of adjacent horizontal layers of block structures are connected along the vertical dimension (e.g., z dimension) by coupling corresponding block structures in adjacent horizontal layers along the vertical dimension (e.g., z dimension). The number of horizontal layers of block structures connected along the vertical dimension (e.g., z dimension) is K (as shown in Fig. 9 In one embodiment, corresponding block structures in adjacent horizontal layers are coupled to each other in a B2B configuration via TSV connectors along a vertical dimension (e.g., z dimension). In another embodiment, corresponding block structures in adjacent horizontal layers are coupled to each other in a F2B configuration via TSV connectors along a vertical dimension (e.g., z dimension). In yet another embodiment, corresponding block structures in adjacent horizontal layers are coupled to each other in a F2B configuration via TSV connectors along a vertical dimension (e.g., z dimension). FIG. 7A to FIG. 7B ) are coupled along a vertical dimension (e.g., z dimension). In some embodiments, two adjacent horizontal layers in the rectangular parallelepiped structure 900 are coupled to each other by connecting their block structures in a B2B configuration, while another two adjacent horizontal layers in the rectangular parallelepiped structure 900 are coupled to each other by connecting their block structures in an F2B configuration.

[0139] The rectangular parallelepiped structure 900 may also include a plurality of heat sinks ( Fig. 9). Each heat sink in the cuboid structure 900 can be directly connected to at least one block structure in the cuboid structure 900 and configured to dissipate heat from at least one block structure. Each heat sink can be implemented as, for example, a silicon (graphene-based) filler, a graphene tube placed on a copper cold plate in contact with silicon, a metal layer on a substrate, some other type of heat sink, or a combination thereof. In one or more embodiments, each heat sink is placed between two adjacent horizontal layers of the block structure and is configured to dissipate heat from two adjacent horizontal layers of the block structure. In addition, a pair of heat sinks can be placed on the upper surface of the cuboid structure 900 and on the lower surface of the cuboid structure 900 (e.g., with respect to the vertical or z dimension), and can be configured to dissipate heat from the outer horizontal layers of the block structure in the cuboid structure 900.

[0140] In some embodiments, a compiler that controls data operations performed on the dies of the cuboid structure 900 can be configured to provide a specific silicon power tradeoff, for example, by running twice the number of dies of the cuboid structure 900 at 50% of the maximum defined clock rate. Additionally or alternatively, the compiler can map utilization to resources within the cuboid structure 900 to stagger heat generation and optimize local heating of the dies and block structures. In one or more embodiments, each die of the block structure in the cuboid structure 900 can have a different size and / or perform a different function. The compiler can optimize the silicon power tradeoff of the cuboid structure 900 by utilizing the granularity of the size and function of each die of the cuboid structure 900.

[0141] In some embodiments, one or more cuboid structures 900 are housed in a rack and can be used in a data center. A data center may contain thousands of such racks. Each cuboid structure 900 in a rack can be configured to operate as a single-core deterministic stream processor (e.g., TSP) that runs a corresponding model (e.g., a machine learning model). Thus, multiple models can be executed asynchronously across multiple cuboid structures 900 in a rack. The rack may also include a central controller that controls the operation of each cuboid structure 900 in the rack (e.g., via a compiler running on the central controller). If the data center features a threshold power budget for the entire rack, each cuboid structure 900 can be managed individually to allocate power among all cuboid structures 900 in the rack. Advantageously, power can be managed not only to prevent exceeding the threshold power budget, but also to maintain the quality of service (QoS) of each cuboid structure 900 in the rack. Example process flow for an integrated circuit with a block structure

[0142] Fig.101 is a flowchart showing a method 1000 according to some embodiments, wherein the method 1000 uses an integrated circuit to perform data processing with model parallelism of multiple dies across one or more block structures. The integrated circuit may also include at least one computer processor (e.g., a deterministic stream processor) and a non-transitory computer-readable storage medium for storing computer executable instructions. The deterministic stream processor may be a TSP. In one or more embodiments, the integrated circuit includes a block structure with a pair of dies, which can be operated as a single-core deterministic stream processor. In this case, the integrated circuit may be, for example, an embodiment of the block structure 200, the block structure 300, the block structure 500, or the block structure 600. In one or more other embodiments, the integrated circuit includes a two-dimensional array of block structures that can be operated as a single-core deterministic stream processor. In this case, the integrated circuit may be, for example, an embodiment of one of the integrated circuits 410, 415, 420, 425, 520, 620. In one or more other embodiments, the integrated circuit includes a three-dimensional array of block structures that can be operated as a single-core deterministic stream processor. In this case, the integrated circuit may be, for example, one of the integrated circuits 700 , 800 , 820 or an implementation of the cuboid structure 900 .

[0143] The operations of method 1000 may be initiated by a compiler running on at least one computer processor and / or a host server separate from the integrated circuit. The compiler may utilize a model of a deterministic stream processor (e.g., a machine learning model) as its input and output instructions for configuring the operation of the deterministic stream processor and the integrated circuit as a whole. Integrated circuit startup 1005 issues instructions for execution by processing units of multiple dies (e.g., computing elements of one or more functional slices) across one or more block structures of the integrated circuit. Integrated circuit startup 1010 executes instructions by streaming data of processing units of multiple dies across one or more block structures. Integrated circuit startup 1015 returns the resulting data to one or more memory slices of one or more block structures.

[0144] Die-to-die architecture for deterministic stream processors

[0145] Embodiments of the present disclosure also relate to die-to-die (D2D) dense packaging of deterministic stream processors (e.g., TSP). Each deterministic stream processor (e.g., TSP) connected in a D2D structure features an extended scalable computing architecture suitable for running next-generation artificial intelligence / machine learning algorithms. Each deterministic stream processor of the D2D structure can support SIMD operations (e.g., 256-byte SIMD operations) by spatial pipeline processing of S (e.g., S=64) concurrent streams per super channel streaming, and features a memory bandwidth of more than, for example, 130TB / s. Each deterministic stream processor in the D2D structure can also support deterministic high-bandwidth memory (HBM), which is insensitive to stride and features a large amount of concurrency (e.g., 1.5TB / s of HBM bandwidth). Each deterministic stream processor in the D2D structure can include four MXM engines, each of which supports 256x256 fused dot product calculations, doubles the FP16 (16-bit floating point) density, and features additional support for INT4 (4-bit integer) operations. Each deterministic stream processor in the D2D structure can also include a programmable high-performance VXM (e.g., 8192 vector ALUs), an SXM with a double aligner (for improved data movement and data reshaping), and an ICU with multiple instruction queues (for achieving instruction parallelism). Each deterministic stream processor in the D2D structure can support extensible network scalability and multiple D2D topologies.

[0146] Fig.11 An example D2D structure 1100 with two deterministic stream processors (or dies) connected in a D2D configuration according to some embodiments is shown. The D2D structure 1100 provides an extension of a super channel across two dies via a D2D interface. The D2D structure 1100 includes a die 1105 having a first deterministic stream processor (or a first TSP core) and a die 1110 having a second deterministic stream processor (or a second TSP core). The die 1105 can be connected to the die 1110 via a D2D interface 1115, thereby forming the D2D structure 1100. The D2D interface 1115 can support four streams at, for example, 2GHz. The D2D structure 1100 is configured to be used as a single-core processor for model parallelism across the die 1105 and 1110.

[0147] In one embodiment, the D2D interface 1115 represents an interface for mapping 16 super channels between the tube cores 1105 and 1110. In this case, the D2D interface 1115 can support a streaming rate of up to 2TB / sec. In another embodiment, the D2D interface 1115 represents an interface for mapping 24 super channels between the tube cores 1105 and 1110. In this case, the D2D interface 1115 can achieve a streaming rate of up to 3TB / sec. The size of the D2D structure 1100 along the horizontal direction (e.g., the x direction) can be, for example, 65mm or 85mm; the size of the D2D structure 1100 along the vertical direction (e.g., the y direction) can be 65mm or 85mm. The D2D structure 1100 can support, for example, the FP16 data format for implementing 2PetaFlops (floating point operations); the INT8 data format for implementing 4PetaOps; and the INT4 data format for implementing 8PetaOps. The D2D structure 1100 may also include, for example, 480 megabytes of SRAM.

[0148] Each die 1105, 1110 (e.g., each TSP core) of the D2D structure 1100 can include a deterministic HBM that provides, for example, 1.6TB / s of streaming bandwidth to the local DRAM of the corresponding die 1105, 1110. Each die 1105, 1110 can be designed to support, for example, a 64-byte word size. The D2D structure 1100 can support a predictable and scalable low-latency interconnect network (e.g., on-chip and off-chip). The D2D structure 1100 can also support energy proportionality to exploit dynamic power ranges.

[0149] Fig.12 An example D2D structure 1200 with an extended super channel across multiple dies according to some embodiments is shown. The D2D structure 1200 provides an extension of the super channel across multiple dies in a horizontal direction (e.g., an x-direction). The D2D structure 1200 includes a die 12051 having a D2D interface 12100 (e.g., for possible extension of the D2D structure 1200 in the x-direction), the D2D interface 12101, a die 12052, a D2D interface 12102, ..., a D2D interface 1210 N-1 , and having a D2D interface 1210 N The tube core 1205 N (e.g., for possible expansion of D2D structure 1200 along the x-direction), where N ≥ 3. D2D interface 12101 may provide mapping of the super channel of die 12051 to the super channel of die 12052, D2D interface 12102 may provide mapping of the super channel of die 12052 to the super channel of die 12053, and so on. In general, D2D interface 1210 iCan provide die 1205 i Super Channel to Die 1205 i+1 The mapping of the super channel, where i = 1, 2, ..., N-1.

[0150] The D2D structure 1200 can support dense packaging of additional dies along the vertical direction (e.g., the y direction). Additionally or alternatively, the D2D structure 1200 can support dense packaging of one or more dies of the D2D structure 1200 on a substrate. The D2D structure 1200 can support expansion of fast SRAM memory capacity and provide a path for very efficient model parallelism across multiple dies. The latency of any D2D interface of the D2D structure 1200 can be only, for example, 18 clock cycles (or 9ns), while each clock cycle transmits S streams (e.g., S=4) in each direction. The bit error rate (BER) of any D2D interface of the D2D structure 1200 can be less than 10 -20 , which makes the D2D interface of the D2D structure 1200 robust and as reliable as wires. The D2D structure 1200 is configured to be used as a cross-die 12051, 12052, ..., 1205 N Model parallelism on a single core processor.

[0151] Fig.13A An example D2D structure 1300 with three dies connected in a D2D configuration is shown according to some embodiments. The D2D structure 1300 provides an extension of a super channel across three dies (i.e., across three deterministic streaming processors or TSP cores). The D2D structure 1300 includes dies 1305, 1310, 1315 connected in a D2D configuration for extending their super channels. The D2D interface 1320 maps the super channel of the die 1305 to the super channel of the die 1310, and the D2D interface 1325 maps the super channel of the die 1310 to the super channel of the die 1315. The D2D structure 1300 may be an embodiment of the D2D structure 1200 with N=3. The size of the D2D structure 1300 in the horizontal direction (e.g., the x direction) may be, for example, 95 mm, and the size of the D2D structure 1300 in the vertical direction (e.g., the y direction) may be, for example, 95 mm. The D2D structure 1300 may support, for example, the FP16 data format for implementing 3 PetaFlops, the INT8 data format for implementing 6 PetaOps, and the INT4 data format for implementing 12 PetaOps. The D2D structure 1300 may also include, for example, 720 megabytes of SRAM. The D2D structure 1300 is configured to be used as a single-core processor for model parallelism across die 1305, 1310, 1315.

[0152] Fig. 13BAn example D2D structure with dies connected in a D2D folded mesh configuration is shown in accordance with some embodiments. Fig. 13B Initially, a D2D structure 1330 is shown with four dies 13350, 13351, 13352, and 13353 having super channels extending in a horizontal direction (e.g., an x-direction). Each die 13350, 13351, 13352, and 13353 includes a respective deterministic stream processor or TSP core. The D2D interface 13400 maps the super channel of die 13350 to the super channel of die 13351 in a horizontal direction (e.g., an x-direction); the D2D interface 13401 maps the super channel of die 13351 to the super channel of die 13352 in a horizontal direction (e.g., an x-direction); and the D2D interface 13402 maps the super channel of die 13352 to the super channel of die 13353 in a horizontal direction (e.g., an x-direction). The D2D structure 1330 may be an implementation of the D2D structure 1200 with N=4. The size of the D2D structure 1330 along the horizontal direction (eg, x-direction) may be, for example, 120 mm, and the size of the D2D structure 1330 along the vertical direction (eg, y-direction) may be, for example, 55 mm.

[0153] The D2D structure 1330 may be converted to a D2D structure 1345 having a D2D folded grid configuration (e.g., a radix-4 grid). The D2D interface 13500 maps the superchannel of the die 13350 to the superchannel of the die 13351 along a vertical direction (e.g., y direction); the D2D interface 13501 maps the superchannel of the die 13351 to the superchannel of the die 13352 along a vertical direction (e.g., y direction); and the D2D interface 13502 maps the superchannel of the die 13352 to the superchannel of the die 13353 along a vertical direction (e.g., y direction). Compared to the D2D structure 1330, the D2D structure 1345 may achieve the same streaming bandwidth with a smaller overall size. The size of the D2D structure 1345 along the horizontal direction (e.g., x direction) can be, for example, 55 mm, and the size of the D2D structure 1345 along the vertical direction (e.g., y direction) can be, for example, 55 mm. Similar to the D2D structure 1330, the D2D structure 1345 is configured to be used as a single-core processor for model parallelism across the dies 13350, 13351, 13352, 13353.

[0154] Fig. 13C An example D2D structure 1360 with dies connected in a D2D torus configuration is shown in accordance with some embodiments. Fig. 13CAs shown, a D2D structure 1345 having a D2D folded mesh configuration (e.g., a radix-4 mesh configuration) can be used to form a D2D structure 1360. The D2D ring configuration (e.g., a radix-8 ring configuration) of the D2D structure 1360 can be created by connecting two D2D folded mesh configurations (e.g., two radix-4 mesh configurations) in a horizontal direction (e.g., an x-direction) via a D2D interface 1370. The D2D structure 1360 includes die 13650, 13651, 13652, 13653 connected in a first D2D folded mesh configuration (e.g., a first radix-4 mesh configuration), and die 13654, 13655, 13656, 13657 connected in a second D2D folded mesh configuration (e.g., a second radix-4 mesh configuration). The first D2D folded grid configuration is connected to the second D2D folded grid configuration via a D2D interface 1370, thereby forming a D2D structure 1360 having eight dies connected in a D2D ring configuration (e.g., a radix-8 ring configuration). The size of the D2D structure 1360 in the horizontal direction (e.g., the x-direction) may be, for example, 85 mm, and the size of the D2D structure 1360 in the vertical direction (e.g., the y-direction) may be, for example, 85 mm. The D2D structure 1360 is configured to be used as a single-core processor for model parallelism across dies 13650, 13651, 13652, 13653, 13654, 13655, 13656, 13657.

[0155] Fig.14 An example D2D mapping structure 1400 for mapping super channels between a pair of die is shown in accordance with some embodiments. The D2D mapping structure 1400 provides a connection between a super channel of a first die (i.e., a first deterministic stream processor or a first TSP core) and a super channel of a second die (i.e., a second deterministic stream processor or a second TSP core). The D2D mapping structure 1400 includes a super cell array 1405 (which may be part of the first die), a D2D interface 1410, and a super channel 1415 (which may be part of the second die). Although Fig.14 The D2D mapping structure 1400 in FIG. 1 shows mapping of 16 super channels, but it should be understood that the D2D mapping structure 1400 can be extended to support mapping of some other number of super channels (eg, 20 super channels or 24 super channels).

[0156] The super cell array 1405 may be implemented as a two-dimensional array of N super cells (e.g., N=4). Each super cell in the super cell array 1405 may include an on-chip stream register for storing data streamed via the D2D interface 1410. The super cell array 1405 may provide a stream bandwidth of, for example, hundreds of TB / sec. The super cell array 1405 is directly connected to the super channel ( Fig.14 1415). The D2D interface 1410 connects the super cell array 1405 of the first die to the super channel 1415 of the second die, thereby providing an extension of the super channel across a pair of dies. The D2D interface 1410 includes a plurality of D2D interface banks 1418, and each D2D interface bank 1418 provides a mapping to a corresponding subset of the super channel 1415. Each D2D interface bank 1418 includes a D2D core interface 1412, a D2D physical layer (PHY) control circuit 1414, and a bidirectional interface slice 1416 for providing a physical connection to the corresponding super channel 1415. Each pair of bidirectional interface slices 1416 can be connected to a corresponding super channel 1415.

[0157] Each super channel 1415 and the corresponding portion of the D2D interface 1410 can stream, for example, 256 bits in each direction (e.g., along the x-axis), including S streams per clock cycle per super channel (e.g., S=4). For a total bandwidth of, for example, 9TB / sec, the D2D interface 1410 can support, for example, up to 4.6TBytes / sec on each hemisphere of the TSP core. The D2D interface 1410 can be placed on both hemispheres of the TSP core to allow physical and data streaming symmetry.

[0158] Fig.14 The D2D interface 1410 shown can map 17 super channels of the first die (i.e., 17 rows of super cells 1405) to 16 super channels 1415 of the second die. Logically, each super channel 1415 outputs, for example, S=4 streams, 4x128 bits=512 bits in each direction. Each interface slice 1416 can support 256 bits of streaming in any direction at the same time, and each interface slice 1416 is part of a corresponding D2D interface macro in multiple D2D interface macros. Multiple D2D interface macros form a D2D interface group 1418. In Fig.14 In an illustrative embodiment, eight D2D interface macros form a D2D interface group 1418. Each super channel 1415 can be directly mapped to, for example, a pair of 256-bit bidirectional interface slices 1416, and the super channels 1415 are grouped into clusters of S super channels 1415 (e.g., S=4 super channels). The eight D2D interface macros forming a D2D interface group 1418 have approximately the same height and matching spacing as the cluster of S super channels 1415 (e.g., S=4 super channels).

[0159] The D2D mapping structure 1400 with the D2D interface 1410 allows both face-to-face (F2F) and back-to-back (B2B) die orientations when stacking multiple dies into a D2D configuration. The D2D mapping structure 1400 enables three-dimensional packaging with system-in-packages having, for example, radix-4 grid configurations and radix-8 ring configurations. The D2D mapping structure 1400 also allows efficient model parallelism within the system-in-package, which can utilize the nearest neighbor communication mode. The D2D mapping structure 1400 provides a preferred communication hierarchy to each die connected in a D2D configuration, for example, on-chip stream register bandwidth on the order of hundreds of TB / sec, D2D stream bandwidth on the order of tens of TB / sec, and off-chip network bandwidth on the order of TB / sec.

[0160] Fig.15A An example die 1500 is shown with a first number of super channels (eg, 16 super channels) and a D2D interface according to some embodiments. The die 1500 includes a plurality of super channels disposed on two hemispheres of the die 1500. Fig.14 The D2D mapping structure 1400 is used to map the super channel of the tube core 1500 to a tube core or two tube cores having a substantially same structure as the tube core 1500 ( Fig.15A The die 1500 includes, for example, 16 (+1) super channels, PCIe, and 12 C2C modules. The D2D interface of the die 1500 includes, for example, a total of 8 D2D interface groups (4 East D2D interface groups and 4 West D2D interface groups) placed on two hemispheres, and 32 interface slices in each hemisphere. The die 1500 can provide a streaming bandwidth of 512 Gb / s per interface slice in each direction, a transmission bandwidth of 2 TB / s per hemisphere, and a receiving bandwidth of 2 TB / s per hemisphere. The die 1500 can also include, for example, 256 MB SRAM. The area of ​​the die 1500 is, for example, 468 mm 2 .

[0161] Fig. 15B An example die 1510 with a second number of super channels (eg, 20 super channels) and a D2D interface is shown in accordance with some embodiments. The die 1510 includes a plurality of super channels disposed on two hemispheres of the die 1510. Fig.14 The D2D mapping structure 1400 is used to map the super channel of the tube core 1510 to a tube core or two tube cores having a substantially same structure as the tube core 1510 ( Fig. 15BThe die 1510 includes, for example, 20 (+1) super channels, PCIe, and 12 C2C modules. The D2D interface of the die 1510 includes, for example, a total of 10 D2D interface groups (5 East D2D interface groups and 5 West D2D interface groups) placed on two hemispheres, and 40 interface slices in each hemisphere. The die 1510 can provide a flow bandwidth of 512Gb / s per interface slice in each direction, a transmission bandwidth of 2.5TB / s per hemisphere, and a receiving bandwidth of 2.5TB / s per hemisphere. The die 1510 can also include, for example, 250MB SRAM. The area of ​​the die 1500 is, for example, 530mm 2 Compared to die 1500 , die 1510 has an area that is approximately 20% larger than die 1500 , yet die 1510 can achieve 57% better performance (ie, higher bandwidth) than die 1500 .

[0162] Fig. 15C An example die 1520 with a third number of super channels (eg, 24 super channels) and a D2D interface is shown in accordance with some embodiments. The die 1520 includes a plurality of super channels disposed on two hemispheres of the die 1520. Fig.14 The D2D mapping structure 1400 in FIG. 1 is used to map the super channel of the die 1520 to another die having a substantially same structure as the die 1520 ( Fig. 15C The die 1520 includes, for example, 24 (+1) super channels, PCIe, and 12 C2C modules. The D2D interface of the die 1510 includes, for example, a total of 12 D2D interface groups (6 East D2D interface groups and 6 West D2D interface groups) placed on two hemispheres, and 48 interface slices in each hemisphere. The die 1520 can provide a flow bandwidth of 512Gb / s per interface slice in each direction, a transmission bandwidth of 3TB / s per hemisphere, and a receiving bandwidth of 3TB / s per hemisphere. The die 1520 can also include, for example, 240MB SRAM. The area of ​​the die 1520 is, for example, 616mm 2 Compared to die 1520 , which has an area that is approximately 50% larger than that of die 1500 , die 1520 can achieve 225% better performance (ie, higher bandwidth) than die 1500 .

[0163] Die 1520 may also support relatively low supply voltages and low clock frequencies to achieve operational efficiency. Optionally, die 1520 may include two 32b channels of low power double data rate (LPDDR) memory placed in each corner of die 1520. In some embodiments ( Fig. 15C), the number of C2C modules can be reduced (e.g., from 12 C2C modules to 9 C2C modules) to make room for additional LPDDR channels. Die 1520 can process, for example, 240 megabytes per chip while executing approximately 1Pops at 750MHz (for power efficiency) or approximately 1PFlops at 1.8GHz. The bandwidth of one channel of LPDDR is, for example, 34GB / sec, which matches the single stream serialization bandwidth (e.g., 16B / cycle) - a bandwidth of 32GB / sec per stream can translate into a bandwidth of 34GB / sec for the LPDDR channel.

[0164] Embodiments of the present disclosure relate to a new method for building a "network-in-package" using a D2D interface. The packaging scheme presented herein allows the construction of a multi-chip module with multiple dies that are "side by side" or "vertically stacked" by "folding" the dies. This packaging scheme extends the "streaming" programming model across multiple chips (or dies) because the stream automatically flows from one chip (i.e., die) to the next chip in an eastward or westward direction. The physical symmetry of the D2D interface placement at the east / west edge of the die allows the die to be considered "edge symmetrical" and allows the die to be "flipped" upside down to be "stacked" on top of its adjacent die. Edge symmetry allows adjacent dies to be stacked "face to face" or "back to back". Edge symmetry enables packaging of other horizontal dies and allows a "network of TSPs" to be built within the package, such as generating a "folded grid" to include, for example, four TSP cores in a single 45x45 mm package.

[0165] The D2D package (or D2D structure) operates as a single "functional unit" with control, which is simplified to passing S streams (e.g., S=4 streams) across each D2D interface macro when, for example, the "valid" (propagation) bit is set. The packaging scheme presented in this article allows extremely efficient streaming activation across multiple chips with only a few cycles of latency, for example, achieving bandwidths of more than TBytes / sec with limited control / instruction overhead. The packaging scheme presented in this article also allows the MXM across multiple chips (i.e., multiple TSP cores) in the D2D package to be considered "asymmetric", because if the MXM is connected to two or three tube cores (or TSP chips) in the D2D package respectively, the size of the MXM will be, for example, 256x512 or 256x768. The packaging scheme presented in this article also allows synchronous streaming between tube cores (or TSP chips), which truly extends the flow model of deterministic fixed latency.

[0166] Fig.161 is a flow chart illustrating a method 1600 of using an integrated circuit for data processing with model parallelism across multiple dies connected in a D2D structure according to some embodiments. The integrated circuit may also include at least one computer processor (e.g., a deterministic stream processor) and a non-transitory computer-readable storage medium for storing computer-executable instructions. Each die in the D2D structure may include a deterministic stream processor. The deterministic stream processor may be a TSP. A D2D structure with multiple dies may be configured to operate as a single-core deterministic stream processor.

[0167] The operations of method 1600 may be initiated by a compiler running on at least one computer processor (e.g., as part of an integrated circuit) and / or a host server (e.g., separate from the integrated circuit). The compiler may utilize a model of a deterministic stream processor (e.g., a machine learning model) as its input and output instructions for configuring the operation of an integrated circuit having multiple dies connected in a D2D structure.

[0168] The integrated circuit initiates 1605 (e.g., via a compiler) to issue a plurality of instructions for execution across a plurality of processing units of a first die and a second die, the second die being connected to the first die in a D2D configuration via a D2D interface circuit, thereby forming a D2D structure with the first die. The integrated circuit initiates 1610 (e.g., via a compiler) streaming of data between a first plurality of super channels of the first die and a second plurality of super channels of the second die via the D2D interface circuit in a first direction or a second direction orthogonal to the first direction, to execute the plurality of instructions.

[0169] The integrated circuit may (e.g., via a compiler) initiate streaming of data across a plurality of dies in at least one of a first direction and a second direction, the plurality of dies being connected in a D2D configuration via a plurality of D2D interface circuits and forming a D2D structure. The integrated circuit may (e.g., via a compiler) configure the plurality of dies forming the D2D structure to function as a single core processor for model parallelism across the plurality of dies of the D2D structure.

[0170] In some embodiments, the D2D interface circuit includes a plurality of bidirectional interface slices, and a pair of bidirectional interface slices are connected to corresponding super channels of a second plurality of super channels of the second die. The D2D interface circuit also includes a D2D core interface circuit connected to a corresponding subset of the first plurality of super channels of the first die.

[0171] In some embodiments, the D2D interface circuit includes a plurality of D2D interface groups, each of the plurality of D2D interface groups connects a cluster of a first plurality of super channels to a cluster of a second plurality of super channels. The size of each of the plurality of D2D interface groups of the D2D interface along the first direction matches the size of the cluster of the second plurality of super channels along the first direction. Each of the plurality of D2D interface groups of the D2D interface includes a plurality of D2D interface macros, and each of the plurality of D2D interface macros includes a corresponding bidirectional interface slice of a plurality of bidirectional interface slices.

[0172] The D2D structure may also include one more die connected to the first die and the second die in a D2D configuration and spanning at least one of the first dimension and the second dimension. The D2D structure may be configured to function as a single core processor for model parallelism across multiple dies of the D2D structure. The multiple dies of the D2D structure may be connected in a D2D folded mesh configuration or a D2D ring configuration.

[0173] The integrated circuit may also include a third die connected to the second die via a second D2D interface circuit in a D2D configuration, thereby forming a D2D structure with the first die and the second die, the second D2D interface connecting the second plurality of super channels of the second die to the third plurality of super channels of the third die, for streaming data between the second die and the third die in a first direction or a second direction. The integrated circuit may also include a fourth die connected to the third die via a third D2D interface circuit in a D2D configuration, thereby forming a D2D structure with the first die, the second die, and the third die, the third D2D interface connecting the third plurality of super channels of the third die to the fourth plurality of super channels of the fourth die, for streaming data between the third die and the fourth die in a first direction or a second direction. The first die, the second die, the third die, and the fourth die forming the D2D structure may be interconnected in a D2D folded grid configuration. The first die, the second die, the third die, and the fourth die connected in a D2D folded mesh configuration are configured to operate as a single core processor for model parallelism across the D2D folded mesh configuration.

[0174] The first die may include a first TSP having a first plurality of functional units connected at least in part via a first plurality of super channels, and the second die may include a second TSP having a second plurality of functional units connected at least in part via a second plurality of super channels, a high bandwidth memory, and a D2D interface circuit. The second die may be connected to the first die in a back-to-back configuration or in a face-to-face configuration to form a D2D structure.

[0175] Example Computer System Architecture

[0176] Fig.17Ais an abstract diagram of an example computer system suitable for implementing embodiments of the claimed disclosure, according to some embodiments.

[0177] exist Fig.17A 17, the structure of the computer system 1710 generally includes at least one computer 1714, which communicates with peripheral devices via a bus subsystem 1712. Typically, the computer includes a processor (e.g., a microprocessor, a graphics processing unit, or a digital signal processor), or its electronic processing equivalent, such as an ASIC or FPGA. Typically, the peripheral devices include a storage subsystem 1724 including a memory subsystem 1726 and a file storage subsystem 1728, a user interface input device 1722, a user interface output device 1720, and / or a network interface subsystem 1716. The input and output devices enable direct and remote user interaction with the computer system 1710. The computer system can use at least one output device and / or a network interface subsystem for important post-processing activities.

[0178] A computer system may be structured as a server, client, workstation, mainframe, personal computer (PC), tablet PC, set-top box (SIB), personal digital assistant (PDA), cellular phone, smart phone, web appliance, rack-mounted "blade", kiosk, television, game station, network router, switch or bridge, or any data processing machine with instructions specifying actions to be taken by the machine. As used herein, the term "server" refers to a computer or processor that typically performs processes for another computer or processor and sends data and information to another computer or processor.

[0179] Computer systems are typically structured in part using at least one operating system program, such as MICROSOFT WINDOWS, APPLE MACOS and IOS, GOOGLE ANDROID, Linux and / or Unix. Computer systems typically include a basic input / output system (BIOS) and processor firmware. The processor uses the operating system, BIOS and firmware to structure and control any subsystems and interfaces connected to the processor. Example processors that implement these operating systems include: Pentium, Itanium and Xeon processors from Intel; Opteron and Athlon processors from AMD (Advanced Microelectronics); Graviton processors from Amazon; POWER processors from IBM; SPARC processors from ORACLE; and ARM processors from ARM Holdings.

[0180] Any implementation of the present disclosure is not limited to electronic digital logic computers structured with programs, nor is it limited to electronic programmable devices. For example, the claimed implementations may use optical computers, quantum computers, analog computers, etc. In addition, when referring to only a single computer system or a single machine, the use of the singular form of these terms may also refer to any configuration of computer systems or machines that use the processes alone or in combination. Due to the ever-changing nature of computers and networks, in Fig.17A The description of computer system 1710 depicted in FIG. 1 is intended to be exemplary only. Many other configurations of computer system 1710 may be used in addition to Fig.17A More components than the computer system depicted in .

[0181] The network interface subsystem 1716 provides an interface to an external network, including an interface to a communication network 1718, and is coupled to corresponding interface devices in other computer systems or machines via the communication network 1718. The communication network 1718 may include many interconnected computer systems, machines, and physical communication connections (represented by "links"). These communication links may be wired links, optical links, wireless links (e.g., using WiFi or Bluetooth protocols), or any other physical device for information communication. The communication network 1718 may be any suitable computer network, such as a wide area network such as the Internet, and / or a local to wide area network such as Ethernet. The communication network is wired and / or wireless, and many communication networks use encryption and decryption processes, such as available with a virtual private network. The communication network uses one or more communication interfaces that receive data from other systems and send data to other systems. Implementations of the communication interface typically include an Ethernet card, a modem (e.g., telephone, satellite, cable TV, or Integrated Services Digital Network (ISDN)), a (asynchronous) Digital Subscriber Line (DSL) unit, a FireWire interface, a Universal Serial Bus (USB) interface, etc. The communication algorithm ("protocol") may be specified using one or more of the following communication languages: for example, the Hypertext Transfer Protocol (HTTP), the Transmission Control Protocol / Internet Protocol (TCP / IP), the Real-time Transport Protocol / Real-time Streaming Protocol (RTP / RTSP), the Internetwork Packet Exchange (IPX) protocol, and / or the User Datagram Protocol (UDP).

[0182] User interface input device 1722 can comprise alphanumeric keyboard, keypad, pointing device such as mouse, trackball, toggle switch, touch pad, stylus, graphic tablet, such as the optical scanner of bar code reader, touch screen electronic device of display device, audio input device such as speech recognition system or microphone, eye gaze recognition, brain wave pattern recognition, optical character recognition system and other types of input devices.These devices are connected to computer system by wire or wirelessly.Usually, term " input device " represents that data and information are transferred to the equipment and process of all possible types in computer system 1710 or on communication network 1718.User interface input device usually enables user to select to appear in some types of user interface output device such as object, icon, text etc. on display subsystem.

[0183] The user interface output device 1720 may include a display subsystem, a printer, a fax machine, or a non-visual communication device such as an audio and tactile device. The display subsystem may include a CRT, a flat panel device such as an LCD, an image projection device, or some other device for creating visible stimuli, such as a virtual reality system. The display subsystem may also provide non-visual stimuli, such as via an audio output, a scent generation, or a tactile / tactile output (e.g., a vibration and force) device. Generally, the term "output device" refers to all possible types of devices and processes that transmit data and information from the computer system 1710 to a user or another machine or computer system. These devices are connected to the computer system by wire or wirelessly. Note that some devices transmit data and information to and from the computer, for example, a tactile device that generates vibrations and forces on the user's hand while also combining sensors to measure the position and movement of the hand. The technical application of ergonomics and semiotics science is used to improve the efficiency of user interaction with any process and computer disclosed herein, such as any interaction related to the design and manufacture of circuits using any of the above-mentioned input or output devices.

[0184] The memory subsystem 1726 typically includes several memories, including a main RAM 1730 (or other volatile storage device) for storing instructions and data during program execution and a ROM 1732 in which fixed instructions are stored. The file storage subsystem 1728 provides persistent storage for program and data files and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, a flash memory such as a USB drive, or a removable media cartridge. If the computer system 1710 includes an input device that performs optical character recognition, text and symbols printed on a physical object (e.g., paper) may be used as a device for storing program and data files. Databases and modules used by some embodiments may be stored by the file storage subsystem 1728.

[0185] The bus subsystem 1712 provides a device for transferring data and information between the various components and subsystems of the computer system 1710. Although the bus subsystem 1712 is depicted as a single bus, alternative implementations of the bus subsystem may use multiple buses. For example, a main memory using RAM may communicate directly with a file storage system using a DMA system.

[0186] Fig. 17B is another abstract diagram of a computer system suitable for implementing embodiments of the claimed disclosure, according to some embodiments. Fig. 17B Memory 1740 is depicted, such as non-transitory processor-readable data and information storage media associated with file storage subsystem 1728 and / or network interface subsystem 1716 (e.g., via bus subsystem 1712), and may include data structures that specify circuit designs. Memory 1740 may be a hard disk, floppy disk, CD-ROM, optical media, removable media cartridge, or any other medium that stores computer-readable data (e.g., text and symbols on a physical object (e.g., paper) that can be processed by an optical character recognition system) in volatile or non-volatile form. Programs transferred to and from the processor from such memory may be converted into physical signals that propagate through a medium (e.g., a network, connector, wire, or circuit trace as electrical impulses); or through a medium (e.g., space or atmosphere as acoustic signals or as electromagnetic radiation in which the wavelength of the electromagnetic spectrum is longer than infrared light).

[0187] Additional Example Computing Systems

[0188] Fig.18 is a block diagram showing components of an example computing machine capable of reading instructions from a computer-readable medium and executing them in a processor (or controller) according to an embodiment. The computer described herein may include Fig.18 The single computing machine, virtual machine, including Fig.18 A distributed computing system of multiple nodes of the computing machine shown, or any other suitable arrangement of computing devices. The computers described herein may be used by any of the elements described in the previous figures to perform the functions described.

[0189] By way of example, Fig.18A diagrammatic representation of a computing machine in the example form of a computer system 1800 is depicted within which instructions 1824 (e.g., software, program code, or machine code) may be stored in a computer-readable medium for causing the machine to perform any one or more of the processes discussed herein. In some embodiments, the computing machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0190] Fig.18 The structure of the computing machine described in the figure may correspond to any software, hardware, or combination of components shown in the above figures. For example, the computing machine may be a tensor streaming processor designed and manufactured by GROQ, Inc. of Mountain View, California, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a network device, a network router, an Internet of Things (IoT) device, a switch or a bridge, or any machine capable of executing instructions 1824 that specify actions to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines that execute instructions 1824 individually or collectively to perform any one or more of the methods discussed herein.

[0191] The example computer system 1800 includes one or more processors (generally processor 1802) (e.g., central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), one or more application specific integrated circuits (ASIC), one or more radio frequency integrated circuits (RFIC), or any combination thereof), main memory 1804 and static memory 1806, which are configured to communicate with each other via bus 1808. The computer system 1800 may also include a graphics display unit 1810 (e.g., plasma display panel (PDP), liquid crystal display (LCD), projector, or cathode ray tube (CRT)). The computer system 1800 may also include an alphanumeric input device 1812 (e.g., keyboard), a cursor control device 1814 (e.g., mouse, trackball, joystick, motion sensor or other pointing tool), a storage unit 1816, a signal generating device 1818 (e.g., speaker), and a network interface device 1820, which are also configured to communicate via bus 1808.

[0192] The storage unit 1816 includes a computer-readable medium 1822 on which are stored instructions 1824 embodying any one or more of the methods or functions described herein. The instructions 1824 may also reside, completely or at least partially, within the main memory 1804 or within the processor 1802 (e.g., within a cache memory of the processor). Thus, during execution of the computer system 1800, the main memory 1804 and the processor 1802 may also constitute computer-readable media. The instructions 1824 may be transmitted or received over a network 1826 via the network interface device 1820.

[0193] Although the computer-readable medium 1822 is shown as a single medium in the example implementation, the term "computer-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instructions 1824). The computer-readable medium 1822 may include any medium capable of storing instructions (e.g., instructions 1824) to be executed by a machine and cause the machine to perform any one or more of the methods disclosed herein. The computer-readable medium 1822 may include, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media. The computer-readable medium 1822 does not include temporary media such as signals or carrier waves.

[0194] Additional considerations

[0195] The disclosed configuration may have the following benefits and advantages, including, for example: by separating the functionality of the processor into specialized functional units, and configuring the timing of data and instructions to each functional unit, each unit is able to operate on received data based on the known timing between the received data and the instructions, thereby achieving more efficient data flow. Because the compiler of the processor is hardware-aware, it is able to configure an explicit schedule for the processor that indicates how and when instructions and data operands are sent to different blocks of the processor. By considering the timing of received instructions and data, data can be transferred between blocks of the processor without unnecessary metadata, thereby improving transfer efficiency. In addition, by separating the transmission of data and instructions, instructions can be iterated and looped independently of the received data operands.

[0196] In addition, since each computing element of the processor is dedicated to a specific function (e.g., MEM, VXM, MXM, SXM), the amount of instructions that need to be processed by the computing element can be reduced. For example, certain computing elements (e.g., computing elements in an MXM functional slice) can be configured to perform a limited set of operations on any received data. Therefore, these computing elements can be able to operate without receiving explicit instructions or only receiving intermittent instructions or limited instructions, thereby possibly simplifying the operation of the processor. For example, data operands read from memory can be intercepted by multiple functional slices when the data is transmitted across the data channel, allowing multiple operations to be performed on the data in a more efficient manner.

[0197] In operation, the host computer programs the DMA engine to actually transfer the data, all of which is also coordinated by the runtime layer. Specifically, the IDU transfers 320-byte vectors from PCIe-Gen4 32 bytes per core-clock cycle (e.g., nominally 900Mhz). Therefore, the 320-element vector arrives in a period of 10 cycles and is placed in multiple streams that move to the MEM. The incoming stream flows on S24-31 (the upper 8 streams), and the MEM performs a "write" from it to commit the vector to the SRAM. Therefore, PCI reception includes: (i) receiving data from the PCI interface; (ii) writing the vector to the specified functional slice of the MEM.

[0198] The above description of the embodiments of the present disclosure has been presented for illustrative purposes; it is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible based on the above disclosure.

[0199] Some parts of this specification sheet describe the implementation of the present disclosure in terms of algorithms and symbolic representations of operations on information. Technicians in the field of data processing typically use these algorithmic descriptions and representations to effectively convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it has also been proven that, without loss of generality, it is convenient to sometimes refer to these operational arrangements as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0200] Any of the steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium, the computer-readable medium containing computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0201] Embodiments of the present disclosure may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that employs a plurality of processor designs to increase computing power.

[0202] Some embodiments of the present disclosure may also be directed to a system including a processor (e.g., a tensor stream processor or an artificial intelligence processor), at least one computer processor (e.g., a host server), and a non-transitory computer-readable storage medium. The storage medium may store computer-executable instructions that, when executed by a compiler running on at least one computer processor, cause the at least one computer processor to be operable to perform the operations and techniques described herein.

[0203] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and it may not be selected to describe or define the subject matter of the present invention. Accordingly, it is intended that the scope of the present disclosure be limited not by this detailed description, but by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not to limit, the scope of the present disclosure, which is set forth in the appended claims.

Claims

1. An integrated circuit, comprising: a first tube core; as well as A second die, wherein the second die is connected to the first die in a D2D configuration via a die-to-die (D2D) interface circuit to form a D2D structure with the first die, wherein the D2D interface connects a first plurality of super channels of the first die with a second plurality of super channels of the second die to stream data between the first die and the second die along a first direction or a second direction orthogonal to the first direction.

2. The integrated circuit according to claim 1, wherein: The D2D interface circuit includes a plurality of bidirectional interface slices, and a pair of the bidirectional interface slices are connected to corresponding super channels of a second plurality of super channels of the second die.

3. The integrated circuit according to claim 1, wherein: The D2D interface circuitry includes a D2D core interface circuitry connected to a corresponding subset of a first plurality of super channels of the first die.

4. The integrated circuit according to claim 1, wherein: The D2D interface circuit includes a plurality of D2D interface groups, each of the plurality of D2D interface groups connecting a cluster of the first plurality of super channels to a cluster of the second plurality of super channels.

5. The integrated circuit according to claim 4, wherein: A size of each of the plurality of D2D interface groups along the first direction matches a size of a cluster of the second plurality of super channels along the first direction.

6. The integrated circuit according to claim 4, wherein: Each of the plurality of D2D interface groups includes a plurality of D2D interface macros, and each of the plurality of D2D macros includes a corresponding bidirectional interface slice of a plurality of bidirectional interface slices.

7. The integrated circuit according to claim 1, wherein: The D2D structure also includes one more die connected to the first die and the second die in the D2D configuration and spanning at least one of the first dimension and the second dimension.

8. The integrated circuit according to claim 7, wherein: The D2D fabric is configured to operate as a single core processor for model parallelism across multiple dies of the D2D fabric.

9. The integrated circuit of claim 8, wherein: The multiple dies of the D2D structure are connected in a D2D folded grid configuration or a D2D ring configuration.

10. The integrated circuit of claim 1 , further comprising: a third die, the third die being connected to the second die in the D2D configuration via a second D2D interface circuit to form the D2D structure with the first die and the second die, the second D2D interface connecting the second plurality of super channels of the second die to the third plurality of super channels of the third die to stream data between the second die and the third die along the first direction or the second direction.

11. The integrated circuit of claim 10, further comprising: A fourth die, the fourth die is connected to the third die in the D2D configuration via a third D2D interface circuit to form the D2D structure with the first die, the second die and the third die, the third D2D interface connecting the third multiple super channels of the third die to the fourth multiple super channels of the fourth die to stream data between the third die and the fourth die along the first direction or the second direction.

12. The integrated circuit of claim 1, wherein: The first die, the second die, the third die, and the fourth die forming the D2D structure are connected to each other in a D2D folded grid configuration.

13. The integrated circuit of claim 11, wherein: The first die, the second die, the third die, and the fourth die connected in the D2D folded mesh configuration are configured to operate as a single core processor for model parallelism across the D2D folded mesh configuration.

14. The integrated circuit of claim 1, wherein: The first die includes a first tensor streaming processor (TSP) having a first plurality of functional units connected at least in part via the first plurality of superchannels; as well as The second die includes a second TSP having a second plurality of functional units connected at least in part via the second plurality of super channels, a high bandwidth memory, and the D2D interface circuit.

15. The integrated circuit of claim 1, wherein: The second die is connected to the first die in a back-to-back (B2B) configuration or a face-to-face (F2F) configuration to form the D2D structure.

16. A method comprising: Initiating, by a compiler, issuing a plurality of instructions for execution by a plurality of processing units across a first die and a second die, the second die being connected to the first die in a D2D configuration via a die-to-die (D2D) interface circuit to form a D2D structure with the first die; as well as The compiler initiates streaming of data between first multiple super channels of the first die and second multiple super channels of the second die via the D2D interface circuit along a first direction or a second direction orthogonal to the first direction to execute the multiple instructions.

17. The method according to claim 16, further comprising: Streaming of data in at least one of the first direction and the second direction across a plurality of dies connected in the D2D configuration via a plurality of D2D interface circuits and forming the D2D structure is initiated by the compiler.

18. The method according to claim 17, further comprising: The plurality of dies forming the D2D structure are configured, by the compiler, to function as a single core processor for model parallelism across the plurality of dies of the D2D structure.

19. A non-transitory computer-readable storage medium comprising computer-executable instructions stored thereon, which when executed by a compiler running on at least one computer processor, cause the at least one computer processor to: Initiate issuing a plurality of instructions for execution by a plurality of processing units across a first die and a second die, the second die being connected to the first die in a D2D configuration via a die-to-die (D2D) interface circuit to form a D2D structure with the first die; and The streaming of data between the first plurality of super channels of the first die and the second plurality of super channels of the second die via the D2D interface circuit along a first direction or a second direction orthogonal to the first direction is initiated to execute the plurality of instructions.

20. The non-transitory computer readable storage medium of claim 19, wherein: The instructions further cause the computer processor to: initiating streaming of data across a plurality of dies in at least one of the first direction and the second direction, the plurality of dies being connected in the D2D configuration via a plurality of D2D interface circuits and forming the D2D structure; and The plurality of dies forming the D2D structure are configured to function as a single core processor for model parallelism across the plurality of dies of the D2D structure.

Images