Processor, power grid routing method and related computer-readable storage medium

Abstract

The embodiment relates to a power grid distribution for a deterministic processor. The deterministic processor includes: a plurality of functional slices; a plurality of data transmission channels for transmitting data across the functional slices along a first spatial dimension; and a plurality of instruction control units (ICUs). An instruction in a subset of the ICUs includes a functional slice-specific opcode and is transmitted along a second spatial dimension orthogonal to the first spatial dimension to a corresponding functional slice. A power grid of metal traces is distributed across the first and second spatial dimensions to supply power to the functional slices and the ICUs. At least a portion of the metal traces is routed as discontinuous stubs along the first or second spatial dimension.

Description

Technical Field

[0001] This invention relates generally to a processor architecture, and more specifically to a power grid distribution for a tensor stream processor. Prior Technology

[0002] Over the past decade, data center operators have installed multi-core processor systems in warehouse-scale computers. These systems, ranging from tens to thousands of processing cores, vary widely in form and function and are used in powerful graphics processing units (GPUs), tensor processing units (TPUs), and field-programmable gate arrays (FPGAs), many of which utilize data network controllers for efficient remote memory access. These systems are commonly used to accelerate the performance of deep neural network training and inference (the application of trained neural networks) for widespread commercial use, such as convolutional neural networks (CNNs) for recommendation algorithms, computer vision and image classification for product search, and recurrent neural networks (RNNs) for natural language processing in user interfaces. The increased computational demands of these models (many involving very large amounts of vector and matrix calculations) when used with millions of consumers and billions of transactions have become a catalyst for a resurgence of architectural innovation.

[0003] In a conventional chip multiprocessor (CMP), the processing cores use a chip-on-a-chip network interconnect to exchange data among all processing cores. In this simple load-store model, a set of general-purpose data registers is used as intermediate storage between the main memory system and the processor cores, which may contain arithmetic logic units (ALUs) that operate on data. Instructions are dispatched to each core and executed by the local integer or floating-point processing module, while intermediate results are stored in the general-purpose registers. This load-store architecture moves data (also called "operands") and operation results between the registers and the main memory. Instruction execution is typically carried out in several stages: 1) instruction fetch; 2) instruction decoding; 3) execution on the ALU; 4) memory read; and 5) memory write to update the results in the registers.

[0004] However, CMP workloads are constantly increasing in both size and complexity, placing stringent demands on traditional CMP architectures in terms of scalability, performance, and availability. Efficiently managing these demands requires sufficient on-chip ALUs for vector and matrix computations, which operate at near-peak performance levels throughout program execution. Unfortunately, the hardware complexity from the added ALUs and other components substantially increases scheduling difficulties within these architectures. Consequently, runtime stalls are difficult to prevent, hindering system performance. Furthermore, while architectural enhancements (such as caches, branch predictors, and prefetchers) greatly improve performance, they do not limit worst-case performance. Summary of the Invention

[0005] Embodiments of this invention relate to an integrated circuit having one or more deterministic processors (e.g., tensor stream processors (TSPs) or artificial intelligence processors), each having a functional slice architecture. In some embodiments, each deterministic processor is configured to process a machine learning model. Each deterministic processor is divided into a plurality of functional units organized into a plurality of functional slices. Each functional slice is configured to perform a specific function within the deterministic processor, and may include a memory functional slice (MEM) for storing computational metadata, an arithmetic functional slice for performing operations (e.g., vector processing, matrix manipulation) on the received computational metadata, and / or the like. The functional units of the deterministic processor are configured to stream computational metadata across a first (e.g., time) dimension in a direction indicated in a corresponding instruction, and to receive instructions across a second (e.g., spatial) dimension. A compiler for the deterministic processor knows the hardware configuration of the deterministic processor and the timing of the configuration data and instruction streams, such that corresponding data and instructions intersect at each computational element at a predetermined time. The functional slices of this deterministic processor can operate on a set of data channels in a Single Instruction Multiple Data (SIMD) manner. This set of data channels may be referred to herein as a "superlane" and represents one cross-section of all functional slices on a processor chip.

[0006] Embodiments of the present invention relate to a power grid distribution for a deterministic processor (e.g., a TSP). The deterministic processor includes: a plurality of functional slices; a plurality of data transmission channels for transmitting data across the functional slices along a first spatial dimension; a plurality of instruction control units (ICUs), wherein one instruction in each subset of the ICUs includes a functional slice-specific opcode and is transmitted along a second spatial dimension orthogonal to the first spatial dimension to a corresponding functional slice; and a power grid of metal traces distributed across the first and second spatial dimensions for supplying power to the functional slices and the ICUs, wherein at least a portion of the metal traces is discontinuous along the first or second spatial dimension.

[0007] A further embodiment of the present invention relates to a procedure (or method) for wiring a power grid for a deterministic processor (e.g., a TSP). The procedure includes: wiring a power grid of metal traces across a first spatial dimension and a second spatial dimension orthogonal to the first spatial dimension for supplying power to a plurality of functional slices and a plurality of ICUs of the deterministic processor; and wiring at least a portion of the metal traces as discontinuous stubs along the first spatial dimension or the second spatial dimension, wherein a plurality of data transmission channels of the deterministic processor are configured to transmit data along the first spatial dimension across the functional slices, and one instruction in each subset of the ICUs includes a functional slice-specific opcode and is configured to transmit data along the second spatial dimension to a corresponding functional slice.

[0008] A further embodiment of the present invention relates to a non-transitory computer-readable storage medium including computer-executable instructions stored thereon, which, when executed by a computer processor, cause the computer processor to: route a power grid of one of the metal traces across a first spatial dimension and a second spatial dimension orthogonal to the first spatial dimension for supplying power to a plurality of functional slices and a plurality of ICUs of the deterministic processor; and route at least a portion of the metal traces as discontinuous stubs along the first spatial dimension or the second spatial dimension, wherein a plurality of data transmission channels of the deterministic processor are configured to transmit data along the first spatial dimension across the functional slices, and one instruction in each subset of the ICUs includes a functional slice-specific opcode and is configured to transmit along the second spatial dimension to a corresponding functional slice. Simple Explanation of the Diagram

[0009] Figure 1A illustrates a configuration of one of the functional slices in a Tensor Stream Processor (TSP) according to one of some embodiments.

[0010] Figure 1B illustrates an example TSP architecture according to one of some embodiments.

[0011] Figure 1C illustrates the organization and data flow within a column of a TSP according to one of some embodiments.

[0012] Figure 2 depicts a streaming register of a TSP according to one of some embodiments, which are numbered to show their positions among functional slices within a hyperchannel.

[0013] Figure 3 illustrates a die photograph of one of the TSP and one of the ASIC implementations according to one of some embodiments.

[0014] Figure 4 illustrates an example of a tile having a continuous metallic trace disposed on a TSP, according to some embodiments.

[0015] Figure 5 illustrates one of the electrical grid blocks of metal traces (some of which are discontinuous) placed on a TSP according to some embodiments.

[0016] Figure 6 illustrates a cross-section of an example of a metal layer in a TSP connected by a through-hole connector according to some embodiments.

[0017] Figure 7 is a flowchart illustrating one method of wiring for a power grid having an integrated circuit with a TSP, according to some embodiments.

[0018] Figure 8A is an abstract diagram of an example computer system suitable for commercial use according to some embodiments of the present invention.

[0019] Figure 8B is another abstract diagram of an example computer system suitable for commercial use of embodiments of the present invention, according to some embodiments.

[0020] Figure 9 illustrates an additional instance of a computing machine in commercial use according to some embodiments.

[0021] The figures are for illustrative purposes only and depict embodiments of the invention. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention or its esteemed benefits as described herein. Implementation

[0022] Cross-reference to related applications

[0023] This application claims the right and priority of U.S. Provisional Patent Application No. 63 / 183,158, filed May 3, 2021, entitled "Integrated Circuit Power Distribution Apparatus and Method," the entire contents of which are incorporated herein by reference.

[0024] The figures and the following description are for illustrative purposes only and relate to preferred embodiments. It should be noted that, from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as feasible alternatives that can be employed without departing from the principles claimed.

[0025] Several embodiments, examples of which are illustrated in the accompanying drawings, will now be discussed in detail. It should be noted that similar or identical element symbols may be used in the drawings and may indicate similar or identical functionality wherever possible. The figures depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. Introductory Overview

[0026] This discloses a configuration comprising an integrated circuit having one or more deterministic stream processors (e.g., tensor stream processors (TSPs) or artificial intelligence processors). Each deterministic stream processor may have a functional slice architecture. In some embodiments, each deterministic stream processor is configured to process a machine learning model. Each deterministic stream processor is divided into a plurality of functional units. The functional units are organized into a plurality of functional slices. Each functional slice is configured to perform a specific function within the deterministic stream processor. The deterministic stream processor may include memory functional slices (MEMs) for storing computational metadata, arithmetic functional slices for performing operations (e.g., vector processing, matrix manipulation) on the received computational metadata, and / or the like. The functional units of the deterministic stream processor are configured to stream computational metadata across a first (e.g., time) dimension in a direction indicated in a corresponding instruction, and to receive instructions across a second (e.g., spatial) dimension. The compiler for a deterministic streaming processor knows the hardware configuration of the deterministic streaming processor and the timing of the configuration data and instruction streams, so that the corresponding data and instructions intersect at each arithmetic element at a predetermined time. Each functional slice of a deterministic streaming processor can operate on a set of data channels in a Single Instruction Multiple Data (SIMD) manner. This set of data channels may be referred to herein as a "hyperchannel" and represents a cross-section of all functional slices on a processor chip.

[0027] The disclosed embodiments relate to a deterministic stream processor having a functional slicing architecture. In some embodiments, the deterministic stream processor may include a tensor stream processor (TSP) having a functional slicing architecture, which can be used for hardware-accelerated machine learning (ML) applications.

[0028] A deterministic streaming processor (e.g., a TSP) comprises a plurality of "operational elements," each corresponding to a functional unit within the processor. The on-chip memory and on-chip network (NoC) of the deterministic streaming processor architecture are integrated to provide storage for both operations and results, and can act as a conduit for transferring operation and / or result data to / from the functional units of the deterministic streaming processor. The operational elements of the deterministic streaming processor are divided among different functionalities (e.g., memory, arithmetic operations, etc.) and organized into functional slices that operate on multidimensional data (e.g., tensors). For example, each functional slice is composed of horizontally and vertically adjacent operational elements. The number of operational elements and the granularity of each operational element can be selected to utilize the underlying technology on which it is built. The number of operational elements (N) and the word granularity (M) of a memory (e.g., static random access memory (SRAM)) together produce the machine's vector length (VL).

[0029] In some embodiments, each functional slice of the deterministic streaming processor operates independently and receives instructions from an instruction control unit (ICU). The ICU can pass instructions to a first arithmetic element of one of the functional slices, which then propagate along the functional slice to the remaining arithmetic elements of the functional slice in a first time dimension of the deterministic streaming processor. Alternatively, data arithmetic elements for storage and / or processing can be passed between different functional slices of the deterministic streaming processor in a second spatial dimension perpendicular to the first time dimension. Thus, the data stream and instruction stream of the deterministic streaming processor are separate streams.

[0030] In some embodiments, a compiler for a deterministic streaming processor is aware of the hardware configuration of the deterministic streaming processor and synchronizes the timing of data and instruction streams, such that corresponding data and instructions are received at each arithmetic element in a predetermined timing relationship (e.g., during the same clock cycle separated by a predetermined delay). In some embodiments, the predetermined timing relationship may be based on the hardware of the deterministic streaming processor, an instruction type, and / or the like. Because the timing relationship between data and instructions is known to the compiler, the arithmetic metadata received by an arithmetic element does not contain any post-processor information indicating what the data is intended for or where it is consumed. Instead, each arithmetic element receives instructions and executes the instructions based on the current data held in a register associated with the arithmetic element, according to the predetermined timing. This allows data and instructions to flow more efficiently through the deterministic streaming processor.

[0031] Embodiments of the present invention relate to a power grid distribution for a deterministic streaming processor (e.g., a TSP). The deterministic streaming processor includes a plurality of functional slices and a plurality of data transmission channels for transmitting data across the functional slices along a first spatial dimension. The deterministic streaming processor further includes a plurality of ICUs, wherein one instruction in a subset of the ICUs includes a functional slice-specific opcode and is transmitted along a second spatial dimension orthogonal to the first spatial dimension to a corresponding functional slice. The deterministic streaming processor further includes a power grid of metal traces distributed across the first and second spatial dimensions for supplying power to the functional slices and ICUs. At least a portion of the metal traces is implemented as discontinuous stubs along the first or second spatial dimension. Tensor Stream Processor Architecture Overview

[0032] According to embodiments of the present invention, a deterministic streaming processor plane includes a TSP, for example, available from GROQ Corporation of Mountain View, California. It should be understood that while many embodiments described herein use a TSP as a preferred processor, other deterministic streaming processors may be used in commercial applications. Figure 1A illustrates one configuration of a functional slice in a TSP according to some embodiments.

[0033] Certain core architectural elements distinguish the TSP from the GPU and accelerators. In a conventional chip multiprocessor (CMP), each "processing element" is an independent core that uses a chip-on-a-network interconnect to exchange data between cores. Instruction execution is carried out through several stages: (i) instruction fetch (IF); (ii) instruction decode (ID); (iii) execution on the arithmetic logic unit (ALU) (EX); (iv) memory access (MEM); and (v) write-back (WB) to update the results in the general purpose register (GPR).

[0034] Compared to conventional multi-core architectures (where each computing element is a heterogeneous set of functional units but homogeneous globally), TSP reverses this to have local functional homogeneity but heterogeneity over the entire chip (global domain). More specifically, TSP reorganizes the homogeneous two-dimensional mesh of the cores into a functional slice microarchitecture as shown in Figure 1A. In this approach, each computing element performs a specific function and is vertically stacked as a specific "functional slice" along one dimension (e.g., the Y dimension) of the two-dimensional chip-on-chip mesh. TSP decomposes the basic elements of conventional multi-core architectures according to their respective functions: instruction control and dispatch (e.g., via the instruction control unit (ICU)), memory (MEM), integer (INT) arithmetic, floating-point unit (FPU) arithmetic, and network (NET) interface, as shown by the functional slice label at the top of Figure 1A. Each column of the two-dimensional chip-on-chip mesh contains a cross-section of all functional slices.

[0035] In this organization, each functional slice is independently controlled by a sequence of instructions specific to its on-chip function. For example, a MEM functional slice supports reading and writing, but not necessarily addition or multiplication, which are typically performed in arithmetic functional slices (e.g., Vector Execution Module (VXM) and Matrix Execution Module (MXM) functional slices) used for some typical machine learning (ML) algorithms (such as linear regression algorithms).

[0036] All functional slices of the arithmetic units execute the same instruction stream—Single Instruction Multiple Data (SIMD) instructions. Therefore, common instruction decoding and dispatch logic can be decomposed into its own arithmetic units (e.g., ICU), and the normal instruction execution pipeline is broken down into two regions: (i) instruction fetch, decode, and pack; and (ii) operand fetch, execution, and write-back. This method decouples the memory subsystem from the functional units that fetch its equivalent operands and store the results.

[0037] In some embodiments, each functional slice implements, for example, a 20-level vector pipeline across one of the arithmetic elements of each functional slice, wherein each arithmetic element produces 16 elements of a maximum vector length of 320 elements. This organization naturally decomposes the instruction flow in the vertical dimension and the data flow in the horizontal dimension as data flows through different functional types. With this processor organization, instruction execution is performed by different arithmetic elements: instruction fetching and decoding in the ICU; and operand decoding, execution, and write-back at each arithmetic element of the functional slice when (vertical flow) dispatched instructions intersect with (horizontal flow) operand data operated on by dispatched instructions. It will be understood that the references to "vertical" and "horizontal" or "north," "south," "east," and "west," used in conjunction with the illustrations shown in the figures, are intended only to aid the reader's understanding and should not be construed as technical limitations.

[0038] Figure 1B illustrates an example TSP 100 according to one of some embodiments. TSP 100 may include memory and arithmetic units optimized for multiplying and adding input data with a set of weights (e.g., trained or in training) for machine learning applications (e.g., training or inference). For example, TSP 100 includes a VXM 110 for performing operations on vectors (i.e., one-dimensional arrays of values). Other components of the system are symmetrically arranged on either side of VXM 110 to optimize processing speed. For example, VXM 110 is adjacent to MEMs 111-112 and SXMs 113-114 to control the routing of data, data domains, and presentation controllers (or numerical interpretation modules (NIM)) 115-116 and MXMs 117-118. For example, an ICU 120 controls the data flow and operation execution across blocks 110-118. TSP 100 may further include communication circuitry, such as chip-to-chip (C2C) circuitry 123 to 124 and an external communication circuitry (e.g., PCIe) 121. For example, TSP 100 may further include a chip control unit (CCU) 122 to control startup operations, clock resets, and other low-level setting operations.

[0039] Figure 1C illustrates the organization and data flow within a column of a TSP 100 according to some embodiments. As shown in Figure 1C, each column of the two-dimensional on-chip grid of the TSP 100 contains a cross-section of all functional slices, such as an N×N array of MXMs (e.g., N=320) configured for both integer (INT) and floating-point (FP) values ​​(e.g., INT8 and FP16), S MEM functional slices (S=44), VXM functional slices (e.g., V=16) with V vector ALUs per channel, and SXM functional slices. In this organization, each functional slice is independently controlled by a sequence of instructions specific to its on-chip action, extracted by a corresponding array of ICUs (e.g., a total of I=144 ICUs). Conceptually, the functional slices are fixed, and data 130 flows across their respective computing elements. When data flows through a specific functional slice, each functional slice may, as needed, intercept data operations and compute a result (e.g., in the case of MXM and VXM), or move data between data transmission channels on the network (e.g., in the case of SXM and MEM). Instructions flow northward from the ICU to the functional slice, while data (operations and results) primarily flows eastward and westward between functional slices. Any data movement between channels within a vector utilizes the chip-to-network functional slice.

[0040] It should be noted that the directions “east, west, north, and south” are provided in this document for ease of discussion and relativity. Furthermore, the directions “east, west, north, and south” are used as one reference to illustrate the processing flow described herein and are not intended to limit us to a specific label of direction. For example, a north-south direction (i.e., along the vertical or Y-axis) can be reoriented to an east-west direction (i.e., along the horizontal or X-axis), and the principles currently used to describe east-west directions can be applied to the reoriented north-south direction. In another instance, where the directionality is not intended to be limited to the description according to the mentioned reference, the directionality can be referenced such that north-south is vertical and east-west is horizontal, and these principles will apply accordingly.

[0041] In one embodiment, 320 channels are superimposed on TSP 100, wherein each computing element in the on-chip grid operates in a SIMD manner on, for example, 16 channels. A 16-channel unit may be referred to herein as a "hyperchannel" and represents a cross-section of all functional slices on the chip. Thus, a hyperchannel can represent, for example, the minimum vector length (minVL) of 16 elements of the architecture. Similarly, a vertical composition of 20 blocks forming a functional slice can produce, for example, a maximum vector length (maxVL) of 20 × 16 = 320 functional units. Each of the 144 independent on-chip ICUs can issue one or more instructions per clock cycle. The compiler has explicit control over the program order in each instruction queue, for example, by generating a combined program 140 executed by the ICUs and functional slices. Each channel has 64 logic streams for moving operands or results on the chip; for example, 32 streams are east-facing and 32 streams are west-facing. The 220 MB of globally shared SRAM can deliver 32 bytes of streaming bandwidth per channel and low-latency access to model parameters. For example, MEM can read, and MXM can mount more than, for example, 100,000 weights into a 320×320 array (i.e., 320 channels × 320 functional units) in less than 30 clock cycles, including SRAM and on-chip network transit delay.

[0042] As shown in Figures 1B and 1C, the chip network is implemented as an X-dimensional and Y-dimensional grid of computational elements with XY-dimensional sequential routing. Each instruction specifies a first jump direction (east or west), thus the memory instruction semantics have both an address and a data flow direction. Streaming is routed in the X-dimensional direction via MEM 111 / 112 and in the Y-dimensional direction using SXM 113 / 114 permuters and channel shifters to vertically move data elements. The SXM 113 / 114 permuters implement a permutation function, which is a mathematical technique for determining the number of possible configurations in a set when the configuration order matters. Common mathematical problems involve selecting items from a set of items in only a specific order.

[0043] MEM 111 / 112 and SXM 113 / 114 provide deterministic routing of streaming data as it flows in the X and Y dimensions, respectively. With the TSP architecture 100, function slices interact with the data stream in a producer-consumer manner. That is, function slices consume computational units from the stream and produce results on a (potentially different) stream, like an assembly line operator (function slice) and a conveyor belt (stream).

[0044] Conceptually, functional slices are fixed, and data flows across computing elements, as shown in Figure 1C. When data flows through a functional slice, each computing element may, as appropriate, intercept data operations and perform a calculation (if the computing element includes an arithmetic logic unit (ALU)), or move data between channels on the network, provided that the computing element includes a switching element.

[0045] Streaming provides a stylized abstraction and serves as a conduit through which data flows between functional slices. Unlike GPR, functional slices operate on parallel data streams flowing east or west (horizontally) across the wafer. Horizontal streams carrying operands intercept vertical (northward) stream instructions (see Figure 1C) to perform an operation at an operand on a functional slice. A compiler accurately maintains the wafer's architectural state and uses this knowledge to ensure that instructions correctly intercept their (some) stream operands.

[0046] Streaming is implemented in hardware via a chip-wide stream register file. Streams are architecturally visible and transfer operands and results between functional slices. A common software pattern involves reading operand data from one or more MEM functional slices, which is then consumed and manipulated by a downstream arithmetic functional slice. The results of the operations are then generated onto another stream, which can be written back to memory or passed to subsequent arithmetic elements. For example, an operation Z = X + Y might require four instructions: Read S1, X and Read S2, Y are executed on two MEM functional slices, and these are directed inward toward an ALU functional slice to execute Add S1, S2, S3. Finally, the result is stored back to memory via Write S3, Z. A stream represents one set of N elements operated on by various functional slices in a SIMD manner.

[0047] By way of example, a TSP architecture makes several careful trade-offs at the hardware-software interface, thereby pushing the scheduling-related complexities into the compiler. Specifically, the compiler is responsible for precisely scheduling instructions to use the hardware correctly and efficiently. Sometimes, this may involve choosing one of several methods by which algorithms or meta-operations can be implemented on the hardware. Removing the control complexity of dynamic instruction scheduling for multi-problem execution units allows for a relatively small ICU, for example, less than 3% of the chip area.

[0048] The compiler accesses, for example, a 320-channel procedural abstraction superimposed on a TSP architecture (e.g., TSP 100 in Figure 1B or one of the TSP chips 300 in Figure 3), where each computing element in the on-chip grid operates in a SIMD manner on 16 channels. A 16-channel cell can be referred to as a "hyperchannel," which is a cross-section and the smallest operational granularity of all functional slices on the chip. Thus, a hyperchannel represents the minimum vector length minVL of 16 elements of the architecture. Similarly, a vertical composition of 20 blocks used to form a functional slice (see TSP chip 300 in Figure 3) produces a maximum vector length maxVL of 20 x 16 = 320 elements.

[0049] The compiler accesses, for example, 144 independent instruction sets (i.e., ICUs) on the chip: (a) six for the westbound MXM, comprising two independent two-dimensional MAC (multiply-accumulate) arrays; (b) 14 for the westbound SXM, used for intra-channel and inter-channel switching by reconfiguring vector elements; (c) 44 for the westbound MEM, comprising 44 parallel function slices of static random access memory (SRAM); (d) 16 for the VXM, comprising 16 vector ALUs per channel; (e) 44 for the eastbound MEM, comprising 44 parallel function slices of SRAM; (f) 14 for the eastbound SXM; and (g) six for the eastbound MXM, comprising two independent two-dimensional MAC arrays, each instruction set can issue one or more instructions per cycle, and the compiler has explicit control over the program order in each instruction set.

[0050] The compiler accesses, for example, 64 logic streams per channel. For instance, 32 logic streams are needed to operate on 16 minVL per channel to move operands or results on the wafer, with 32 streams going east and 32 streams going west, as shown in Figure 2.

[0051] In one embodiment, the compiler accesses, for example, a 220-megabyte global shared SRAM, delivering a streaming bandwidth of 32 bytes per channel and low-latency access to model parameters. For example, MEM is readable, and MXM can load 400 K weights into all four 320×320 arrays in fewer than 40 operating cycles, including SRAM and on-chip network transfer latency.

[0052] A stream is specified by both an identifier (0,...,31) and its direction. For example, in (28) specifies an inward stream 28, and out (24) indicates a stream 24 directed toward the outer edge of the wafer. The direction of a stream can be specified as inward (towards the wafer bisection) or outward (towards the outer edge of the wafer), or the direction can be specified as east or west, as shown in Figures 1C and 2.

[0053] The components of a hyperchannel are spatially organized as shown in Figure 1C. The TSP's instruction set architecture (ISA) defines instructions that span different functional regions. The partitioned global address space (PGAS), presented by the MEM functional slices, provides memory semantics for vectors to be addressed from SRAM and loaded into an architecture-visible stream, where one direction of the data flow is toward the functional slice intended to operate on them.

[0054] The first functional area (i.e., ICU) uses (several) IFetch instructions to provide explicit instruction fetching and uses Sync and Notify instructions to provide inter-slice synchronization to perform chip-wide barrier synchronization within the participating functional slices. A repetitive NOP (no-op) instruction allows for precise loop-by-loop control of inter-instruction delays. For example, when scheduling two operations A and B using an intermediate NOP, the compiler has precise loop control so that N loops separate them equally, e.g., OpA NOP(N) OpB.

[0055] The second functional area (i.e., VXM) consists of a 4x4 grid of one of the ALUs in each channel used for pointwise arithmetic operations.

[0056] The third functional area (i.e., MXM) consists of four independent two-dimensional MAC arrays that operate on data types such as INT8, FP16, or FP32.

[0057] On-chip data movement utilizes the Fourth Functional Area (SXM) for intra-channel and inter-channel switching by reconfiguring vector elements. SXM is analogous to the NET interface used for communication between cores in Figure 1A. MEM and SXM work together to enable XY-dimensional data movement across the chip network.

[0058] The fifth functional region (i.e., the eastern and western hemispheres of the on-chip MEM module) consists of 44 parallel MEM functional slices of SRAM, providing the memory access parallelism required to fully utilize the 32 streams in each east or west direction. Each functional slice provides 13 bits for physical addressing of a 16-bit memory word, with each byte mapped to a channel, for a total of 220 million bytes of on-chip SRAM.

[0059] An additional sixth functional area contains a C2C module configured to provide transmit and receive primitives for exchanging 320-byte vectors between a pair of TSP chips. One possible TSP implementation (see TSP die 300 in Figure 3) has, for example, a total external bandwidth of 16 × 4 × 30 Gbps × 2 directions = 3.84 Tb / s (megabytes per second) with a total of 16 × 4 links each operating at 30 Gbps. This external bandwidth can be flexibly partitioned to support high-radius interconnect networks for TSPs in large-scale systems. A host interface for Peripheral Component Interconnect Fast (PCIe) Gen4 can also be disposed in this module. The host interface provides a lightweight direct memory access (DMA) engine to place a model onto the TSP memory and provides an entry point for initiating model execution. The host interface also provides a general mechanism for passing interrupts to the host, which may be necessary, for example, in the event of a multi-bit memory error.

[0060] According to some embodiments, Table I provides a summary of one of the example instructions for each functional slice. Table I Instruction summary for each function slice instruction describe ICU NOP N Ifetch Sync Notify Config Repeat n, d No operation required; can be repeated N times with a delay of up to N loops. Fetch instructions from the streaming stream or local memory Stop at the head of the instruction dispatch queue and wait for barrier notification. Releasing pending barrier operations leads to the resumption of the instruction flow. Configure low power mode Repeat the previous instruction n times, with d loops between each repetition. MEM Read a, s Write a, s Gather s, map Scatter s, map Load the vector at address a into stream s. Store the contents of the stream s register to the main memory address a. Indirectly read the address pointed to by the map placed on the stream s. Indirectly store the stream s into the address in the map stream. VXM unary operation binary operationtype conversions ReLU TanH Exp RSqrt z = op x, performs pointwise operations on an operand x to produce a result z (e.g., mask, negate). z = x op y, uses two operands x and y to perform pointwise operations, resulting in a single result z (e.g., add, mul, sub). Convert a fixed point to a floating point and vice versa. The startup function for the rectifier linear unit is max(0,x). Hyperbolic Tangent - Startup Function exponentiation ex Reciprocal square root MXM LW IW ABC ACC Load the weights (LW) from the stream into the weight buffer. Install the weights (IW) from the stream or LW buffer into the 320×320 array. Startup buffer control (ABC) to initiate and coordinate the start-up process. Accumulation (ACC) is derived from MXM's INT32 or FP32 results. SXM Shift up / down N Permute map Distribute map Rotate stream Transpose sg16 Shift the stream up / down by up to N channels, and select between north / south shift vectors. One-to-one permutation of 320 inputs Output Reconfigure or copy data within a super channel (16 channels). Rotate n×n input data to produce n² output streams (n=3 or n=4) with all possible rotations. Transpose 16×16 elements to produce 16 output streams with columns and rows interchanged. C2C Deskew Send Receive Managing skew across quasi-synchronous links Send a 320-bit vector Receive a 320-bit vector and place it into main memory.

[0061] Instructions executed on different functional slices can be chained together to create more complex actions without writing intermediate results back to memory. This allows for efficient streaming processing with full bandwidth and minimal latency.

[0062] Machine learning algorithms typically operate on vectors of coefficients with a specified data type (e.g., INT8, FP16, etc.). These vectors can be interpreted as an abstraction of the underlying data, whose elements can be processed in a SIMD manner using the same operation. TSP operates on vectors, sometimes organized as rank-2 tensors, and relies on a graph-lowering compiler to convert higher-rank tensors to rank-2 tensors.

[0063] The TSP's stylized model is a producer-consumer model, where each functional slice acts as both a consumer and a producer of one or more streams. When a vector is read from a main memory, it is assigned a stream identifier (0, ..., 31) and a direction: east or west. Once the vector is read into a stream register, it becomes a stream and "flows" in a given direction in the following sense: given spatially adjacent functional slices at coordinates x0, x1, x2 (where spatial coordinates increase in the flow direction), then at a given time ti, the vector representing stream s1 at functional slice x1 can be accessed by that functional slice as an operand. Similarly, functional slices at x0 and x2 will access different stream values ​​in the same stream register. In the subsequent loop t i+1, the value s1 is propagated to the functional slice at x2; otherwise, in loop t, the value s1 is overwritten using a result r1 produced by the functional slice at x1. Similarly, at time ti, the stream value s0 consumed by the functional slice at coordinate x0 will be available for the functional slice at x1 in the next cycle ti+1 (the lack of x0 overwrites the value of time ti). Streaming operators are manipulated toward the functional slices that consume their equivalents and produce a resulting stream. The stream continuously flows across the wafer, thus serving as a means for functional slices to communicate with each other. Figure 2 provides a graphical depiction of the interleaving of functional units and stream registers that support this stylized model.

[0064] In the TSP programming model, an instruction is issued on a function slice at a given compiler scheduling time t and executed as a SIMD operation on a stream of operand vectors (e.g., up to 320 elements), producing a vector of the same length on the result stream. For example, at the microarchitectural level, the 320-element SIMD instruction is pipelined vertically across the operands in the function slice. That is, at scheduling time t, the instruction is issued to the bottom operand of the function slice, for example, the 16-element hyperpass corresponding to the operand / result vector. In subsequent operation loops, the instruction is propagated to the next operand north in the function slice, which then executes the instruction on the next 16-element hyperpass below the operand vector. This process continues loop by loop until, for example, all 20 operands in the function slice have been traversed. This combination of vertical instruction pipelined architecture, along with the need for precise timing overlap between operands and instructions, results in a spatial "interleaving" of SIMD operand and result data. Power grid distribution for deterministic processors

[0065] Embodiments of this invention relate to an integrated circuit having a power grid distributed across a first and second spatial dimension of the integrated circuit for supplying power to various functional elements (e.g., functional units) of the integrated circuit. At least a portion of the metal traces is routed as discontinuous stubs along the first or second spatial dimension. The integrated circuit may be a deterministic stream processor, such as a TSP available from GROQ, for example, the TSP 100 of Figure 1B or the TSP 300 of Figure 3.

[0066] High-level complexity and density integrated circuits (such as the TSP 100 or TSP 300) require significant power to operate. For example, typical integrated circuit operating voltages can range from approximately 1 V to 5 V. Providing this power necessitates the use of metal layers placed on the deterministic streaming processor to mitigate the operating voltage drop attributed to "IR drop," which represents the voltage reduction that occurs when current "I" flows through a conductor with a finite resistance "R."

[0067] Figure 4 illustrates an example of a grid block 400 having continuous metal traces disposed on an integrated circuit according to some embodiments. The integrated circuit may be a deterministic stream processor, such as a TSP 100 or a TSP 300. In order to supply power to the various active elements of the integrated circuit at a relatively consistent operating voltage, a grid structure depicted in Figure 4 may be used for at least a portion of the metal layers in the integrated circuit. The grid block 400 may also be referred to as a "grid block with rails".

[0068] Grid block 400 may represent a portion of a power supply grid disposed on an integrated circuit. While the power supply grid of an integrated circuit comprises metal traces belonging to a plurality of metal layers (e.g., more than ten metal layers) distributed across the first and second spatial dimensions (e.g., the X and Y axes) of the integrated circuit, grid block 400 represents a portion of a power supply grid having metal traces belonging to a pair of metal layers 405 and 410. Metal layer 405 may be a layer of a lower order relative to metal layer 410, and the metal traces of metal layer 405 may be spatially (e.g., along the Z-axis) positioned below the metal traces of metal layer 410. For example, metal layer 405 may be a metal layer of order N, and metal layer 410 may be a metal layer of order N+1, where, for example, N≥2.

[0069] As shown in Figure 4, multiple metal traces for one of the source power rails (or sink power rails) of metal layer 405 can be connected to a single metal trace for one of the source power rails (or sink power rails) of metal layer 410 via multiple through-hole connectors 415 along a third spatial dimension (e.g., along the Z axis) orthogonal to the first and second spatial dimensions (e.g., the X and Y axes). Similarly, multiple metal traces for one of the source power rails (or sink power rails) of metal layer 410 can be connected to a single metal trace for one of the source power rails (or sink power rails) of metal layer 405 via multiple through-hole connectors 415 along the third spatial dimension. Advantageously, the power grid block 400 can be made symmetrical about the X-axis, Y-axis, or both X and Y axes. In this way, the IR voltage drop of the power grid across the integrated circuit (e.g., across the first and second spatial dimensions of TSP 100 or TSP 300) can be made more regular and predictable. Alternatively, in some embodiments, the grid block 400 is asymmetrical about the X-axis, Y-axis, or both X and Y axes. For example, a grid density, usage, and power supply percentage may vary for different types of power supply to the grid block 400 (e.g., between VDD, VDDM, and VSS power supply rails in a MEM functional slice).

[0070] One of the source power rails in grid block 400 can be a positive (voltage or current) power rail, and one of the slot power rails in grid block 400 can be a grounded, neutral, or negative (voltage or current) power rail. An integrated circuit having grid block 400 can include two or more power rails for use in the various metal layers of grid block 400. Examples of power rails in grid block 400 can include a VDD power rail, VDDM power rail, VSS power rail, VDDIO power rail, VDDIO2 power rail, VDDREF power rail, VSSIO power rail, VDDanalog power rail, VSSanalog power rail, or some other type of power rail.

[0071] Figure 5 illustrates a power grid block 500 of metal traces disposed on an integrated circuit according to some embodiments, some of which are discontinuous. Power grid block 500 represents a portion of the full power grid of an integrated circuit having metal traces belonging to a pair of metal layers (i.e., metal layers 505 and 510). Metal layer 505 may be a lower-order layer relative to metal layer 510, and the metal traces of metal layer 505 may be spatially (e.g., along the Z-axis) positioned below the metal traces of metal layer 510. For example, metal layer 505 may be a metal layer of order M, and metal layer 510 may be a metal layer of order M+1, where, for example, M≥2 and M≠N. As can be observed from Figure 5, a portion of the metal traces of power grid block 500 (i.e., the metal traces belonging to metal layer 505) is discontinuous along one spatial dimension of the integrated circuit (e.g., along the X-axis) and may be referred to as a power stub 515. The grid block 500 can also be referred to as a "grid block with stubs". Therefore, the metal traces of the metal layer 505 are routed as discontinuous power supply stubs 515 along one spatial dimension of the integrated circuit.

[0072] Multiple metal traces (i.e., multiple power stubs 515) for a source power stub (or slot power stub) of metal layer 505 can be connected to a single metal trace for a source power rail (or slot power rail) of metal layer 510 along a third spatial dimension (e.g., along the Z axis) orthogonal to the first and second spatial dimensions (e.g., the X and Y axes) using multiple through-hole connectors 520. Similarly, multiple metal traces for a source power rail (or slot power rail) of metal layer 510 can be connected to a single metal trace (i.e., a single power stub 505) for a source power stub (or slot power stub) of metal layer 505 along the third spatial dimension using multiple through-hole connectors 520. Each power stub 515 of metal layer 505 has a shorter length along the first spatial dimension (e.g., along the X axis) compared to the length of a corresponding metal trace of metal layer 510 along the second spatial dimension (e.g., along the Y axis).

[0073] One power supply stub in grid block 500 can be a positive (voltage or current) power supply stub, and one slot power supply stub in grid block 500 can be a grounded, neutral, or negative (voltage or current) power supply stub. The integrated circuit may include two or more power supply stubs for use in one of a pair of metal layers in grid block 500. Examples of power supply stubs in grid block 500 may include a VDD power supply stub, a VDDM power supply stub, a VSS power supply stub, a VDDIO power supply stub, a VDDIO2 power supply stub, a VDDREF power supply stub, a VSSIO power supply stub, a VDDanalog power supply stub, a VSSanalog power supply stub, or some other type of power supply stub. Similarly, the integrated circuit may include two or more power rails for use in the other metal layer of the pair of metal layers in grid block 500. Examples of power supply rails in grid block 500 may include a VDD power supply rail, a VDDM power supply rail, a VSS power supply rail, a VDDIO power supply rail, a VDDIO2 power supply rail, a VDDREF power supply rail, a VSSIO power supply rail, a VDDanalog power supply rail, a VSSanalog power supply rail, or some other type of power supply rail.

[0074] It can be observed that using a power stub 515 in grid block 500 along one spatial dimension (e.g., along the X-axis) instead of a continuous track in grid block 400 requires an additional via 520 (i.e., an additional conductor along the Z-axis). However, the additional via 520 advantageously has a smaller resistance compared to the resistance of a continuous metal trace along that one spatial dimension (e.g., along the X-axis). The resistance of one via connector 520 along a third spatial dimension (e.g., along the Z-axis) is substantially smaller than the resistance of one metal trace in grid block 500 along a first spatial dimension (e.g., along the X-axis) or a second spatial dimension (e.g., along the Y-axis). The lower resistance of the via 520 reduces the IR voltage drop of the power grid compared to the resistance of the metal trace. Additionally, grid block 500 can be made symmetrical about the X-axis, Y-axis, or both X and Y axes. In this way, the IR drop of the power grid across the first and second spatial dimensions of the integrated circuit (e.g., TSP 100 or TSP 300) can be made more regular and predictable. Alternatively, in some embodiments, the grid block 500 is asymmetrical about the X-axis, Y-axis, or both X and Y axes. For example, a grid density, usage, and power supply percentage can vary for different types of power supply to the grid block 500 (e.g., between VDD, VDDM, and VSS power supply stubs or rails in a MEM functional slice).

[0075] The power grid of an integrated circuit (e.g., TSP 100 or TSP 300) may include at least one grid block 400 and at least one grid block 500. Metal traces of a subset of the metal layers in the integrated circuit may be distributed as one or more grid blocks 400 across the first and second spatial dimensions, while metal traces of the remaining metal layers in the integrated circuit may be distributed as one or more grid blocks 500 across the first and second spatial dimensions. In other words, the power grid of the integrated circuit may be implemented as a combination of at least one grid block 400 and at least one grid block 500. For example, in one embodiment, the power grid of the integrated circuit includes three grid blocks 500 (e.g., for metal layers 2 and 3, metal layers 6 and 7, metal layers 8 and 9) and five grid blocks 400 (e.g., for metal layers 4 and 5, metal layers 10 and 11, metal layers 12 and 13, metal layers 14 and 15, metal layers 16 and 17). Furthermore, the metal traces of a higher-order metal layer can be wider than the metal traces of a lower-order metal layer. For example, the metal traces of layers 12 and 13 are approximately 3 times wider than the metal traces of layers 3 and 4, the metal traces of layers 14 and 15 are approximately 6 times wider than the metal traces of layers 3 and 4, and the metal traces of layers 16 and 17 are approximately more than 21 times wider than the metal traces of layers 3 and 4. Therefore, vias connecting corresponding metal layers of different orders can have different sizes (e.g., different widths).

[0076] Figure 6 illustrates a cross-section 600 of an example of metal layers in an integrated circuit connected via via connectors according to some embodiments. The integrated circuit may be a deterministic stream processor, such as a TSP 100 or a TSP 300. Cross-section 600 illustrates one possible embodiment of an integrated circuit having 14 metal layers (e.g., metal layers M0, M1, ..., M13). However, the integrated circuit may have fewer metal layers (e.g., 12 or 11 metal layers) or more metal layers (e.g., 17 or 19 metal layers). Cross-section 600 shows that each pair of adjacent metal layers is interconnected along a spatial dimension (e.g., along the depth or Z-axis of the integrated circuit) using corresponding via connectors (e.g., vias V0, V1, ..., V12). In some embodiments, the corresponding spatial dimension (e.g., spatial dimension along the X and Y axes) of the through-hole connector connecting one of the two lower metal layers (e.g., through-hole connector V3 or V4) may be smaller than the corresponding spatial dimension of the other through-hole connector connecting the two upper metal layers (e.g., through-hole connector V11 or V12), for example, up to about 16 times for each spatial dimension. Example Program Flow

[0077] Figure 7 is a flowchart illustrating a method 700 of wiring for a power grid having an integrated circuit with a deterministic stream processor, according to some embodiments. The integrated circuit may further include at least one computer processor and a non-transitory computer-readable storage medium for storing computer-executable instructions. The deterministic stream processor may be a TSP. The deterministic stream processor may be an embodiment of TSP 100 or an embodiment of TSP 300.

[0078] The operation of method 700 may be initiated by a compiler operating on at least one computer processor and / or a host server separate from the integrated circuitry. Alternatively, the operation of method 700 may be initiated by a compiler operating on at least one computer processor and / or a host server. The compiler may take a model of a deterministic stream processor (e.g., a machine learning model) as its input and output instructions for configuring the overall operation of the deterministic stream processor and the integrated circuitry.

[0079] At 710, a power supply grid for a metal trace of the deterministic streaming processor may be routed across a first spatial dimension and a second spatial dimension orthogonal to the first spatial dimension to supply power to a plurality of functional slices and a plurality of ICUs of the deterministic streaming processor. At 720, at least a portion of the metal trace may be routed as a discontinuous stub along the first or second spatial dimension. The deterministic streaming processor having a wiring power supply grid may further include a plurality of data transfer channels configured to transfer data across functional slices along the first spatial dimension. Additionally, one of the instructions in each subset of the ICU may include a function slice-specific opcode for configuring the transfer of instructions along the second spatial dimension to a corresponding functional slice.

[0080] The power grid deterministic streaming processor may include a plurality of metal layers, and at least one of the metal layers may include a set of metal traces routed as discontinuous stubs along a first or second spatial dimension. Each metal trace from this set may be connected via a via connector along a third spatial dimension orthogonal to either the first or second spatial dimension to a corresponding metal trace in another metal layer of the plurality of metal layers. The resistance of the via connector along the third spatial dimension may be less than the resistance of any metal trace of the power grid along the first or second spatial dimension. A first portion of one of the discontinuous metal traces may include a plurality of source-powered stubs (e.g., source voltage or source current powered stubs), and a second portion of one of the discontinuous metal traces may include a plurality of slot-powered stubs (e.g., slot voltage or slot current powered stubs).

[0081] A first subset of metal traces distributed across a first spatial dimension may be discontinuous along the first spatial dimension, and a second subset of metal traces distributed across a second spatial dimension may be continuous along the second spatial dimension. Each metal trace from the first subset can be connected to a corresponding metal trace from the second subset along a third spatial dimension orthogonal to either the first or second spatial dimensions using a through-hole connector. Each metal trace from the first subset may have a shorter length along the first spatial dimension compared to the length of the corresponding metal trace from the second subset along the second spatial dimension. The first subset of metal traces may belong to a first of a plurality of metal layers, and the second subset of metal traces may belong to a second of a plurality of metal layers. The first metal layer comprising the first subset of discontinuous metal traces may be spatially positioned below the second metal layer comprising the second subset of continuous metal traces. A metal trace from the second subset can be connected to multiple metal traces from the first subset using multiple through-hole connectors along a third spatial dimension orthogonal to either the first or second spatial dimensions. The power grid of the metallic trace can be symmetrical about a first axis and / or a second axis orthogonal to the first axis. Example computer system architecture

[0082] Figure 8A is an abstract diagram of an example computer system suitable for implementing one embodiment of the present invention, according to some embodiments. In some embodiments described herein, a host processor may include the computer system of Figure 8A.

[0083] In Figure 8A, the architecture of computer system 810 typically includes at least one computer 814, which communicates with peripheral devices via bus subsystem 812. Typically, the computer includes a processor (e.g., a microprocessor, graphics processing unit, or digital signal processor) or its electronic processing equivalent, such as an ASIC or FPGA. Peripheral devices typically include a storage subsystem 824 (including a memory subsystem 826 and a file storage subsystem 828), a user interface input device 822, a user interface output device 820, and / or a network interface subsystem 816. The input and output devices enable direct and remote interaction with the computer system 810. The computer system uses at least one output device and / or the network interface subsystem to perform important post-processing activities.

[0084] A computer system can be configured as a server, a client, a workstation, a host, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smartphone, a network device, a rack-mounted "blade," an information station, a television set, a game console, a network router, a switch or bridge, or any data processing machine having instructions specifying actions to be taken by that machine. As used herein, the term "server" refers to a computer or processor that typically executes programs for another computer or processor and sends data and information to another computer or processor.

[0085] A computer system is typically partially built with at least one operating system program, such as Microsoft Windows, Apple MacOS and iOS, Google Android, Linux, and / or Unix. A computer system typically includes a Basic Input / Output System (BIOS) and processor firmware. The processor uses the operating system, BIOS, and firmware to build and control any subsystems and interfaces connected to the processor. Examples of processors implementing such operating systems include: Pentium, Itanium, and Xeon processors from Intel; Opteron and Athlon processors from AMD (Advanced Micro Devices); Graviton processors from Amazon; POWER processors from IBM; SPARC processors from Oracle; and ARM processors from ARM Holdings.

[0086] Any embodiment of the present invention is neither limited to an electronic digital logic computer constructed with a program, nor limited to an electronically programmable device. For example, the claimed embodiments may use an optical computer, a quantum computer, an analog computer, or the like. Furthermore, the use of the singular form of these terms, where only a single computer system or a single machine is referred to, may also refer to any structure of a computer system or machine using programs individually or in combination. Due to the constantly evolving nature of computers and networks, the description of the computer system 810 depicted in FIG8A is intended only as an example. Many other structures of computer system 810 have more components than the computer system depicted in FIG8A.

[0087] The network interface subsystem 816 provides an interface to an external network, including an interface to a communication network 818, and is coupled via the communication network 818 to corresponding interface devices in other computer systems or machines. The communication network 818 may include a number of interconnected computer systems, machines, and physical communication connections (represented by "links"). These communication links may be wired links, optical links, wireless links (e.g., using WiFi or Bluetooth protocols), or any other physical device used for information communication. The communication network 818 may be any suitable computer network, such as a wide area network like the Internet and / or a local area network like Ethernet. The communication network is wired and / or wireless, and many communication networks use encryption and decryption procedures, such as those applicable to a virtual private network (VPN). The communication network uses one or more communication interfaces, which or others receive data from other systems and transmit data to other systems. Embodiments of communication interfaces typically include an Ethernet card, a modem (e.g., telephone, satellite, cable, or Integrated Services Digital Network (ISDN)), an (asynchronous) Digital Subscriber Line (DSL) unit, a FireWire interface, a Universal Serial Bus (USB) interface, and the like. Communication algorithms ("protocols") may be specified using one or more communication languages, such as Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Real-Time Transport Protocol / Real-Time Streaming Protocol (RTP / RTSP), Internet Protocol Packet Exchange (IPX), and / or User Data Packet Protocol (UDP).

[0088] User interface input device 822 may include an alphanumeric keypad, a numeric keypad, pointing devices (such as a mouse), a trackball, a toggle switch, a touch pad, a stylus, a graphics tablet, an optical scanner (such as a barcode reader), touch screen electronics for a display device, audio input devices (such as a voice recognition system or microphone), gaze recognition, brainwave pattern recognition, optical character recognition systems, and other types of input devices. These devices are connected to a computer system via wired or wireless means. Generally, the term "input device" refers to all possible types of devices and programs used to transmit data and information to computer system 810 or to communication network 818. User interface input devices typically enable a user to select objects, icons, text, and the like that appearing on some type of user interface output device (e.g., a display subsystem).

[0089] User interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual communication device (such as an audio and haptic device). The display subsystem may include a CRT, a flat panel device (such as an LCD), an image projection device, or another device for generating visible stimuli (such as a virtual reality system). The display subsystem may also provide non-visual stimuli, such as via audio output, aroma generation, or tactile / haptic output (e.g., vibration and force). Generally, the term "output device" refers to all possible types of devices and programs used to transmit data and information from computer system 810 to a user or to another machine or computer system. These devices are connected to a computer system wired or wirelessly. It should be noted that some devices transmit data and information both to and out of the computer; for example, a haptic device that generates vibration and force on a user's hand while also incorporating sensors to measure the hand's position and movement. The application of ergonomic and semiotic science techniques to improve the efficiency of user interaction with any programs and computers disclosed herein, such as any interaction relating to the design and manufacture of circuits using any of the input or output devices described above.

[0090] The memory subsystem 826 typically includes several memories, including a primary RAM 1030 (or other volatile storage device) for storing instructions and data during program execution and a ROM 1032 for storing fixed instructions therein. The file storage subsystem 828 provides permanent storage for program and data files and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical disc drive, flash memory (such as a USB flash drive), or a removable media cartridge. If the computer system 810 includes an input device that performs optical character recognition, text and symbols printed on a physical object (such as paper) can be used as a means of storing program and data files. Databases and modules used in some embodiments may be stored by the file storage subsystem 828.

[0091] Bus subsystem 812 provides a means for transmitting data and information between various components and subsystems of computer system 810. Although bus subsystem 812 is depicted as a single bus, alternative embodiments of the bus subsystem may use multiple buses. For example, a primary memory such as RAM may be used to communicate directly with a file storage system using a DMA system.

[0092] Figure 8B is another abstract diagram of a computer system suitable for implementing one embodiment of the present invention, according to some embodiments. In some embodiments described herein, a host processor may include the computer system of Figure 8B.

[0093] Figure 8B depicts a memory 840, such as a non-transitory processor-readable data and information storage medium associated with a file storage subsystem 828 and / or a network interface subsystem 816 (e.g., via a bus subsystem 812), and may contain a data structure specifying a circuit design. The memory 840 may be a hard disk, a floppy disk, a CD-ROM, an optical medium, a removable media cartridge, or any other medium storing computer-readable data in a volatile or non-volatile form, such as text and symbols on a physical object (such as paper) that can be processed by an optical character recognition system. A program transmitted to and from a processor from this memory can be converted into a physical signal propagated through a medium (such as a network, connector, wire, or circuit trace as an electrical pulse); or as an acoustic signal through a medium such as space or atmosphere; or as electromagnetic radiation having a wavelength longer than infrared light in the electromagnetic spectrum.

[0094] Those familiar with this technology will recognize that any computer system illustrated in Figures 8A to 8B includes a machine used to execute a program to achieve a desired result by managing work performed by controlled electronic movement. Additional instance computing system

[0095] Figure 9 is a block diagram illustrating one of the components of an instance of a computing machine according to an embodiment, capable of reading instructions from a computer-readable medium and executing them in a processor (or controller). A computer described herein may include a single computing machine shown in Figure 9, a virtual machine, a distributed computing system comprising multiple nodes of the computing machine shown in Figure 9, or any other computing device suitably configured. The computer described herein may use any of the elements described in the preceding figures to perform the described functions.

[0096] By way of example, Figure 9 depicts a graphical representation of a computer system 900, in which instructions 924 (e.g., software, program code, or machine code) (which may be stored on a computer-readable medium) cause the machine to execute any one or more programs discussed herein. In some embodiments, the computer operates as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0097] The architecture of one of the computing machines depicted in Figure 9 can correspond to any of the software, hardware, or combined components shown in the figures above. By way of example, a computing machine can be a tensor stream processor, a personal computer (PC), a tablet PC, a set-top box (STB), a digital assistant (PDA), a cellular phone, a smartphone, a network device, a network router, an Internet of Things (IoT) device, a switch or bridge, or any machine capable of executing instructions 924 specifying actions to be taken by that machine, even though only a single machine is depicted. Furthermore, the term "machine" should also be considered as any collection of machines that individually or jointly execute instructions 924 to perform any one or more of the methods discussed herein.

[0098] Example computer system 900 includes one or more processors (generally, a processor 902) (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASIC), one or more radio frequency integrated circuits (RFIC), or any combination thereof), a main memory 904, and a static memory 906, which are configured to communicate with each other via a bus 908. Computer system 900 may further include a graphics display unit 910 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system 900 may also include an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse, a trackball, a joystick, a motion sensor or other pointing instrument), a storage unit 916, a signal generation device 918 (e.g., a speaker), and a network interface device 920, which are also configured to communicate via bus 908.

[0099] Storage unit 916 includes a computer-readable medium 922 on which instructions 924 embodying any one or more of the methods or functions described herein are stored. Instructions 924 may also reside wholly or at least partially within main memory 904 and / or processor 902 (e.g., a processor cache). Therefore, main memory 904 and processor 902 may also constitute a computer-readable medium during execution of instructions by computer system 900. Instructions 924 may be transmitted or received via a network 926 through network interface device 920.

[0100] Although computer-readable media 922 is shown as a single medium in one example embodiment, the term "computer-readable media" should be considered as including one or more media (e.g., a centralized or distributed database or associated cache and server) capable of storing instructions (e.g., instruction 924). Computer-readable media 922 may include any medium capable of storing instructions (e.g., instruction 924) for execution by a machine and causing the machine to perform any or more of the methods disclosed herein. Computer-readable media 922 includes, but is not limited to, data storage in the form of solid-state memory, optical media, and magnetic media. Computer-readable media 922 does not include transient media such as a signal or a carrier wave. Additional considerations

[0101] The disclosed configuration can offer benefits and advantages, including, for example, by separating the processor's functions into dedicated functional units and configuring the timing of data and instructions to each functional unit, enabling each unit to operate on the received data based on a known timing between the received data and instructions, thus achieving a more efficient data flow. Since the compiler used for the processor is hardware-aware, it can configure an explicit plan for the processor, instructing how and when instruction and data operands are transferred to different blocks of the processor. By considering the timing of received instructions and data, data can be transferred between processor blocks without unnecessary data back-ends, thereby increasing transfer efficiency. Furthermore, by separating the transfer of data and instructions, instructions can be iterated and looped independently of the received data operands.

[0102] Furthermore, because each arithmetic element of the processor is dedicated to a specific function (e.g., MEM, VXM, MXM, SXM), the amount of instructions that need to be processed by the arithmetic elements can be reduced. For example, some arithmetic elements (e.g., in an MXM function slice) can be configured to perform a limited set of operations on any received data. Thus, these arithmetic elements can operate without receiving explicit instructions or only receiving intermittent or limited instructions, potentially simplifying processor operation. For example, when data is transferred across a data channel, data operations read from memory can be intercepted by multiple function slices, allowing multiple operations to be performed on the data in a more efficient manner.

[0103] In operation, a host computer programs a DMA engine to actually transfer data, and again, all of this is coordinated by the runtime layer. Specifically, the IDU transfers a 320-byte vector from a PCIe-Gen4 32-byte array in each core clock cycle (e.g., nominal 900 MHz). Thus, the 320-element vector arrives within one cycle of 10 cycles and is placed on multiple streams moving toward the MEM. The incoming streams flow on S24 to 31 (the upper eight streams), from which the MEM performs a "write" to submit the vector to SRAM. Thus, a PCI receive consists of (i) receiving data from the PCI interface; and (ii) writing the vector to a designated functional slice of the MEM.

[0104] The embodiments of the present invention have been presented for illustrative purposes; they are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the foregoing disclosure.

[0105] Some parts of this description describe embodiments of the invention based on algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are typically used by those skilled in data processing techniques to effectively communicate the essence of their work to others skilled in the art. Although these operations are described functionally, computationally, or logically, they should be understood as being implemented by computer programs or equivalent circuits, microcode, or the like. Furthermore, it has been shown that, without loss of generality, it is sometimes convenient to configure these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0106] One or more hardware or software modules may be used alone or in combination with other devices to perform or implement any of the steps, operations, or procedures described herein. In one embodiment, a software module is implemented using a computer program product comprising a computer-readable medium containing computer code that can be executed by a computer processor to perform any or all of the described steps, operations, or procedures.

[0107] Embodiments of the present invention may also relate to an apparatus for performing the operations described herein. This apparatus may be specifically constructed for a desired purpose, and / or may include a general-purpose computing device that can be selectively activated or reconfigured by a computer program stored in a computer. This computer program may be stored on a non-transitory tangible computer-readable storage medium or any type of media suitable for storing electronic instructions, such media being coupled to a computer system bus. Furthermore, any computing system referred to herein may comprise a single processor or may be an architecture employing multiple processors to increase computing power.

[0108] Some embodiments of the present invention may further relate to a system comprising a processor (e.g., a single-stream processor or an artificial intelligence processor), at least one computer processor (e.g., a host server), and a non-transitory computer-readable storage medium. The storage medium may store computer-executable instructions that, when executed by a compiler operating on at least one computer processor, cause at least one computer processor to be operable for performing the operations and techniques described herein.

[0109] Finally, the language used in this specification has been chosen primarily for readability and guidance purposes, and may not have been selected to define or limit the subject matter of the invention. Therefore, the scope of the invention is not intended to be limited to this detailed description, but rather to explore any claims based on this application. Thus, the disclosure of embodiments is intended to illustrate (and not limit) the scope of the invention, which is set forth in the following claims.

[0110] 100: Tensor Stream Processor (TSP) 110: Vector Execution Module (VXM) 111: Functional Memory Slice (MEM) 112: Functional Memory Slice (MEM) 113:SXM 114:SXM 115: Data, Data Domain, and Presentation Controller (or Numerical Interpretation Module (NIM)) 116: Data, Data Domain, and Presentation Controller (or Numerical Interpretation Module (NIM)) 117: Matrix Execution Module (MXM) 118: Matrix Execution Module (MXM) 120: Command and Control Unit (ICU) 121: External communication circuit 122: Chip Control Unit (CCU) 123: Chip-to-Chip (C2C) Circuit 124: Chip-to-Chip (C2C) Circuit 130: Data 140: Composite Program 300: Tensor Stream Processor (TSP) Die / Tensor Stream Processor (TSP) 400: Power Grid Block 405: Metal layer 410: Metal layer 415: Through-hole connector 500: Power Grid Block 505: Metallic layer 510: Metal layer 515: Power supply short circuit 520: Through-hole connector 600: Cross-section 700: Method 710: Operation 720: Operation 810: Computer System 812: Busbar System 814: Computer 816: Network Interface Subsystem 818: Communication Network 820: User Interface Output Device 822: User Interface Input Device 824: Storage Subsystem 826: Memory Subsystem 828: Archive Storage Subsystem 840: Memory 900: Computer System 902: Processor 904: Primary Memory 906: Static Memory 908: Busbar 910: Graphics Display Unit 912: Alphanumeric Input Device 914: Vernier Control Device 916: Storage Unit 918: Signal generating device 920: Network Interface Device 922: Computer-readable media 924: Instruction 926: Internet

Claims

1. A processor comprising: Multiple functional slices; A plurality of data transmission channels, each configured to transmit data across the functional slices along a first spatial dimension; a plurality of instruction control units (ICUs), each subset of which includes an instruction comprising a functional slice-specific opcode and transmitted along a second spatial dimension orthogonal to the first spatial dimension to a corresponding functional slice; and a power grid of metal traces distributed across the first and second spatial dimensions for supplying power to the functional slices and the ICUs, at least a portion of which is discontinuous along the first or second spatial dimension.

2. The processor of claim 1, wherein the power grid comprises a plurality of metal layers, at least one of which has a set of metal traces that are discontinuous along the first spatial dimension or the second spatial dimension.

3. The processor of claim 2, wherein each metal trace from the group is connected to a corresponding metal trace of another metal layer of the plurality of metal layers via a through-hole connector along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

4. The processor of claim 3, wherein the resistance of one of the via connectors along the third spatial dimension is less than the resistance of any metal trace of the power grid along the first or second spatial dimension.

5. The processor of claim 2, wherein a first portion of one of the discontinuous metal traces includes a plurality of source power supply stubs, and a second portion of one of the discontinuous metal traces includes a plurality of slot power supply stubs.

6. The processor of claim 1, wherein a first subset of the metal traces distributed across the first spatial dimension is discontinuous along the first spatial dimension, and a second subset of the metal traces distributed across the second spatial dimension is continuous along the second spatial dimension.

7. The processor of claim 6, wherein each metal trace from the first subset is connected to a corresponding metal trace from the second subset via a through-hole connector along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

8. The processor of claim 7, wherein each metal trace from the first subset has a shorter length along one of the second spatial dimensions compared to the length of the corresponding metal trace from the second subset along one of the second spatial dimensions.

9. The processor of claim 6, wherein the first subset of the metal traces belongs to one of the plurality of metal layers, and the second subset of the metal traces belongs to one of the plurality of metal layers, respectively.

10. The processor of claim 9, wherein the first metal layer of the first subset of discontinuous metal traces is spatially positioned below the second metal layer of the second subset of continuous metal traces.

11. The processor of claim 6, wherein a metal trace from one of the second subset is connected to a plurality of metal traces from the first subset via a plurality of through-hole connectors along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

12. The processor of claim 1, wherein the power grid of the metal trace is symmetrical about a first axis and a second axis orthogonal to the first axis.

13. The processor of claim 1, wherein the power grid of the metal trace is asymmetrical about at least one of a first axis and a second axis orthogonal to the first axis.

14. A method for wiring electrical grids, comprising: One of the metal traces is routed across a first spatial dimension and a second spatial dimension orthogonal to the first spatial dimension to supply power to a plurality of functional slices of the processor and a plurality of instruction control units (ICUs) of the processor; and at least a portion of the metal traces is routed as discontinuous stubs along the first spatial dimension or the second spatial dimension, wherein a plurality of data transmission channels of the processor are configured to transmit data along the first spatial dimension across the functional slices, and one instruction in each subset of the ICUs includes a functional slice-specific opcode and is configured to transmit along the second spatial dimension to a corresponding functional slice.

15. The power grid cabling method of claim 14, further comprising: The power grid is wired using a plurality of metal layers, at least one of which has a set of metal traces as discontinuous stubs along the first or second spatial dimension; and a through-hole connector is used to connect each metal trace from the set to a corresponding metal trace in another metal layer of the plurality of metal layers along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

16. The power grid cabling method of claim 14, further comprising: A first subset of such metal traces is routed as discontinuous stubs across the first spatial dimension; And to route a second subset of one of these metal traces as a continuous track across the second spatial dimension.

17. The power grid cabling method of claim 16, further comprising: Each metal trace from the first subset is routed to have a shorter length along one of the first spatial dimensions compared to the length of the corresponding metal trace from one of the second subset along one of the second spatial dimensions; and each metal trace from the first subset is connected to the corresponding metal trace from the second subset along a third spatial dimension orthogonal to one of the first and second spatial dimensions using a through-hole connector.

18. The power grid cabling method of claim 16, further comprising: Using multiple through-hole connectors, a metal trace from one of the second subsets is connected to multiple metal traces from the first subset along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

19. A non-transitory computer-readable storage medium having executable instructions stored thereon, which, when executed by a computer processor, cause the computer processor to: route a power grid of one of the metal traces across a first spatial dimension and a second spatial dimension orthogonal to the first spatial dimension for supplying power to a plurality of functional slices of the processor and a plurality of instruction control units (ICUs) of the processor; and route at least a portion of the metal traces as discontinuous stubs along the first spatial dimension or the second spatial dimension, wherein a plurality of data transmission channels of the processor are configured to transmit data along the first spatial dimension across the functional slices, and one instruction in each subset of the ICUs includes a functional slice-specific opcode and is configured to transmit along the second spatial dimension to a corresponding functional slice.

20. The non-transitory computer-readable storage medium of claim 19 further includes, when executed by the computer processor, instructions that cause the computer processor to: wire the power grid using a plurality of metal layers, at least one of the metal layers having a set of metal traces wired as discontinuous stubs along the first spatial dimension or the second spatial dimension; and using a through-hole connector to connect each of the metal traces from the set to a corresponding metal trace in another metal layer of the plurality of metal layers along a third spatial dimension orthogonal to one of the first and second spatial dimensions.

Images