General Purpose Systolic Array

Abstract

A systolic array cell is described, the cell including two general purpose arithmetic logic units (ALUs) and a register file. The cells may be arranged in a matrix or array such that an output of a first ALU of a first cell is provided to a second cell adjacent to the right of the first cell, and an output of the second ALU of the first cell is provided to a third cell below the first cell. The two ALUs in each cell of the array can process a different instruction each cycle.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 703,479, filed March 24, 2022, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] A general parallel computing problem asks how to build a general computer with multiple parallel operating elements that can be used effectively on a single computing problem. Existing solutions include systolic arrays that are fixed function, with the matrix multiplication units (MXUs) of the tensor processing units (TPUs) and the tensor cores of the graphics processing units (GPUs) performing the matrix multiplication. Early systolic arrays had very few nodes, such as 8 nodes. GPUs can only peak at matrix multiplication. Many-core central processing units (CPUs) only reach tens, and sometimes hundreds, of parallel cores. Vector machines have computational units that are one-dimensional. Summary of the Invention

[0003] The present disclosure provides a computational unit capable of operations beyond dense linear algebra. Such operations may include sparse linear algebra, sorting, compaction, dynamic programming, and various permutation and table lookup tasks. The computational unit comprises a systolic array of cells, each cell containing a crossbar switch, one or more general purpose arithmetic logic units (ALUs), and a register file that receives output from the one or more general purpose ALUs and returns it to the crossbar switch. The cells are linked by buses that carry the right output from one cell to the left input of the next cell above, or the bottom output from one cell to the top input of the next cell below.

[0004] Systolic arrays can be programmed to execute loop blocks, each of which is a sequence of instructions that is executed once on every cell in the systolic array. Loop blocks start at the origin in the top left of the array (0,0) and propagate through the array along the opposite corner, with the only ordering constraint in hardware being that the preceding ones above and to the left must have executed before the current cell. Different loop blocks change the behavior of the systolic array. By executing different loop blocks, it is possible to perform dense or sparse matrix multiplication, integer, reordering, and string sorting, compression, as well as dynamic programming algorithms to map into the systolic array. At the edge of the array, data is input as vectors along the top and left sides, propagates through the array, is modified by the loop block program, and then produces output vectors below and to the right. These input and output vectors can be supplied by standard vector or SIMD load / store engines.

[0005] One aspect of the present disclosure provides a systolic cell comprising a crossbar switch, a first arithmetic logic unit (ALU) coupled to a first output of the crossbar switch, a second ALU coupled to a second output of the crossbar switch, and a register file configured to receive outputs from the first ALU and the second ALU as inputs to the register file. An output of the register file may be provided to an input of the crossbar switch. The crossbar switch may be configured to receive outputs from one or more adjacent systolic cells as inputs to the crossbar switch. According to some examples, at least one of the first ALU or the second ALU may include a multiplier.

[0006] The register file may use relatively small registers, such as 64-bit registers or smaller. According to some examples, the crossbar switch is a 4×4 crossbar switch and is configured to receive two inputs from the register file and two inputs from adjacent cells, and to provide two outputs to a first ALU and two outputs to a second ALU.

[0007] In another example, the systolic cell further comprises a third arithmetic logic unit coupled between the crossbar switch and the register file.

[0008] Another aspect of the disclosure provides a systolic array comprising a plurality of cells configured such that a first output of a first cell is provided as an input to a second adjacent cell and a second output of the first cell is provided as an input to a third adjacent cell. Each cell in the array may comprise a crossbar switch, one or more ALUs coupled to an output of the crossbar switch, and a register file configured to receive outputs from the first and second ALUs as inputs to the register file. The one or more ALUs may be general ALUs. According to some examples, an output of the register file of each cell is provided to an input of a crossbar switch in the same cell.

[0009] The one or more ALUs may include a first ALU coupled to a first output of the crossbar switch and a second ALU coupled to a second output of the crossbar switch. In some examples, the crossbar switch of the fourth cell is configured to receive as inputs to the crossbar switch an output from the first ALU of the third cell and an output from the second ALU of the second cell.

[0010] According to some examples, the array is configured to receive two source vectors and generate at least one result vector per cycle.

[0011] Another aspect of the present disclosure provides a systolic array cell, the cell including two general purpose arithmetic logic units (ALUs) and a register file. The cells may be arranged in a matrix or array such that an output of a first ALU of a first cell is provided to a second cell adjacent to the right of the first cell, and an output of the second ALU of the first cell is provided to a third cell below the first cell. The two ALUs in each cell of the array can process a different instruction in each cycle.

[0012] Another aspect of the disclosure provides a computing system comprising one or more memories, one or more processors in communication with the one or more memories, and a plurality of cells in communication with the one or more processors, the plurality of cells configured such that a first output of a first cell is provided as an input to a second adjacent cell and a second output of the first cell is provided as an input to a third adjacent cell. Each cell may comprise a crossbar switch, a first arithmetic logic unit (ALU) coupled to the first output of the crossbar switch, a second ALU coupled to the second output of the crossbar switch, and a register file configured to receive outputs from the first ALU and the second ALU as inputs to the register file.

[0013] The one or more processors may include at least one of a scalar core and a vector processing unit. The one or more memories may include a vector data cache.

[0014] According to some examples, the system further includes a sequencer configured to control instructions sent to the one or more processors and the plurality of cells.

[0015] The array may be configured to receive two source vectors and generate at least one result vector per cycle. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a circuit diagram illustrating an exemplary cell according to aspects of the present disclosure. [Diagram 2] 2 illustrates an exemplary array of the cells of FIG. 1. [Diagram 3] 1 illustrates an exemplary hardware implementation in accordance with an aspect of the present disclosure. [Figure 4] 1 illustrates an example system according to an aspect of the present disclosure. [Diagram 5] 1 illustrates an exemplary computing environment in accordance with an aspect of the present disclosure. [Figure 6] 1A-C show examples of multi-cycle fragments propagating from cell to cell over time, according to embodiments of the present disclosure. [Figure 7A] 13 illustrates example execution of operations with various degrees of staggering, according to aspects of the present disclosure. [Figure 7B] 13 illustrates example execution of operations with various degrees of staggering, according to aspects of the present disclosure. [Figure 7C] 13 illustrates example execution of operations with various degrees of staggering, according to aspects of the present disclosure. [Figure 7D] 13 illustrates example execution of operations with various degrees of staggering, according to aspects of the present disclosure. [Figure 7E] 13 illustrates example execution of operations with various degrees of staggering, according to aspects of the present disclosure. [Figure 8] FIG. 13 is an example timing diagram showing the combined timing of the Vector Unit, Stagger Insertion, LHS Corner Turn, Matrix Unit, RHS Corner Turn, Destagger, Skid Buffer, and the return of values ​​to the Vector Unit. [Figure 9] 1 illustrates various exemplary operations according to aspects of the present disclosure. [Figure 10] 1 illustrates various exemplary operations according to aspects of the present disclosure. [Figure 11] 1 illustrates various exemplary operations according to aspects of the present disclosure. [Figure 12] 1 illustrates various exemplary operations according to aspects of the present disclosure. [Figure 13] 1 illustrates various exemplary operations according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] According to the present disclosure, a systolic array cell includes one or more general purpose arithmetic logic units (ALUs) and a register file. Each of the one or more ALUs may receive input from an output port of a crossbar switch. The output of the ALU may be input to the register file, which then returns the input to the crossbar switch. The output from the ALU may be further input to an adjacent cell. For example, the output from a first ALU may be input to the crossbar switch of a second cell to the right in the array. The output from the second ALU may be input to the crossbar switch of a third cell below in the array. The buses may be generalized such that there are one or more horizontal buses and one or more vertical buses controlled by a programmer. The buses may be unidirectional such that only bottom and right connections are included. This may avoid arbitration.

[0018] Instructions may be sent systolicly so that the array operates in a single instruction / multiple data (SIMD) fashion. For example, instruction flow may be performed using a spanning tree of cells such that the preceding cell to the left or above the current cell has the previous instruction. A single control unit at the top left corner of the array may send an instruction to the top left corner of the array. The instruction then propagates through the array along the opposite diagonal wavefront. In other words, for a given cell in the array, the cell may receive an input from an upstream adjacent neighbor and pass an output to a downstream adjacent neighbor. Here, the terms upstream and downstream are relative to the direction of the instruction or data propagating through the array. Each instruction visits every cell of the rectangular systolic array once. This transmission provides a SIMD type control with one control unit for every cell but with pipelined transmission. According to some examples, the array may be a one-dimensional vector unit using pipelined instruction control.

[0019] The array can be configured to perform different tasks by modifying the programs running in the cells: the array can operate as a dense or sparse matrix multiplier, or as a sorting network, or as a compressor, or as a stage in a dynamic programming calculation.

[0020] The parallel computing approach described herein, by design, has no contention or arbitration between simultaneously operating cells: like a systolic array multiplier or a Very Long Instruction Word (VLIW), the execution time of a program is data independent and entirely determinable by analyzing the source code.

[0021] FIG. 1 illustrates an exemplary cell 100 that may be used in a computational unit 200 as shown in FIG. 2. The cell 100 may be a flexible arithmetic logic systolic two-dimensional unit. The cell 100 may have an architecture similar to a matrix multiplication unit (MXU). However, rather than fixed-function multiply-accumulate hardware, the cell 100 includes one or more arithmetic logic units (ALUs) 120, 130. According to some examples, a floating-point ALU may be used. Additionally, a SIMD intra-register ALU may be used that performs four 8-bit operations inside a 32-bit register file. Although two ALUs are shown and described in this example, in other examples, additional or fewer ALUs may be implemented.

[0022] The ALUs 120, 130 may be, for example, general purpose ALUs or special purpose ALUs. The ALUs 120, 130 may be 16-bit, 32-bit, 64-bit, or any other value. In some embodiments, both ALUs 120, 130 are the same type and size, while in other embodiments, the ALUs 120, 130 may differ from one another. According to some examples, the ALU 120 may include a multiplier, such as a 16-bit, 32-bit, or other multiplier. The size of the multiplier may vary in relation to the size of the ALU. According to other examples, the ALU may be capable of other types of special operators, such as population count and find-first-set / find-first-zero / count-leading-sign bit operations.

[0023] 1 shows two ALUs 120, 130, other embodiments may include additional or fewer ALUs. The ALUs called may all be driven by the same VLIW instruction bundle, such that the cell executes the bundle in a single cycle. An instruction bundle may contain as many operators as there are ALUs in the cell.

[0024] Each of the two ALUs 120, 130 may be coupled to an output of the crossbar switch 110. As shown in the example of FIG. 1, the crossbar switch 110 is a 4×4 crossbar switch. In other examples, other types of crossbar switches may be used. Inputs to the crossbar switch 110 may include outputs from adjacent cells, such as cells to the left or above in the array. Inputs to the crossbar switch 110 may further include one or more outputs from a general purpose register file 150, which is described further below.

[0025] The outputs of the two ALUs 120, 130 may be input to a multiplexer 140 and may further be input to a register file 150. According to some examples, the multiplexer 140 may select one of the ALU outputs to write to the register file. For example, the multiplexer 140 may select an output based on the instruction being passed. For example, given a bundle where a first ALU writes to a corresponding bus and a second ALU writes to the register file, the multiplexer may select the second ALU. In embodiments where the register file includes additional ports, both ALU outputs may be written to the register file every cycle. In such embodiments, the multiplexer 140 may be omitted.

[0026] Register file 150 may be relatively small, such as having eight entries. Register file 150 in cell 100 may be general purpose registers rather than special purpose registers as used in MXU cells. According to some examples, the cell does not include large local storage. According to other examples, addressable memory may be included.

[0027] Each of the two ALUs 120, 130 can be controlled with a true instruction stream. In this regard, each cell 100 in the array can execute instructions in different instruction cycles. Also, the control unit propagates instructions along opposite corner wavefronts of the rectangular array, so that each instruction visits every cell in the array once.

[0028] FIG. 2 shows an exemplary computational unit 200 including an array of cells 100 with an architecture as described above in connection with FIG. 1. In this example, the computational unit 200 is a rectangular two-dimensional systolic array. Wires 202, 205 forming connections between the cells of the array flow from top to bottom and left to right. For example, wire 202 may couple the output of cell 100 to the input of cell 102 to the right of cell 100, and wire 205 may couple the output of cell 100 to the input of cell 103 below cell 100. Each cell in the array receives inputs from above and left, performs a computation, and sends an output below and right. Data propagates through the rectangular array along opposing wavefronts. The orthogonal wires of the systolic array 200 may correspond to metal layers of modern semiconductor processes. Although examples of connections between cells according to specific directions and specific orientations are provided herein, e.g., from top to bottom and left to right, it should be understood that other arrangements are possible, e.g., from bottom to top and right to left.

[0029] The array 200 may have a variety of sizes. For example, the array 200 may be square with the same number of cells in each dimension, e.g., horizontally and vertically. Such a square array may have a size of, for example, 16, 32, 64, or any other number. According to another example, the array 200 may be rectangular with a different number of cells in each dimension. The size of the array 200 may be related to the vector length of the base core. In this regard, vector load and vector store operations in the base core may coincide with push / pop operations in each cell.

[0030] Array 200 can fit into a tiny corner of a chip, and it consumes very little power while still being effective at performing computations.

[0031] Array 200 may be a rectangular systolic array, although software may be implemented to emulate other topologies. Other exemplary topologies include, for example, a multidimensional torus, mesh, hypercube, butterfly, fat tree, or ring. For example, array 200 may perform a variety of functions, including, but not limited to, permutations and multiport table lookups, sorting of integers, strings, and records, compression using the Burrows Wheeler transform, alignment of genetic sequences using dynamic programming with the Smith-Waterman algorithm, and the like.

[0032] The cells 100 may execute different instructions every cycle. A sequencer may control the instructions sent to each cell 100. Each cell 100 includes one or more general purpose ALUs, each capable of performing a standard set of operations. For example, each cell 100 may include one, two, four, six, or more general purpose ALUs. Each cell 100 is input by an instruction stream that tells it what to do every cycle. In an example where the cells include two ALUs, each cell 100 can do twice as much work per cycle as a system with one ALU per cell. Systolic wavefronts propagate on anti-diagonal pipelines through the array 200 as such work is done by the cells 100 every cycle. For example, instructions are sent systolically so that the array operates in a SIMD fashion. Instructions propagate along anti-diagonal wavefronts, for example, from the left side of the array to the right side. Each instruction takes B cycles to progress through the array, where B=2*(dimension of array)-1. Each instruction uses a different ALU set every cycle.

[0033] According to some examples, flip-flops can be used to enforce cycle boundaries. For example, a flip-flop in a given cell can hold a value used for an adjacent cell. Flip-flops can be placed at all cell boundaries. Within a single cycle, all register reads and register write-backs occur in one register cycle. However, more advanced systems can use pipelining and bypass networks to reduce clock cycle times and increase clock rates. Inter-cell registers that are not implemented as part of a register file also need to be on the same clock.

[0034] A collective operation over the array of cells 200 defines the behavior of the cells. As an example, a sort kernel looks like this:

[0035] b = min(t,l); r = max(t,l) Here, the bundle of two operations specifies that in a single cycle, one ALU writes its bottom (b) output with the minimum of its top (t) and left (l) inputs. At the same time, the other ALU writes its right (r) output with the maximum of its top and left inputs. Each cell performs the "compare-and-swap" operation of the sorting network. However, the Nth iteration of the compare-and-swap operation 2 When grouped across a systolic array of -cells, the unit rearranges any input vector into a sorted order, effectively implementing a two-dimensional bubble sorting network.

[0036] FIG. 3 illustrates an exemplary hardware implementation. As shown, the implementation includes a matrix unit 310 with an introduction from the left and top. The matrix unit 310 may receive input from a vector unit 320. For example, as shown, data from the vector unit 320 is input to the top and left side of the matrix unit 310. A scalar core 360 ​​provides input to the matrix unit 310 and the vector unit 320. The vector unit 320 communicates with a vector data cache 330, which communicates with a memory controller 340. The scalar core 360 ​​also receives instructions from an instruction cache 380 coupled to the memory controller 340, and an L1 cache 370. The L1 cache 370 may be backed by one or more other cache levels. The memory controller 340 may communicate with a main memory 350 of the computing system. In some implementations, the vector data cache 330 is an L2 data cache, from which L1 data cache 370 misses are served. In some implementations, coherence is enforced by hardware, including the memory controller among other things.

[0037] Although not shown in FIG. 3, outputs from matrix unit 310 may be fed back to vector unit 320. For example, outputs may be generated from the bottom and right side of matrix unit 310. Such outputs may be looped back as inputs to vector unit 320. In this regard, connectivity across full periods may be obtained. For example, later calculations may be used to process data from previous waves.

[0038] The scalar core 360 ​​and the vector unit 320 may be part of a base core built with an instruction set architecture (ISA). The scalar core 360 ​​executes branches and generates addresses. The vector unit 320 moves data between a memory system and each cell of the matrix unit 310. The memory system may include, for example, one or more of the main memory 350, the memory controller 340, and / or the vector data cache 330. The base core may include a vector coprocessor extension port for connecting to the matrix unit 310. From the perspective of the base vector unit 320, each cell of the matrix 310 may be viewed as a coprocessor or an extended vector ALU, to which two source vectors are sent and one result vector is received in return per cycle.

[0039] Matrix unit 310 may be, for example, an array of cells, such as array 200 of Figure 2 consisting of cells 100 of Figure 1. Based on such an architecture, matrix unit 310 may perform semiring operations such as sort operations, compare, max-reduce, etc.

[0040] Vector unit 320 may be a processor or computational unit that can manipulate an entire vector with one instruction. Vector unit 320 may include, for example, a RISC-V instruction set architecture (ISA) or other type of architecture. Vector unit 320 may contain vectors of data, indexes, block addresses, or other information.

[0041] Vector data cache 330 may be a cache or other type of memory or storage unit that holds vector data accessed by vector unit 320.

[0042] Scalar core 360 ​​may be a processor that performs calculations on one number or one data set at a time. Scalar core 360 ​​may include, for example, a RISC-V ISA extension. Scalar core 360 ​​may execute branches and generate addresses. A sequencer may control the order in which instructions are executed by scalar core 360, vector unit 320, and matrix unit 310. In some implementations, scalar, vector, and matrix instructions all occur in a single instruction stream.

[0043] 4 is a block diagram of a data processing system 401 implementing an exemplary computational unit 400. Computational unit 400 may be any of a variety of different computational units, such as, for example, the array 200 of cells 100 described herein with reference to Figures 1-3. Computational unit 400 may implement any of a variety of combinations of horizontal and vertical circuits as described throughout this specification.

[0044] The data processing system can include a host interface 405, a sequencer circuit 410, one or more processor(s) 415, a memory 420, and a timing circuit 425. The data processing system 401 can be implemented in one or more devices across one or more physical locations, as described herein with reference to FIG. 5. In some examples, the components of the described data processing system 401 can be implemented in one or more chips, which can interface with a host device according to any of a variety of data buses or other physical interconnect interfaces. In some examples, the data processing system 401 can be implemented in one or more devices on a network, for example, one or more servers of a cloud platform.

[0045] The processor(s) 415 and memory 420 can be any of a variety of different types of processors and memories, as described herein with reference to Figure 5. In some examples, the processor(s) 415 receive instructions executable by the computing unit 400 to process data. For example, the instructions can be part of a computer program written to perform operations using the computing unit 400.

[0046] The sequencer circuit 410 can convert received instructions into one or more signals understood by the computation unit 400, which causes the computation unit 400 to perform any of a variety of preconfigured operations. These operations can include, for example, loading data from memory 420 into a systolic array of the computation unit 400, moving the data to one or more of the processing elements of the systolic array, processing the data by one or more of the processing elements, and pushing the data out of the systolic array. The sequencer circuit 410 can also be configured to generate one or more control signals to control when instructions are pushed to the computation unit 400.

[0047] The host interface 405 may be configured to receive data from outside the data processing system 401, such as from a processor or another device, and to transmit data generated by the computation unit 400, such as products of matrix multiplication, to one or more devices or processors.

[0048] The timing circuit 425 may be configured to control the timing of the computing unit, e.g., its clock frequency or clock rate. For example, the operations performed by the computing unit 400 may be performed once per clock cycle, such clock cycles being managed by the timing circuit 425.

[0049] The data processing system 401 may also be connected to a power source 1030. The power source 430 may be a battery or other form of power source available to the host device implementing the data processing system, or may be a power source external to the host device and connected to the host device and the data processing system 401 by some wireless or physical connection, e.g., via a wire. The power source 430 may provide a voltage to the computing unit 400, which may be managed by the processor(s) 415, e.g., adjusted higher or lower.

[0050] 5 is a block diagram of an exemplary environment 500 for implementing a data processing system 401 including a computing unit 400. The system 501 may be implemented in one or more devices having one or more processors at one or more locations, such as a server computing device 505. The user computing device 512 and the server computing device 505 may be communicatively connected to one or more storage devices 530 via a network 560. The storage device(s) 530 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 512, 505 or in a different physical location. For example, the storage device(s) 530 may include any type of non-transitory computer-readable medium that may store information, such as hard drives, solid-state drives, tape drives, optical storage, memory cards, ROM, RAM, DVDs, CD-ROMs, writable, and read-only memory.

[0051] The server computing device 505 may include one or more processors 513 and memory 514. The memory 514 may store information accessible by the processor(s) 513, including instructions 521 that may be executed by the processor(s) 513. The memory 514 may also include data 523 that may be retrieved, manipulated, or stored by the processor(s) 513. The memory 514 may be a type of non-transitory computer-readable medium that may store information accessible by the processor(s) 513, such as volatile and non-volatile memory. The processor(s) 513 may include one or more central processing units (CPUs), graphic processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0052] The instructions 521 may include one or more instructions that, when executed by the processor(s) 513, cause one or more processors to perform actions defined by the instructions. The instructions 521 may be stored in object code format for direct processing by the processor(s) 513, or in other formats including interpretable scripts or collections of separate source code modules that are interpreted on-demand or pre-compiled. The instructions 521 may include instructions for implementing a system 401 consistent with aspects of the present disclosure. The system 401 may be executed using the processor(s) 513 and / or using other processors located remotely from the server computing device 505.

[0053] Data 523 may be retrieved, stored, or modified by processor(s) 513 according to instructions 521. Data 523 may be stored in computer registers, in relational or non-relational databases, as tables with multiple different fields and records, or as JSON, YAML, proto, or XML documents. Data 523 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, data 523 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate related data.

[0054] The user computing device 512 may also be configured similarly to the server computing device 1105, with one or more processors 516, memory 517, instructions 518, and data 519. The user computing device 512 may also include a user output 526 and a user input 524. The user input 524 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0055] The server computing device 505 may be configured to transmit data to the user computing device 512, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 526. The user output 526 may also be used to display an interface between the user computing device 512 and the server computing device 505. The user output 526 may alternatively or additionally include one or more speakers, transducers, or other audio output, a haptic interface, or other haptic feedback that provides non-visual and non-auditory information to a platform user of the user computing device 512.

[0056] 5 depicts the processor 513 and memory 514 as being within the computing device 505, the components described herein, including the processor 513 and memory 514, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, the processor 513 may include a collection of processors that may perform simultaneous and / or sequential operations.

[0057] The server computing device 505 may be configured to receive requests to process data from the user computing device 512. For example, the environment 500 may be part of a computing platform configured to provide various services to users via various user interfaces and / or APIs that expose platform services. One or more services may be a machine learning framework or toolset for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 512 may send or receive data specifying operations to be performed by the computation unit 400.

[0058] The devices 512, 505 may be capable of direct and indirect communication over the network 560. The devices 505, 512 may set up listening sockets capable of accepting initiating connections to send and receive information. The network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using communication protocols proprietary to one or more companies. The network 560 may support a variety of short-range and long-range connections. The short-range and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz commonly associated with the Bluetooth® standard, 2.4 GHz and 11 GHz commonly associated with the Wi-Fi® communication protocol, or using a variety of communication standards, such as the LTE® standard for wireless broadband communication. The network 1160 may additionally or alternatively support wired connections between the devices 1112, 1105, including various types of Ethernet® connections.

[0059] Although a single server computing device 505, user computing device 512, and data processing system 4001 are shown in FIG. 5, it is understood that aspects of the disclosure can be implemented according to various different configurations and quantities of computing devices, including paradigms for sequential or parallel processing, or a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on a single device, and any combination thereof. In some examples, one or more devices implement one or more data processing systems, each data processing system including one or more computing units according to aspects of the disclosure. In some examples, a single device can implement multiple computing units, each of the multiple computing units configured to communicate with at least one other computing unit to perform distributed data processing tasks, e.g., in sequential or parallel processing.

[0060] Aspects of the present disclosure may be implemented as one or more computer programs in a digital circuit, a computer-readable storage medium, or a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example executable by a cloud computing platform, as one or more instructions stored on a tangible storage device.

[0061] The systolic array above can be programmed to execute loop blocks, each of which is a sequence of instructions that is executed once on every cell of the systolic array. Loop blocks start at the origin in the top left of the array (0,0) and propagate through the array along the opposite corner, with the only ordering constraint in hardware being that the preceding ones above and to the left must have executed before the current cell. Different loop blocks change the behavior of the systolic array. By executing different loop blocks, it is possible to perform dense or sparse matrix multiplication, integer, reordering, and string sorting, compression, as well as dynamic programming algorithms to map into the systolic array. At the edge of the array, data is input as vectors along the top and left sides, propagates through the array, is modified by the loop block program, and then produces output vectors below and to the right. These input and output vectors can be supplied by standard vector or SIMD load / store engines.

[0062] Software pipelining may be performed on each of the cell 100 and the array 200. A software pipeline loop typically has three parts: a prologue, a body, and an epilogue. The body is the steady-state software pipeline program, which exhibits the best throughput in terms of loops launched per cycle. The prologue has the function of "spinning up" the software pipeline and getting enough work executed on the Inductive Logic Programming (ILP) machine to reach a steady state in the body. The epilogue does the reverse, "spinning down" the machine when there are no more iterations. The height of the periodic body is called the initiation interval (II). Software pipelining searches for the minimum II (MinII) achievable for the loop. Resources, including registers, may limit the II. This resource limitation is called the resource II (ResII).

[0063] Within the array 200, a cell fragment or loop block is written into each cell 100. Each cell fragment may begin with a vector push into the array, have per-cell instructions that propagate along anti-angle wavefronts through the array, and end with returning a result via a vector pop, approximately 2N cycles after the initial vector push. Each cell fragment is a type of 2D hardware-supported loop that executes in just a handful of array cycles, as described further below in connection with Figures 7A-7E.

[0064] 6A-6C show examples of multi-cycle fragments, specifically two-cycle fragments, and how they propagate from cell to cell over time within the array.

[0065] Figure 6A shows an instance where there are no dependencies between cells. The only dependencies, indicated by arrows, are from one instance of instruction A to the next instance of instruction B. Such dependencies tend to execute through the register file, where A writes to a register and B reads the value of that register. Because these dependencies do not transmit across cell boundaries, they are called temporal dependencies. This case corresponds to a stagger of 0, but requires that the data is already present in each cell.

[0066] FIG. 6B shows an instance including dependencies between cells, but only from an instance of an instruction to an instance of the same instruction in the next cell. An instruction may be, for example, a VLIW bundle of one or more RISC-style operations. Dependencies between cells are conveyed via wires and flip-flops between cells, so that the right instance of each instruction receives its input one cycle later than the left instance. In this case, it is correct that the cell fragments are offset in time by one cycle. This corresponds to a stagger of one. A to B time dependencies do not affect the stagger. Even if operation A requires many cycles of latency to complete, the spatial dependencies from A to A and B to B can proceed with a stagger of one. Either left to right or top to bottom processing is completed in a single cycle.

[0067] Figure 6C shows an instance where a dependency between cells shifts from instruction B to instruction A. In this case, the left instance of instruction B finishes its work before passing its output to the right instance of instruction A, so execution is slower. This case corresponds to a stagger of 2. Instead of 2N-1 cycles to execute a single instruction, in this example it takes 4N-2 cycles to execute this pair of instructions.

[0068] When a cell fragment uses a bus, the bus maintains flow preservation, so that a cell fragment has as many writes to the bus as reads from the bus, and the program depends on in-order delivery of those values ​​on the bus. The write of each value occurs in the same or a later cycle within the cell fragment as the original read of that value. If the read and write occur in the same cycle, the dependency causes a stagger of 1. If the write occurs d cycles after the read, the dependency causes a stagger of d+1. The overall program stagger is the maximum for all such bus write / read pairs. To support control staggering, a short instruction FIFO is included between the cells. As an example, the FIFO may be on the order of 8 instructions or less.

[0069] 7A-7E show an example array cycle diagram of the array 200 of FIG. 2. An array cycle may be algebraically defined as a set of space-time triples (i,j,i+j);i∈[0,N),j∈[0,N], forming a wavefront that sweeps a sequence of anti-angles successively in time. Each array cycle visits N2 cells, and can do N2 work, taking a latency of 2N-1 cycles to complete. Such array cycles are pipelined such that one array cycle may be launched from the top left corner cell of the array per cycle. After 2N cycles, the sequence fully utilizes the array. A sequence of M consecutive array cycles takes M+2N-2 cycles to complete.

[0070] 7A represents the array in a single clock cycle. The array includes a number of cells in a first dimension, shown as four cells in the i dimension, although it should be understood that other numbers of cells may be included to provide different values ​​for the i dimension. The array shown in FIG. 7A further includes a number of cells in the j dimension, shown here as four cells. Similarly, it should be understood that the value of j may vary and may be the same or different than the value of i.

[0071] Figure 7B shows a 3D diagram in which the two spatial dimensions of the systolic array are two-dimensional and time is the third dimension. An array cycle is a diagonal plane that cuts this 3D diagram at a 45 degree angle between the time dimension and both spatial dimensions. In this array cycle diagram, N2=42=16 lighter shaded blocks take 2N-1=7 cycles to execute, and the lighter shaded wavefronts are shifted to each opposite corner of successive spaces at a rate of one opposite corner per cycle.

[0072] 7C-7D show an example stagger with one multi-cycle cell program. FIG. 7C shows two clock cycles, each with a different shading. FIG. 7D shows the execution of a multi-cycle cell program using array 200 (FIG. 2). Although the cell program has a multi-cycle length, the program still has a stagger of one between cells. These array cycles in FIG. 7D look like a pipelined replica of FIG. 7B, where the blue array cycle is the most recent (starting at t=1) while the green array cycle is older / earlier (starting at t=0).

[0073] Figure 7E shows an example where the stagger is greater than 1. Figure 7E shows a two cycle program with a true dependency via a bus from the blue (second) instruction to the green (first) instruction in the next cell.

[0074] While the above examples show a stagger of one or two instructions, additional staggers may be supported, for example, an array may support a stagger of eight or more instructions, amounting to the delay between when an instruction begins execution in a cell and when the instruction begins execution in a subsequent cell.

[0075] FIG. 8 provides an exemplary timing diagram showing the combined timing of the Vector Unit, Stagger Insertion, LHS Corner Turn, Matrix Unit, RHS Corner Turn, Destagger, Skid Buffer, and the return of values ​​to the Vector Unit. Staggering is the difference between the timing of parallel processing, for example SIMD / vector style, and systolic / pipelined timing. Data flows through the Matrix Unit, progressing from top to bottom in time. The left half of the diagram shows the four vector lanes and corresponding matrix columns, with time progressing vertically in conceptual stages. Vector data arrives at the Vector Matrix Unit in parallel. The Stagger FIFO introduces i cycles of latency to data that will be introduced at the i th column or i th row of the Matrix Unit. The left corner turn takes two cycles of wire and flip-flop delay. To ensure that the left and top inputs arrive at the Matrix Unit in sync, the top input is delayed by two cycles. The Matrix Unit has a top to bottom latency of four cycles. Matching the right corner turn with the bottom delay takes an additional two cycles. The destagger FIFO adds Ni-1 cycles of delay to the ith column. The skid buffers have room for 2N+2 cycles worth of work, so all preceding logic can always safely flush to them. The right half of the figure shows the cycle-by-cycle timing. With some examples, this can be scaled, such as by using 4 corner turn cycles, a total of 31 stagger / unstagger cycles, and a latency of 32 cycles. The right half of the figure shows the effect of staggering as the operands move through the matrix unit.

[0076] The computing system described above may be capable of a variety of operations, including matrix-vector multiplication, sparse matrix-vector multiplication, permutations, scatter and gather steps, vector operations such as shift, rotate, strided slice, concat, append, sorting, and other operations. According to some examples, the computing system may perform a different permutation every cycle. Some example operations are described below in conjunction with FIGS. 8-12.

[0077] 9 illustrates an example of a gather operation. A gather operation may include a parallel generalization of a load. For example, an index 604 may be used to gather data from a table 602 and provide an output 606. The value of the index 604 corresponds to the location in the table 602 from which the data was gathered. For example, program code for the operation may include the following:

[0078] def serial_gather(table,indices): out = np.zeros(indices.size) for i,index in enumerate(indices): out[i] = table[index] return out Figure 10 shows another gather operation, which includes a parallel generalization of another load. For example, such an operation may be coded as follows:

[0079] def expand_1hot(indices,height=None): out=np.zeros((height,indices.size)) out[indices,np.arange(indices.size)]=1 return out def gather(table, indices): return table@expand_1hot(indices,\ len(table) 11A-11B show an example scatter operation. Referring to FIG. 11A, an index 804 may be used to scatter data from a table 802 to an output 806. However, in some cases, an address collision may occur. For example, in FIG. 11A, there are two values ​​of "2" in the index that are directed to the same address space of the output, so an address collision occurs at the output corresponding to index value "2". FIG. 11B shows an example of using a matrix subtraction operation to resolve the address collision. For example, an addition operation may be used to add the collision. In another example, a maximum operation may be used to determine that the maximum value among any collided values ​​is returned.

[0080] Figure 11 = 2A-B shows an example of a sort such as argsort on a vector of integers. As shown in Figure 12A, a comparison is made between the top and left vector data 920 by matrix 910. Matrix 910 takes a linear input, does quadratic work, and provides a linear output, pipelined once per cycle. The output 906 can be input to a scatter to sort the data. As shown in Figure 12B, collisions can be broken to assign unique locations to the replicated data items. This tweaks the replicas so that each gets a unique scatter index.

[0081] FIG. 12=3 shows an example of a merge-sort-step operation. A first sorted vector 1012 is pushed to the top of the matrix unit 1010. A second reverse sorted vector 1013 is pushed to the left side of the matrix unit 1010. The result on the bottom is a sorted vector 1016. Building on the right-most output, a reverse sorted vector 1018 may be generated. In some examples, the second cycle may be performed in reverse polarity. Rather than using a second cycle with a modified program to obtain the right-side (RHS) output, the system may optionally include a wire to return the RHS output. For example, such a wire may return either the bottom or right-side array fringe output in one cycle, although both may be available. Both outputs may be obtained through multiple cycles.

[0082] While some exemplary operations are shown above, it should be understood that these are only a few example operations and that many more are possible using the computing system described above in connection with Figures 1-5. For example, the computing system may be capable of mapping a large class of doubly nested loops and accelerating them in hardware. Due to memory limitations, the bodies of these doubly nested loops can only depend on previous loop iterations in each dimension and can only store what fits in a local register file. However, with prior loop iterations in each spatial dimension and time storage in the register file, very high dependency chains can be mapped onto the computing system.

[0083] The computing units described herein are beneficial in that they are completely statically schedulable and predictable. There is no contention, arbitration, or queuing in the architecture, i.e., there is no tail latency. Furthermore, there are no dynamic variations in the timing of the array. The rate at which data propagates through the array can be determined completely statically by analysis by the compiler prior to program execution.

[0084] In this specification, the phrase "configured to" is used in various contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system, which, during operation, causes the system to perform one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive inputs and generate outputs according to the inputs to correspond to the one or more operations. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions, which, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0085] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it should be understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Additionally, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The described components, modules, programs, and engines may be integrated together as a single system or may be part of multiple systems.

[0086] Unless otherwise stated, the above alternatives are not mutually exclusive and may be implemented in various combinations to achieve inherent advantages. These and other variations and combinations of the above features may be utilized without departing from the subject matter defined by the claims, and therefore the above description of the embodiments should be construed as illustrative, and not limiting, of the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses that use expressions such as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example. Rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference symbols in different drawings may identify the same or similar elements.

Claims

1. A crossbar switch; a first arithmetic logic unit (ALU) coupled to a first output of the crossbar switch; a second ALU coupled to a second output of the crossbar switch; a register file configured to receive outputs from the first ALU and the second ALU as inputs to the register file; A systolic cell including:

2. 2. The systolic cell of claim 1, wherein an output of the register file is provided to an input of the crossbar switch.

3. The systolic cell of claim 1 , wherein the crossbar switch is configured to receive outputs from one or more adjacent systolic cells as inputs to the crossbar switch.

4. The systolic cell of claim 1 , wherein at least one of the first ALU or the second ALU includes a multiplier.

5. 2. The systolic cell of claim 1, wherein the register file uses registers of 64 bits or less.

6. 2. The systolic cell of claim 1, wherein the crossbar switch is a 4x4 crossbar switch configured to receive two inputs from the register file and two inputs from adjacent cells and to provide two outputs to the first ALU and two outputs to the second ALU.

7. 2. The systolic cell of claim 1 further comprising a third arithmetic logic unit coupled between said crossbar switch and said register file.

8. a plurality of cells configured such that a first output of a first cell is provided as an input to a second adjacent cell and a second output of the first cell is provided as an input to a third adjacent cell, each of the cells comprising: A crossbar switch; one or more arithmetic logic units (ALUs) coupled to outputs of the crossbar switch; a register file configured to receive outputs from the one or more ALUs as inputs to the register file; A systolic array including:

9. 9. The systolic array of claim 8, wherein an output of the register file of each cell is provided to an input of the crossbar switch within the same cell.

10. 9. The systolic array of claim 8, wherein the one or more ALUs are general ALUs.

11. 9. The systolic array of claim 8, wherein the one or more ALUs include a first ALU coupled to a first output of the crossbar switch and a second ALU coupled to a second output of the crossbar switch.

12. 12. The systolic array of claim 11, wherein the crossbar switch of a fourth cell is configured to receive as inputs to the crossbar switch an output from the first ALU of the third cell and an output from the second ALU of the second cell.

13. 9. The systolic array of claim 8, wherein the array is configured to receive two source vectors and to produce at least one result vector per cycle.

14. one or more memories; one or more processors in communication with the one or more memories; a plurality of cells in communication with the one or more processors, the plurality of cells being configured such that a first output of a first cell is provided as an input to a second adjacent cell and a second output of the first cell is provided as an input to a third adjacent cell; Each of the cells comprises: A crossbar switch; one or more arithmetic logic units (ALUs) coupled to outputs of the crossbar switch; a register file configured to receive outputs from the one or more ALUs as inputs to the register file; A computing system comprising:

15. 15. The computing system of claim 14, wherein the one or more processors include at least one of a scalar core and a vector processing unit.

16. The computing system of claim 14 , wherein the one or more memories include a vector data cache.

17. The computing system of claim 14 , further comprising a sequencer configured to control instructions sent to the one or more processors and the plurality of cells.

18. 15. The computing system of claim 14, wherein the array is configured to receive two source vectors and to produce at least one result vector per cycle.

19. 15. The computing system of claim 14, wherein an output of the register file of each cell is provided to an input of the crossbar switch within the same cell.

20. The computing system of claim 14 , wherein the crossbar switch is configured to receive outputs from one or more adjacent cells as inputs to the crossbar switch.

21. The computing system of claim 14 , wherein the one or more ALUs are general ALUs.

22. 15. The computing system of claim 14, wherein the one or more ALUs include a first ALU coupled to a first output of the crossbar switch and a second ALU coupled to a second output of the crossbar switch.

Images