General-purpose systolic array

Abstract

A systolic array cell is described, the cell including two general purpose arithmetic logic units (ALUs) and a register file. The cells may be arranged in a matrix or array such that an output of a first ALU of a first cell is provided to a second cell adjacent to the right of the first cell, and an output of the second ALU of the first cell is provided to a third cell below the first cell. The two ALUs in each cell of the array can process a different instruction each cycle.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a continuation of U.S. Patent Application No. 17 / 703,479, filed on March 24, 2022, the disclosure of which is incorporated herein by reference.

Background Art

[0002] A common parallel computing problem is how to build a general - purpose computer with multiple parallel operating elements that can be effectively used in single - computing problems. Existing solutions include systolic arrays, which are fixed functions, and the matrix - multiplication units (MXUs) of tensor processing units (TPUs) and the tensor cores of graphics processing units (GPUs) perform matrix multiplication. Initial systolic arrays had a very small number of nodes, such as 8 nodes. GPUs can only peak in matrix multiplication. Multi - core central processing units (CPUs) only reach from dozens to sometimes hundreds of parallel cores. Vector machines have one - dimensional computing units.

Summary of the Invention

[0003] This disclosure provides a computing unit capable of operations beyond dense linear algebra. Such operations may include sparse linear algebra, permutation, compression, dynamic programming, and various permutation and table - lookup tasks. The computing unit includes a systolic array of cells, each cell including a cross - bar switch, one or more general - purpose arithmetic logic units (ALUs), and a register file that receives outputs from one or more general - purpose ALUs and returns them to the cross - bar switch. The cells are linked by buses, and the buses transfer the right output from a cell to the left input of the next cell above, or the bottom output from a cell to the top input of the next cell below.

[0004] A systolic array can be programmed to execute loop blocks, each being a sequence of instructions executed once in every cell of the systolic array. Loop blocks begin at the top-left (0,0) origin of the array and propagate along the opposite corners, with the only hardware ordering constraint being that those preceding it on the top and left must have executed before the current cell. Different loop blocks alter the behavior of the systolic array. Executing different loop blocks allows for mapping of dense or sparse matrix multiplication, integer, recording, and string sorting, compression, and dynamic programming algorithms onto the systolic array. At the edges of the array, data is input as vectors along the top and left sides, propagates through the array, is modified by the loop block program, and then generates output vectors down and to the right. These input and output vectors can be supplied by standard vectors or SIMD load / store engines.

[0005] One aspect of the present disclosure provides a systolic cell comprising a crossbar switch, a first arithmetic logic unit (ALU) coupled to a first output of the crossbar switch, a second ALU coupled to a second output of the crossbar switch, and a register file configured to receive the outputs from the first and second ALUs as inputs to the register file. The output of the register file may be provided to the input of the crossbar switch. The crossbar switch may be configured to receive the outputs from one or more adjacent systolic cells as inputs to the crossbar switch. According to some examples, at least one of the first or second ALU may include a multiplier.

[0006] The register file may use relatively small registers, such as 64-bit registers or smaller. In some examples, the crossbar switch is a 4x4 crossbar switch configured to receive two inputs from the register file and two inputs from adjacent cells, and to provide two outputs to the first ALU and two outputs to the second ALU.

[0007] In other examples, the systolic cell further comprises a third arithmetic logic unit coupled between the crossbar switch and the register file.

[0008] In another aspect of this disclosure, a systolic array is provided comprising a plurality of cells configured such that the first output of a first cell is provided as an input to a second neighboring cell, and the second output of a first cell is provided as an input to a third neighboring cell. Each cell in the array may comprise a crossbar switch, one or more ALUs coupled to the output of the crossbar switch, and a register file configured to receive the outputs from the first and second ALUs as input to the register file. The one or more ALUs may be general-purpose ALUs. According to some examples, the output of the register file of each cell is provided to the input of a crossbar switch in the same cell.

[0009] One or more ALUs may include a first ALU coupled to the first output of the crossbar switch and a second ALU coupled to the second output of the crossbar switch. In some examples, the crossbar switch of the fourth cell is configured to receive the output from the first ALU of the third cell and the output from the second ALU of the second cell as inputs to the crossbar switch.

[0010] According to some examples, an array is configured to take two source vectors and produce at least one result vector per cycle.

[0011] Another aspect of the present disclosure provides a systolic array cell, each cell comprising two general-purpose arithmetic logic units (ALUs) and a register file. Multiple cells may be configured in a matrix or array such that the output of the first ALU of the first cell is provided to the second cell to the right of the first cell, and the output of the second ALU of the first cell is provided to the third cell below the first cell. The two ALUs in each cell of the array can process different instructions in each cycle.

[0012] Another aspect of the present disclosure provides a computing system comprising one or more memories, one or more processors communicating with one or more memories, and a plurality of cells communicating with one or more processors, wherein the plurality of cells are configured such that a first output of a first cell is provided as an input to a second neighboring cell, and a second output of the first cell is provided as an input to a third neighboring cell. Each cell may comprise a crossbar switch, a first arithmetic logic unit (ALU) coupled to a first output of the crossbar switch, a second ALU coupled to a second output of the crossbar switch, and a register file configured to receive the outputs from the first and second ALUs as inputs to the register file.

[0013] One or more processors may include at least one of a scalar core and a vector processing unit. One or more memory may include a vector data cache.

[0014] According to some examples, the system further includes one or more processors and a sequencer configured to control instructions sent to multiple cells.

[0015] The array may be configured to accept two source vectors and generate at least one result vector per cycle. [Brief explanation of the drawing]

[0016] [Figure 1] This is a circuit diagram showing an exemplary cell according to an aspect of the present disclosure. [Figure 2] An exemplary array of cells in FIG. 1 is shown. [Figure 3] Exemplary hardware embodiments according to aspects of the present disclosure are shown. [Figure 4] A system example according to aspects of the present disclosure is shown. [Figure 5] An exemplary computing environment according to aspects of the present disclosure is shown. [Figure 6] A - C show examples of multi - cycle fragments that propagate from cell to cell over time according to aspects of the present disclosure. [Figure 7A] Examples of executing operations with various degrees of stagger according to aspects of the present disclosure are shown. [Figure 7B] Examples of executing operations with various degrees of stagger according to aspects of the present disclosure are shown. [Figure 7C] Examples of executing operations with various degrees of stagger according to aspects of the present disclosure are shown. [Figure 7D] Examples of executing operations with various degrees of stagger according to aspects of the present disclosure are shown. [Figure 7E] Examples of executing operations with various degrees of stagger according to aspects of the present disclosure are shown. [Figure 8] An exemplary timing diagram showing the combined timing of a vector unit, stagger insertion, LHS corner turn, matrix unit, RHS corner turn, destagger, skid buffer, and return of values to the vector unit. [Figure 9] Various exemplary operations according to aspects of the present disclosure are shown. [Figure 10] Various exemplary operations according to aspects of the present disclosure are shown. [Figure 11] Various exemplary operations according to aspects of the present disclosure are shown. [Figure 12] Various exemplary operations according to aspects of the present disclosure are shown. [Figure 13] Various exemplary operations according to aspects of the present disclosure are shown.

Mode for Carrying Out the Invention

[0017] According to the present disclosure, a systolic array cell includes one or more general-purpose arithmetic logic units (ALUs) and a register file. Each of the one or more ALUs may receive an input from an output port of a crossbar switch. The output of the ALU can be input to the register file, and then the register file can return the input to the crossbar switch. The output from the ALU can be further input to an adjacent cell. For example, the output from the first ALU can be input to the crossbar switch of the second cell adjacent to the right within the array. The output from the second ALU can be input to the crossbar switch of the third cell below within the array. The bus may be generalized such that there are one or more horizontal buses and one or more vertical buses controlled by a programmer. The bus may be unidirectional such that only downward and right connections are included. Thereby, arbitration can be avoided.

[0018] Since instructions can be sent systolicly, the array operates in a single instruction / multiple data (SIMD) fashion. For example, the instruction flow can be executed using a spanning tree of cells such that the preceding cell to the left or above the current cell has the previous instruction. A single control unit at the upper left corner of the array can send an instruction to the upper left corner of the array. Next, the instruction propagates through the array along a wavefront of the diagonal. In other words, for a given cell within the array, the cell can receive an input from an upstream adjacent neighbor cell and pass an output to a downstream adjacent neighbor cell. Here, the terms upstream and downstream are relative to the direction of the instruction or data propagated through the array. Each instruction visits all cells of the rectangular systolic array once. This transmission provides SIMD-type control with a pipelined transmission having one control unit for all cells. According to some examples, the array can be a one-dimensional vector unit using pipelined instruction control.

[0019] tArrays can be configured to perform different tasks by changing the programs executed in their cells. Arrays can operate as multipliers for dense or sparse matrices, as sorting networks, as compressors, or as stages for dynamic programming computations.

[0020] The parallel computing approaches described herein, by design, do not involve contention or arbitration between concurrently operating cells. Similar to systolic array multipliers or very long instruction words (VLIWs), the program execution time is data-independent and entirely determinable by analyzing the source code.

[0021] Figure 1 shows an exemplary cell 100 that may be used in a computing unit 200 as shown in Figure 2. Cell 100 may be a flexible arithmetic logic systolic two-dimensional unit. Cell 100 may have an architecture similar to that of a matrix multiplication unit (MXU). However, instead of fixed-function multiply-accumulate hardware, cell 100 includes one or more arithmetic logic units (ALUs) 120, 130. According to some examples, floating-point ALUs may be used. Furthermore, SIMD-in-register ALUs that perform four 8-bit operations within a 32-bit register file may be used. Two ALUs are illustrated and described in this example, but additional or fewer ALUs may be implemented in other examples.

[0022] ALU120, 130 may be, for example, a general-purpose ALU or a dedicated ALU. ALU120, 130 may be 16-bit, 32-bit, 64-bit, or any other value. In some embodiments, both ALU120 and 130 are of the same type and size, but in other embodiments, ALU120 and 130 may be different from each other. According to some examples, ALU120 may have a multiplier such as a 16-bit, 32-bit, or other multiplier. The size of the multiplier may vary in relation to the size of the ALU. According to other examples, the ALU may be capable of other types of special operators, such as populationcount and bitwise operations such as find-first-set / find-first-zero / count-leading-sign.

[0023] Figure 1 shows two ALUs, 120 and 130, but other embodiments may include additional or fewer ALUs. All called ALUs can be driven by the same VLIW instruction bundle so that the cell executes the bundle in a single cycle. The instruction bundle may contain as many operators as there are ALUs in the cell.

[0024] Each of the two ALUs 120 and 130 may be coupled to the output of the crossbar switch 110. As shown in the example in Figure 1, the crossbar switch 110 is a 4x4 crossbar switch. In other examples, other types of crossbar switches may be used. The input to the crossbar switch 110 may include outputs from adjacent cells, such as the left cell or the cell above in the array. The input to the crossbar switch 110 may further include one or more outputs from the general-purpose register file 150, which will be described further below.

[0025] The outputs of the two ALUs 120 and 130 may be input to the multiplexer 140, which may further input to the register file 150. According to some examples, the multiplexer 140 can select one of the ALU outputs to write to the register file. For example, the multiplexer 140 can select an output based on the instruction being passed. For example, considering a bundle in which the first ALU writes to the corresponding bus and the second ALU writes to the register file, the multiplexer may select the second ALU. In embodiments where the register file includes additional ports, both ALU outputs may be written to the register file cycle by cycle. In such embodiments, the multiplexer 140 may be omitted.

[0026] The register file 150 may be relatively small, such as having 8 entries. The register file 150 within cell 100 may be general-purpose registers, rather than dedicated registers like those used in MXU cells. In some examples, the cell does not contain large local storage. In other examples, it may contain addressable memory.

[0027] Each of the two ALUs, 120 and 130, can be controlled by a true instruction stream. In this regard, each cell 100 in the array can execute instructions in a different instruction cycle. Also, since the control unit propagates instructions along the wavefront at opposite angles of the rectangular array, each instruction visits every cell in the array once.

[0028] Figure 2 shows an exemplary computing unit 200 including an array of cells 100 having the architecture described above in relation to Figure 1. In this example, the computing unit 200 is a rectangular two-dimensional systolic array. Wires 202, 205 that form the connections between cells in the array flow from top to bottom and from left to right. For example, wire 202 may couple the output of cell 100 to the input of cell 102 to the right of cell 100, and wire 205 may couple the output of cell 100 to the input of cell 103 below cell 100. Each cell in the array receives inputs from above and from the left, performs calculations, and sends outputs down and to the right. The data propagates through the rectangular array along wavefronts of opposite angles. The orthogonal wires of the systolic array 200 may correspond to the metal layers of modern semiconductor processes. While embodiments of cell-to-cell connections depending on specific directions and orientations, such as top to bottom and left to right, are provided herein, it should be understood that other arrangements are possible, such as bottom to top and right to left.

[0029] Array 200 can have various sizes. For example, array 200 may be a square with the same number of cells in each dimension, e.g., horizontally and vertically. Such a square array may have sizes of, for example, 16, 32, 64, or any other number. According to another example, array 200 may be a rectangle with a different number of cells in each dimension. The size of array 200 may be related to the vector length of the base core. In this regard, vector load and vector store operations in the base core may coincide with push / pop operations in each cell.

[0030] The Array 200 can fit into a tiny corner of a chip. Furthermore, it consumes almost no power while still being efficient at performing calculations.

[0031] Array 200 may be a rectangular systolic array, but the software may be implemented to emulate other topologies. Other exemplary topologies include, for example, multidimensional torus, meshes, hypercubes, butterflies, fat trees, or rings. For example, Array 200 can perform a variety of functions, including but not limited to permutations and multiport table lookups, sorting of integers, strings, and records, compression using Burroughs-Wheeler transforms, and sorting of gene sequences using dynamic programming with the Smith-Waterman algorithm.

[0032] Cell 100 may execute different instructions in each cycle. The sequencer may control the instructions sent to each cell 100. Each cell 100 contains one or more general-purpose ALUs, each capable of performing a standard set of operations. For example, each cell 100 may contain one, two, four, six, or more general-purpose ALUs. Each cell 100 is input by an instruction stream that tells it what to do in each cycle. In the example where a cell contains two ALUs, each cell 100 can perform twice the work per cycle of a system with one ALU per cell. The systolic wavefront propagates through the array 200 along the opposite-angle pipeline as such work is performed by cell 100 in each cycle. For example, since instructions are sent systolic, the array operates in a SIMD manner. Instructions propagate along the opposite-angle wavefront, for example, from left to right across the array. Each instruction requires B cycles to travel through the array (where B = 2 * (dimensions of the array) - 1). Each instruction uses a different ALU set for each cycle.

[0033] According to some examples, flip-flops can be used to reinforce cycle boundaries. For example, a flip-flop in a given cell may hold a value used in an adjacent cell. Flip-flops can be placed at the boundaries of all cells. Within a single cycle, all register reads and write-backs occur in one register cycle. However, in more advanced systems, pipelining and bypass networks can be used to reduce clock cycle time and increase the clock rate. Inter-cell registers that are not implemented as part of the register file also need to be on the same clock.

[0034] The set operations on the entire array of 200 cells define the behavior of the cells. As an example, the sort kernel is as follows:

[0035] b = min(t, l); r = max(t, l) In the formula, the bundle of two operations specifies that in a single cycle, one ALU writes the lower (b) output with the minimum values ​​of the upper (t) and left (l) inputs. Simultaneously, the other ALU writes the right (r) output with the maximum values ​​of the upper and left inputs. Each individual cell performs the "compare-and-swap" operation of the sorting network. However, the N of the compare-and-swap operation 2 -When grouped across the entire systolic array of cells, the unit sorts any input vectors into a sorted order, effectively implementing a two-dimensional bubble sorting network.

[0036] Figure 3 shows an exemplary hardware embodiment. As shown in the figure, the embodiment includes a matrix unit 310 with inputs from the left and top. The matrix unit 310 may receive input from a vector unit 320. For example, as shown, data from the vector unit 320 is input to the top and left sides of the matrix unit 310. A scalar core 360 ​​provides input to the matrix unit 310 and the vector unit 320. The vector unit 320 communicates with a vector data cache 330, which communicates with a memory controller 340. The scalar core 360 ​​also receives instructions from an instruction cache 380 and an L1 cache 370 coupled to the memory controller 340. The L1 cache 370 may be supported by one or more other cache levels. The memory controller 340 may communicate with the main memory 350 of the computing system. In some embodiments, the vector data cache 330 is an L2 data cache, and misses in the L1 data cache 370 come from it. In some embodiments, coherence is performed by hardware, including the memory controller, among all caches.

[0037] Although not shown in Figure 3, the output from the matrix unit 310 can be fed back to the vector unit 320. For example, the output can be generated from the bottom and right sides of the matrix unit 310. Such an output can be looped back as input to the vector unit 320. In this regard, connectivity throughout the entire period can be obtained. For example, data from previous waves can be processed using subsequent calculations.

[0038] The scalar core 360 ​​and the vector unit 320 may be part of a base core built by an instruction set architecture (ISA). The scalar core 360 ​​executes branches and generates addresses. The vector unit 320 moves data between the memory system and each cell of the matrix unit 310. The memory system may include, for example, one or more of the main memory 350, the memory controller 340, and / or the vector data cache 330. The base core may include a vector coprocessor extension port for connecting to the matrix unit 310. From the perspective of the base vector unit 320, each cell of the matrix 310 can be represented as a coprocessor or extended vector ALU, to which two source vectors are sent and one result vector is received in return per cycle.

[0039] The matrix unit 310 may be an array of cells, such as the array 200 in Figure 2, which consists of the cell 100 in Figure 1. Based on such an architecture, the matrix unit 310 may perform semiring operations such as sorting, compare, and max-reduce.

[0040] The vector unit 320 may be a processor or computing unit capable of manipulating an entire vector with a single instruction. The vector unit 320 may include, for example, a RISC-V instruction set architecture (ISA) or other types of architectures. The vector unit 320 may contain vectors of data, indices, block addresses, or other information.

[0041] The vector data cache 330 may be a cache or other type of memory or storage unit that holds vector data accessed by the vector unit 320.

[0042] The scalar core 360 ​​may be a processor that performs calculations on one number or one dataset at a time. The scalar core 360 ​​may include, for example, a RISC-V ISA extension. The scalar core 360 ​​may perform branches and generate addresses. The sequencer may control the order in which instructions are executed by the scalar core 360, the vector unit 320, and the matrix unit 310. In some embodiments, scalar instructions, vector instructions, and matrix instructions all occur in a single instruction stream.

[0043] Figure 4 is a block diagram of a data processing system 401 implementing an exemplary computing unit 400. The computing unit 400 may be any of several different computing units, such as the array 200 of cells 100 described herein with reference to Figures 1 to 3. The computing unit 400 may implement any of several combinations of horizontal and vertical circuits, as described throughout this specification.

[0044] The data processing system may include a host interface 405, a sequencer circuit 410, one or more processors 415, memory 420, and timing circuit 425. The data processing system 401 may be implemented in one or more devices across one or more physical locations, as described herein with reference to Figure 5. In some examples, the components of the described data processing system 401 may be implemented on one or more chips, which may interface with a host device according to a variety of data buses or other physical interconnection interfaces. In some examples, the data processing system 401 may be implemented on one or more devices on a network, for example, one or more servers on a cloud platform.

[0045] The processor(s) 415 and memory 420 can be any of various different types of processors and memory, as described herein with reference to Figure 5. In some examples, the processor(s) 415 receive instructions that can be executed by the compute unit 400 to process data. For example, the instructions may be part of a computer program written to perform calculations using the compute unit 400.

[0046] The sequencer circuit 410 can convert received instructions into one or more signals that the computing unit 400 can understand, thereby enabling the computing unit 400 to perform one of a variety of pre-configured operations. These operations may include, for example, loading data from memory 420 into the systolic array of the computing unit 400, moving data to one or more processing elements of the systolic array, processing the data by one or more processing elements, and pushing data out of the systolic array. The sequencer circuit 410 may also be configured to generate one or more control signals to control when instructions are pushed to the computing unit 400.

[0047] The host interface 405 may be configured to receive data from outside the data processing system 401, for example from a processor or another device, and to send data generated by the computing unit 400, such as the product of matrix multiplication, to one or more devices or processors.

[0048] The timing circuit 425 may be configured to control the timing of the computing unit, for example, its clock frequency or clock rate. For example, an operation performed by the computing unit 400 may be performed once per clock cycle, and such clock cycles are managed by the timing circuit 425.

[0049] The data processing system 401 may also be connected to the power supply 1030. The power supply 430 may be a battery or other form of power source available in the host device implementing the data processing system, or it may be a power source located outside the host device and connected to the host device and the data processing system 401 by some wireless or physical connection, such as a wire connection. The power supply 430 can supply voltage to the computing unit 400, which is managed by the processor 415 and can be adjusted, for example, higher or lower.

[0050] Figure 5 is a block diagram of an exemplary environment 500 for implementing a data processing system 401 including a computing unit 400. The system 501 may be implemented in one or more devices having one or more processors in one or more locations, such as a server computing device 505. The user computing device 512 and the server computing device 505 may be communicably connected to one or more storage devices 530 via a network 560. The storage device(s) 530 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 512, 505 or in a different physical location. For example, the storage device(s) 530 may include any type of non-temporary computer-readable medium capable of storing information, such as hard drives, solid-state drives, tape drives, optical storage, memory cards, ROM, RAM, DVDs, CD-ROMs, writable and read-only memory.

[0051] The server computing device 505 may include one or more processors 513 and memory 514. The memory 514 may store information accessible by the processor(s) 513, including instructions 521 that can be executed by the processor(s) 513. The memory 514 may also contain data 523 that can be retrieved, manipulated, or stored by the processor(s) 513. The memory 514 may be a type of non-temporary computer-readable medium that can store information accessible by the processor(s) 513, such as volatile and non-volatile memory. The processor(s) 513 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0052] Instruction 521 may include one or more instructions that, when executed by processor(s) 513, cause one or more processors to perform an action defined by the instruction. Instruction 521 may be stored in object code form for direct processing by processor(s) 513, or in other form including an interpretable script or collection of independent source code modules that are interpreted on demand or pre-compiled. Instruction 521 may include instructions for implementing a system 401 consistent with aspects of this disclosure. System 401 may be executed using processor(s) 513 and / or other processors located remotely from server computing device 505.

[0053] Data 523 may be retrieved, stored, or modified by processor(s) 513 in accordance with instruction 521. Data 523 may be stored in computer registers, in relational or non-relational databases as tables with multiple distinct fields and records, or as JSON, YAML, proto, or XML documents. Data 523 may also be formatted in a computer-readable format, such as but not limited to binary values, ASCII, or Unicode. Furthermore, Data 523 may contain enough information to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to other data stored in memory including other network locations, or information used by functions to compute the relevant data.

[0054] The user computing device 512 may also be configured similarly to the server computing device 1105, using one or more processors 516, memory 517, instructions 518, and data 519. The user computing device 512 may also include user outputs 526 and user inputs 524. The user inputs 524 may include any suitable mechanism or technology for receiving input from the user, such as a keyboard, mouse, mechanical actuator, soft actuator, touchscreen, microphone, and sensor.

[0055] The server computing device 505 may be configured to transmit data to the user computing device 512, and the user computing device 512 may be configured to display at least a portion of the received data on a display implemented as part of the user output 526. The user output 526 may also be used to display the interface between the user computing device 512 and the server computing device 505. The user output 526 may optionally or additionally include one or more speakers, transducers, or other audio outputs, haptic interfaces, or other haptic feedback that provides non-visual and non-auditory information to the platform user of the user computing device 512.

[0056] Although Figure 5 shows the processor 513 and memory 514 as being located inside the computing device 505, the components described herein, including the processor 513 and memory 514, may include multiple processors and memories that can operate in different physical locations rather than within the same computing device. For example, the processor 513 may include a set of processors that can perform simultaneous and / or sequential operations.

[0057] The server computing device 505 may be configured to receive requests from the user computing device 512 to process data. For example, the environment 500 may be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services may be machine learning frameworks or toolsets for generating neural networks or other machine learning models according to specified tasks and training data. The user computing device 512 may send and receive data specifying the actions to be performed by the computing unit 400.

[0058] Devices 512 and 505 may be capable of direct and indirect communication over network 560. Devices 505 and 512 may set up listening sockets that can accept initiating connections for sending and receiving information. Network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using proprietary communication protocols for one or more companies. Network 560 may support a variety of short-range and long-range connections. Short-range and long-range connections may be made over different bandwidths, such as 2.402GHz to 2.480GHz, commonly associated with the Bluetooth® standard, 2.4GHz and 11GHz, commonly associated with the Wi-Fi® communication protocol, or using various communication standards, such as the LTE® standard for wireless broadband communication. Network 1160 may further, or alternatively, support wired connections between devices 1112 and 1105, including various types of Ethernet® connections.

[0059] A single server computing device 505, a user computing device 512, and a data processing system 4001 are shown in Figure 5, but it is understood that aspects of the present disclosure can be implemented according to various different configurations and quantities of computing devices, including paradigms for sequential or parallel processing, or distributed networks of multiple devices. In some embodiments, aspects of the present disclosure can be implemented in a single device and any combination thereof. In some examples, one or more devices implement one or more data processing systems, and each data processing system includes one or more compute units according to aspects of the present disclosure. In some examples, a single device can implement multiple compute units, each of which is configured to communicate with at least one other compute unit to perform a distributed data processing task, for example, in sequential or parallel processing.

[0060] Aspects of this disclosure can be implemented in digital circuits, computer-readable storage media, as one or more computer programs, or as one or more combinations thereof. The computer-readable storage media can be non-temporary, for example, as one or more instructions stored in a tangible storage device, and can be run by a cloud computing platform.

[0061] The systolic array described above can be programmed to execute loop blocks, each of which is a sequence of instructions executed once in every cell of the systolic array. Loop blocks begin at the top-left (0,0) origin of the array and propagate along the opposite corners, but the only hardware ordering constraint is that those preceding above and to the left must have executed before the current cell. Different loop blocks alter the behavior of the systolic array. Executing different loop blocks makes it possible to perform dense or sparse matrix multiplication, integer, recording, and string sorting, compression, and dynamic programming algorithms and map them to the systolic array. At the edges of the array, data is input as vectors along the top and left sides, propagates through the array, is modified by the loop block program, and then generates output vectors down and to the right. These input and output vectors can be supplied by standard vectors or a SIMD load / store engine.

[0062] Software pipelined programming can be performed in both cell 100 and array 200. A software pipelined loop typically has three parts: a prologue, a body, and an epilogue. The body is a steady-state software pipelined program that exhibits the best throughput for the loops invoked per cycle. The prologue has the function of "spinning up" the software pipeline and obtaining enough work to be executed on the Inductive Logic Programming (ILP) machine to reach a steady state in the body. The epilogue is similar in reverse, "spinning down" the machine when there are no further iterations. The periodic height of the body is called the start interval (II). Software pipelined programming searches for the minimum II (MinII) achievable in the loop. Resources, including registers, can limit II. This resource limit is called resource II (ResII).

[0063] Within array 200, cell fragments or loop blocks are written to each cell 100. Each cell fragment has a per-cell instruction that begins with a vector push to the array, propagates through the array along the opposite-angle wavefront, and can terminate approximately 2N cycles after the initial vector push by returning the result via a vector pop. Each cell fragment is a type of 2D hardware support loop that runs for just a handful of array cycles, as further described below in relation to Figures 7A-7E.

[0064] Figures 6A to 6C show examples of multicycle fragments, specifically two-cycle fragments, and how they propagate from cell to cell within the array over time.

[0065] Figure 6A shows an instance with no dependencies between cells. The only dependencies, indicated by arrows, are from one instance of instruction A to the next instance of instruction B. Such dependencies tend to be executed via register files, with A writing to a register and B reading the value from that register. These dependencies are called temporal dependencies because they are not transmitted across cell boundaries. This case corresponds to a stagger of zeros, but the data must already exist in each cell.

[0066] Figure 6B shows an instance that includes inter-cell dependencies, but only includes the instance of an instruction from one instance to the next instance of the same instruction in the next cell. An instruction may be, for example, a VLIW bundle consisting of one or more RISC-style operations. Since inter-cell dependencies are communicated via inter-cell wires and flip-flops, the right instance of each instruction receives its input one cycle later than the left instance. In this case, it is correct that the cell fragment is offset by one cycle in time. This corresponds to a stagger of 1. The temporal dependency from A to B does not affect the stagger. Even if operation A requires many cycles of latency to complete, the spatial dependencies from A to A and from B to B can proceed with a stagger of 1. Whether the operation is from left to right or from top to bottom, it can be completed in a single cycle.

[0067] Figure 6C shows an instance where the dependency between cells shifts from instruction B to instruction A. In this case, the left instance of instruction B finishes its work before passing its output to the right instance of instruction A, resulting in slower execution. This case corresponds to a stagger of 2. Instead of 2N-1 cycles to execute a single instruction, this example takes 4N-2 cycles to execute this pair of instructions.

[0068] When a cell fragment uses the bus, the bus maintains flow storage, and as a result, the cell fragment has an equal number of writes to and reads from the bus, and the program relies on in-order delivery of those values ​​on the bus. Each write of a value occurs in the same cycle as the original read of that value, or in a subsequent cycle within the cell fragment. If the read and write occur in the same cycle, the dependency causes a stagger of 1. If the write occurs d cycles after the read, the dependency causes a stagger of d+1. The stagger of the entire program is the maximum value for all such bus write / read pairs. To support control stagger, short instruction FIFOs are included between cells. For example, a FIFO may be about 8 instructions or less.

[0069] Figures 7A–7E show exemplary array cycle diagrams for Array 200 in Figure 2. An array cycle may be algebraically defined as a set of space-time triples(i,j,i+j); i∈[0,N)),j∈[0,N], forming a wavefront that sweeps a sequence of opposite angles in time. Each array cycle visits N2 cells, so it can perform N2 work and requires a latency of 2N-1 cycles to complete. Such array cycles are pipelined so that one array cycle can be launched from the cell in the upper left corner of the array with each cycle. After 2N cycles, the sequence fully utilizes the array. A sequence of M consecutive array cycles requires M+2N-2 cycles to complete.

[0070] Figure 7A represents an array of a single clock cycle. The array contains multiple cells in the first dimension, shown as four cells in dimension i, but it should be understood that it may contain other numbers of cells to provide different values ​​for dimension i. The array shown in Figure 7A further contains multiple cells in dimension j, shown here as four cells. Similarly, it should be understood that the value of j may vary and may be the same as or different from the value of i.

[0071] Figure 7B shows a 3D diagram of a systolic array where the two spatial dimensions are 2D and time is 3D. The array cycle is the diagonal plane that cuts this 3D diagram at a 45-degree angle between the time dimension and both spatial dimensions. In this array cycle diagram, N2=42=16 thinner shaded blocks take 2N-1=7 cycles to execute, and the wavefronts of the thinner shaded blocks shift to each opposite angle in the continuous space at a velocity of one opposite angle per cycle.

[0072] Figures 7C and 7D show an exemplary stagger containing a single multicycle cell program. Figure 7C represents two clock cycles, each with a different shading. Figure 7D shows the execution of a multicycle cell program using Array 200 (Figure 2). Although the cell program has a multicycle length, the program still has a stagger of one between cells. These array cycles in Figure 7D look like a pipelining replica of Figure 7B, where the blue array cycle is the most recent (starting at t=1), while the green array cycle is older / earlier (starting at t=0).

[0073] Figure 7E shows an example where the stagger is greater than 1. Figure 7E shows two cycle programs with a true dependency via a bus from the blue (second) instruction to the green (first) instruction in the next cell.

[0074] The above examples show a stagger of one or two instructions, but additional staggers may be supported. For example, an array may support a stagger of eight or more instructions, which would account for the delay between when an instruction begins execution in a cell and when an instruction begins execution in a subsequent cell.

[0075] Figure 8 provides an exemplary timing diagram showing the combined timing of vector units, staggered insertions, LHS corner turns, matrix units, RHS corner turns, destaggers, skid buffers, and value returns to vector units. Stagger is, for example, the difference between the timing of parallel processing in a SIMD / vector scheme and the timing of systolic / pipelined processing. Data flows through matrix units, progressing from top to bottom in time. The left half of the figure shows four vector lanes and corresponding matrix columns, with time progressing vertically in a conceptual stage. Vector data arrives in parallel to the vector-matrix unit. The staggered FIFO introduces an i-cycle latency to the data that will be introduced in the i-th column or i-th row of the matrix unit. The left corner turn has a 2-cycle wire and flip-flop delay. The upper input is delayed by only 2 cycles to ensure that the left and upper inputs arrive at the matrix unit synchronously. The matrix unit has a 4-cycle top-down latency. The right corner turn and matching with the lower delay take an additional 2 cycles. The destaggered FIFO adds a delay of Ni-1 cycles to the i-th column. The skid buffer has enough free space for work equivalent to 2N+2 cycles, so that all preceding logic can always safely flow into it. The right half of the diagram shows the timing per cycle. As some examples show, this can be scaled by using four corner turn cycles, a total of 31 stagger / unstagger cycles, and a latency of 32 cycles. The right half of the diagram shows the effect of staggering as the operand moves through the matrix units.

[0076] The computing system described above may be capable of various operations, including matrix-vector multiplication, sparse matrix-vector multiplication, permutations, scatter and gather steps, vector operations such as shift, rotate, strided slice, concat, and append, sorting, and other operations. According to some examples, the computing system may perform a different permutation in each cycle. Several exemplary operations are described below in relation to Figures 8 to 12.

[0077] Figure 9 shows an example of a gather operation. A gather operation can involve parallel generalization of loading. For example, index 604 may be used to collect data from table 602 and provide output 606. The value of index 604 corresponds to the position in table 602 from which the data is collected. For example, program code for the operation may include the following:

[0078] def serial_gather(table,indices): out = np.zeros(indices.size) for i,index in enumerate(indices): out[i]=table[index] return Figure 10 shows another gather operation, including a parallel generalization of another load. For example, such an operation could be coded as follows:

[0079] def expand_1hot(indices,height=None): out=np.zeros((height,indices.size)) out[indices,np.arange(indices.size)]=1 return def gather(table,indices): return table@expand_1hot(indices,\ len(table)) Figures 11A and 11B illustrate exemplary scatter operations. Referring to Figure 11A, index 804 may be used to scatter data from table 802 to output 806. However, in some cases, address collisions may occur. For example, in Figure 11A, an address collision occurs in the output corresponding to the index value "2" because there are two values ​​of "2" in the index pointed to the same address space in the output. Figure 11B shows an example of using matrix subtraction to resolve address collisions. For example, addition may be used to add collisions. In another example, a maximum operation may be used to determine that the maximum value among any collisions is returned.

[0080] Figures 11=2A-B show an example of sorting, such as argsort, on a vector of integers. As shown in Figure 12A, a comparison is made between the top and left vector data 920 by matrix 910. Matrix 910 takes a linear input, performs quadratic work, and provides a linear output, pipelined once per cycle. The output 906 can be fed into a scatter to sort the data. As shown in Figure 12B, collisions can be broken to assign a unique location to the duplicated data items. This fine-tunes the duplicates, each obtaining a unique scatter index.

[0081] Figure 12=3 shows an example of a merge-sort-step operation. The first sorted vector 1012 is pushed to the top of matrix unit 1010. The second inversely sorted vector 1013 is pushed to the left of matrix unit 1010. The result on the bottom is the sorted vector 1016. Constructing the rightmost output may produce the inversely sorted vector 1018. In some examples, the second cycle can be performed with reverse polarity. Rather than using the second cycle with a modified program to obtain the right-hand (RHS) output, the system may include a wire to return the RHS output by choice. For example, such a wire may, in one cycle, return one of the fringe outputs of the bottom or right array, although both may be available. Both outputs may be obtained through multiple cycles.

[0082] While several exemplary operations are shown above, these are only a few examples, and it should be understood that numerous operations are possible using the computing system described above in relation to Figures 1-5. For example, the computing system may be able to map a large class of double-nested loops and accelerate them in hardware. Due to memory limitations, the bodies of these double-nested loops can only depend on previous loop iterations in each dimension and can only store what fits in the local register file. However, prior loop iterations in each spatial dimension and time storage in the register file allow for very high dependency chains to be mapped onto the computing system.

[0083] The computing units described herein are beneficial in that they are completely static, schedulable, and predictable. The architecture has no contention, arbitration, or queuing; in other words, there is no tail latency. Furthermore, there is no dynamic variation in the timing of the array. The rate at which data propagates through the array can be determined completely statically by analysis by the compiler before program execution.

[0084] In this specification, the phrase “configured to” is used in various contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, it means that the system has appropriate software, firmware, and / or hardware installed on it that, during operation, causes the system to perform one or more operations. When some hardware is said to be configured to perform one or more operations, it means that the hardware includes one or more circuits that, during operation, receive inputs and produce outputs according to the inputs to correspond to one or more operations. When a computer program, engine, or module is said to be configured to perform one or more operations, it means that the computer program includes one or more program instructions that, when executed by one or more computers, causes one or more computers to perform one or more operations.

[0085] While the operations shown in the drawings and described in the claims are presented in a specific order, it should be understood that operations can be performed in a different order than shown, some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the separation of components. The described components, modules, programs, and engines may be integrated together as a single system or may be part of multiple systems.

[0086] Unless otherwise stated, the aforementioned alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. Since these and other variations and combinations of the above features can be used without departing from the subject matter defined by the claims, the foregoing description of embodiments should be interpreted as illustrative rather than as an limitation of the subject matter defined by the claims. Furthermore, the provision of examples described herein, and sections expressed as "such as," "including," etc., should not be interpreted as limiting the subject matter of the claims to specific examples. Rather, the examples are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numerals in different drawings may identify identical or similar elements.

Claims

1. A systolic cell, Crossbar switch and A first arithmetic logic unit (ALU) coupled to the first output of the crossbar switch, A second ALU coupled to the second output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive outputs from the first ALU and the second ALU, The output is provided to the outside of the systolic cell without going through the crossbar switch.

2. The systolic cell according to claim 1, wherein the output of the register file is provided to the input of the crossbar switch.

3. The systolic cell according to claim 1, wherein the crossbar switch is configured to receive outputs from one or more adjacent systolic cells as input to the crossbar switch.

4. The systolic cell according to claim 1, wherein at least one of the first ALU or the second ALU includes a multiplier.

5. The systolic cell according to claim 1, wherein the register file uses registers of 64 bits or less.

6. A crossbar switch and A first arithmetic logic unit (ALU) coupled to the first output of the crossbar switch, A second ALU coupled to the second output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive outputs from the first ALU and the second ALU, The crossbar switch is a 4x4 crossbar switch, and is configured to receive two inputs from the register file and two inputs from adjacent cells, and to provide two outputs to the first ALU and two outputs to the second ALU, forming a systolic cell.

7. The systolic cell according to claim 1, further comprising a third arithmetic logic unit coupled between the crossbar switch and the register file.

8. The system includes a plurality of cells configured such that the first output of the first cell is provided as an input to a second adjacent cell, and the second output of the first cell is provided as an input to a third adjacent cell, and each of the cells is Crossbar switch and One or more arithmetic logic units (ALUs) coupled to the output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive the output from one or more ALUs, The first output is provided as an input to the second adjacent cell without going through the crossbar switch. A systolic array in which the second output is provided as an input to the third adjacent cell without going through the crossbar switch.

9. The systolic array according to claim 8, wherein the output of the register file of each cell is provided to the input of the crossbar switch in the same cell.

10. The systolic array according to claim 8, wherein the one or more ALUs are a plurality of ALUs.

11. The systolic array according to claim 8, wherein the one or more ALUs include a first ALU coupled to the first output of the crossbar switch and a second ALU coupled to the second output of the crossbar switch.

12. A plurality of cells comprising a first cell whose first output is provided as an input to a second adjacent cell, and the second output of the first cell which is provided as an input to a third adjacent cell, each of the cells is Crossbar switch and One or more arithmetic logic units (ALUs) coupled to the output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive the output from one or more ALUs, The one or more ALUs include a first ALU coupled to the first output of the crossbar switch and a second ALU coupled to the second output of the crossbar switch. A systolic array in which the crossbar switch of the fourth cell is configured to receive the output from the first ALU of the third adjacent cell and the output from the second ALU of the second adjacent cell as inputs to the crossbar switch.

13. A systolic array, The system includes a plurality of cells configured such that the first output of the first cell is provided as an input to a second adjacent cell, and the second output of the first cell is provided as an input to a third adjacent cell, and each of the cells is Crossbar switch and One or more arithmetic logic units (ALUs) coupled to the output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive the output from one or more ALUs, The systolic array is configured to receive two source vectors and generate at least one result vector per cycle.

14. One or more memory units, One or more processors that communicate with the one or more memory, The system comprises one or more processors and a plurality of cells that communicate with each other, wherein the plurality of cells are configured such that the first output of a first cell is provided as an input to a second adjacent cell, and the second output of the first cell is provided as an input to a third adjacent cell. Each of the aforementioned cells is Crossbar switch and One or more arithmetic logic units (ALUs) coupled to the output of the crossbar switch, The register file includes, as input to the register file, the register file configured to receive the output from one or more ALUs, The first output is provided as an input to the second adjacent cell without going through the crossbar switch. A computing system in which the second output is provided as an input to the third adjacent cell without going through the crossbar switch.

15. The computing system according to claim 14, wherein the one or more processors include at least one of a scalar core and a vector processing unit.

16. The computing system according to claim 14, wherein the one or more memory includes a vector data cache.

17. The computing system according to claim 14, further comprising one or more processors and a sequencer configured to control instructions transmitted to the plurality of cells.

18. The computing system according to claim 14, wherein the output of the register file of each cell is provided to the input of the crossbar switch in the same cell.

19. The computing system according to claim 14, wherein the crossbar switch is configured to receive outputs from one or more adjacent cells as inputs to the crossbar switch.

20. The computing system according to claim 14, wherein the one or more ALUs are a plurality of ALUs.

21. The computing system according to claim 14, wherein the one or more ALUs include a first ALU coupled to the first output of the crossbar switch and a second ALU coupled to the second output of the crossbar switch.

Images