Hardware circuit for accelerating neural network computations

Abstract

To provide a method, a system, and an apparatus including a computer-readable medium for a hardware circuit that implements a neural network.SOLUTION: In a computing system 100, a hardware circuit 101 includes multiple super tiles 102. Each super tile includes a unified memory 106 for storing inputs to a neural network layer and weights for the layer. Each super tile includes multiple compute tiles 108n. Each compute tile executes a compute thread that is used to perform the computations to generate an output for the neural network layer. Each super tile includes arbitration logic coupled to the unified memory and each compute tile. The arbitration logic is configured to pass inputs stored in the unified memory to the compute tiles, pass weights also stored in the unified memory to the compute tiles, and pass, to the unified memory, the output generated for the layer based on computations performed at the compute tiles using the inputs and the weights for the layer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] background This specification relates generally to circuits for hardware accelerators used to perform neural network computations. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to one or more other layers in the network, e.g., other hidden layers or the output layer of the network. Some of the layers of the network generate an output from the received input according to the current values ​​of a respective set of parameters. Some neural networks are convolutional neural networks (CNNs) (e.g., used in image processing) or recurrent neural networks (RNNs) (e.g., used in speech and language processing).

[0003] CNNs and RNNs are neural networks that include respective sets of convolutional or recurrent neural network layers. A neural network layer can have an associated set of kernels, which may correspond to parameters or weights that are used to process inputs through the layer and generate corresponding outputs for computing neural network inferences. Kernels can be represented as tensors, i.e., multidimensional arrays, of weights. As an example, a neural network layer in a sequence of layers can process a set of inputs, such as image pixel data or activation values ​​generated by another neural network layer in the sequence. The set of inputs or activation values ​​can also be represented as a tensor. Summary of the Invention [Problem to be solved by the invention]

[0004] overview This document describes an improved hardware circuit that may be used in a hardware accelerator configured to accelerate the computation of an exemplary neural network model, such as the computation of a layer of an artificial neural network. The circuit architecture includes multiple supertiles, each configured to execute multiple computational threads based on data retrieved from the supertile's unified memory. The unified memory provides a memory configuration that can be efficiently shared among each of the computational threads, such that the computations of each of the computational threads can be performed simultaneously in the supertile.

[0005] In some implementations, the described hardware circuits and processing techniques may be used in an exemplary computing system, such as a small or large-scale distributed system, that includes circuitry for multiple dedicated processors (e.g., hardware accelerators) used to perform inference (or training) calculations for exemplary machine learning workloads. The circuit architectures described herein may be integrated into each of the multiple dedicated processors to increase the speed and efficiency with which the multiple dedicated processors perform calculations to perform tasks for various types of machine learning models. [Means for solving the problem]

[0006] One aspect of the subject matter described herein may be embodied in a circuit for a hardware accelerator configured to implement a neural network including multiple neural network layers and to perform computations to generate outputs for the neural network layers. The circuit includes multiple supertiles, each of which includes a unified memory configured to store inputs to the neural network layers and multiple weights for the neural network layers, and multiple computation tiles, each configured to execute computational threads used to perform the computations and generate outputs, and each of the multiple supertiles further includes an arbitration logic unit coupled to the unified memory and each of the multiple computation tiles. The arbitration logic unit is configured to pass one or more of the inputs stored in the unified memory to each of the computation tiles, pass a respective set of weights stored in the unified memory to each of the computation tiles, and pass to the unified memory an output generated for the neural network layer based on a computation performed in each of the computation tiles using the one or more of the inputs and the respective set of weights.

[0007] These and other implementations may each optionally include one or more of the following features: For example, in some implementations, the circuitry includes a respective controller for each supertile, each controller configured to generate one or more control signals, the one or more control signals used to store each of the inputs to the neural network layer in a corresponding location in the unified memory, each corresponding location identified by a respective address, the one or more control signals further used to store each weight of the plurality of weights for the neural network layer in a corresponding location in the unified memory, each corresponding location identified by a respective address, and the one or more control signals further causing the arbitration logic to pass the one or more inputs to a computational cell of a particular computational tile and pass each set of weights to a particular computational tile.

[0008] In some implementations, the controller is configured to store each set of weights for a particular computational tile in a respective register file of the particular computational tile that is local to the particular computational tile. In some implementations, the controller is configured to determine partitions of addresses within the unified memory for storing each batch of inputs to be passed to a corresponding computational tile of the SuperTile, each partition of addresses being assigned to a respective computational tile of the SuperTile.

[0009] In some implementations, each address in the partition of addresses corresponds to an input in a batch of inputs forming a sample of input features, the sample of input features including a plurality of sets of input features, the plurality of sets of input features corresponding to a stream of image or audio data. In some implementations, the arbitration logic unit is configured, for a first partition of addresses, to retrieve the first batch of inputs from memory locations identified by addresses in the partition of addresses and pass the first batch of inputs to cells of a first computational tile, the first computational tile being assigned to receive each input in the first batch of inputs based on the determined partition of addresses in the unified memory.

[0010] In some implementations, for each respective supertile, multiple computation tiles are Each computational tile is configured to execute two or more computational threads in parallel in the computational tile, and each computational tile executes the computational threads to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate a partial output for the neural network layer.

[0011] In some implementations, for each respective supertile, each computational tile of the plurality of computational tiles performs a portion of the computation to generate an output for a neural network layer in response to executing two or more computational threads in parallel in the computational tile. In response to executing the unit, the unit is configured to generate one or more partial outputs used to generate outputs for the neural network layers.

[0012] In some implementations, the circuitry is configured to, for each respective computational tile of a plurality of computational tiles in a supertile, execute two or more computational threads in parallel in the computational tile. And, for each respective supertile of the plurality of supertiles, the circuitry is configured to execute two or more computational threads assigned to each computational tile in parallel to generate output for the neural network layer. In some implementations, a first portion of the operations performed using the computational threads corresponds to a first set of tensor operations for traversing one or more dimensions of a first multidimensional tensor, the first multidimensional tensor being an input tensor including data elements corresponding to inputs stored in the unified memory.

[0013] In some implementations, the second portion of operations performed using the computational threads corresponds to a second set of tensor operations for traversing one or more dimensions of a second multidimensional tensor that is different from the first multidimensional tensor, the second multidimensional tensor being a weight tensor that includes data elements corresponding to a plurality of weights stored in the unified memory.

[0014] One aspect of the subject matter described herein may be embodied in a method for performing computations to generate outputs for neural network layers of a neural network using circuitry for a hardware accelerator configured to implement a neural network including multiple neural network layers. The method includes receiving, at a supertile of multiple supertiles, inputs to the neural network layer and multiple weights for the neural network layer, and storing the inputs to the neural network layer and the multiple weights for the neural network layer in a unified memory of the supertile. The method also includes passing, using an arbitration logic unit of the supertile, one or more of the inputs stored in the unified memory to each computational tile of multiple computational tiles in the supertile, the arbitration logic unit being coupled to the unified memory and each computational tile of the multiple computational tiles, and the method further includes passing, using the arbitration logic unit of the supertile, a respective set of weights stored in the unified memory to each of the computational tiles. The method includes executing a computational thread in each of the computational tiles in the SuperTile to perform computations to generate outputs for the neural network layer, and generating outputs for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and a respective set of weights.

[0015] These and other implementations may each optionally include one or more of the following features: For example, in some implementations, the method includes passing the outputs generated for the neural network layers to a unified memory using an arbitration logic unit, and passing the outputs generated for the neural network layers to another SuperTile in the circuit using a controller of each of the SuperTile.

[0016] In some implementations, the method includes generating a control signal by a controller of each of the supertiles, and storing each of the inputs to the neural network layer in a corresponding location of the unified memory based on the control signal, each of the corresponding locations identified by a respective address, the method further includes storing each weight of the plurality of weights for the neural network layer in a corresponding location of the unified memory based on the control signal, each of the corresponding locations identified by a respective address, the method further includes causing arbitration logic to pass one or more inputs to a computational cell of a particular computational tile and pass each set of weights to a particular computational tile based on the control signal.

[0017] In some implementations, the method comprises performing multiple calculations for each respective supertile. and executing each computational thread of two or more computational threads in parallel in each computational tile of the tile, each computational tile executing the computational thread to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate a partial output for the neural network layer.

[0018] One aspect of the subject matter described herein may be embodied in a system-on-chip (SoC) comprising: circuitry for a hardware accelerator configured to implement a neural network comprising a plurality of neural network layers and perform calculations to generate outputs for the neural network layers; and a host controller configured to access a memory external to the circuitry for the hardware accelerator, the memory configured to store data for processing by the neural network layers; and a host interface configured to exchange data communications between the circuitry for the hardware accelerator and the host controller.

[0019] The SoC includes a plurality of supertiles arranged in a circuit. Each supertile of the plurality of supertiles includes a unified memory configured to store inputs to a neural network layer and a plurality of weights for the neural network layer. The inputs and the plurality of weights correspond to data stored in memory accessible by a host controller. Each supertile includes a plurality of computational tiles, each computational tile configured to execute a computational thread used to perform a computation and generate an output. Each supertile of the plurality of supertiles includes an arbitration logic unit coupled to the unified memory and to each computational tile of the plurality of computational tiles.

[0020] The arbitration logic unit is configured to pass one or more of the inputs stored in the unified memory to each of the computational tiles, pass a respective set of weights stored in the unified memory to each of the computational tiles, and pass to the unified memory an output generated for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and the respective set of weights.

[0021] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof that is installed on the system and that, when executed by a data processing device, causes the device to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions. [Effects of the Invention]

[0022] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages: The circuit architectures and data processing techniques described herein can be integrated into exemplary distributed systems to reduce the processing time required to process a set of inputs through layers of a neural network, such as a convolutional or recurrent neural network.

[0023] The circuit architecture and data processing techniques provide a different combination of approaches for optimizing how computations are parallelized across tiles compared to prior art circuit designs for performing neural network computations. For example, the described techniques allow for optimizing how computations are parallelized across tiles for use cases with significant reuse of data between computations, such as when activations are reused across different dimensions of a filter and parameters are reused in batches across multiple activations. do.

[0024] The present technology can be used to implement a circuit architecture and software stack that provides one or more supertiles that allow for multiple simultaneous computational threads within the supertile. This architecture enables processing techniques that include determining whether to broadcast and / or slice both parameters (weights) and activations. This determination can be different for different types of workloads to optimize the performance of an exemplary hardware accelerator incorporating this architecture.

[0025] The optimization can be linked to the utilization of the sum-of-products cells in the computational units of the hardware circuit. The utilization can be evaluated with reference to different approaches for partitioning the dimensions of a tensor across supertiles based on the improved circuit architecture, for example, partitioning the Z dimension of the tensor across four supertiles or partitioning the X and Y dimensions of the tensor across two supertiles. For example, using the techniques described herein, multiple approaches can be used to parallelize computation across tiles, thereby allowing the sum-of-products cells of the circuit to achieve a threshold utilization (e.g., 70%) higher than the utilization of the cells in prior art circuit designs.

[0026] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram of a computing system including an exemplary circuit for a hardware accelerator. [Figure 2] FIG. 1 is a block diagram of an exemplary computational tile architecture of a circuit for a hardware accelerator. [Figure 3] 1 shows an example tensor and example program code for processing data corresponding to elements of the tensor. [Figure 4] 10 shows a table containing example instructions of an instruction set architecture for one or more supertiles. [Figure 5] FIG. 1 is a flow diagram illustrating an exemplary process for accelerating neural network computations. DETAILED DESCRIPTION OF THE INVENTION

[0028] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description This specification describes improved hardware circuits and data processing techniques that can be implemented using the improved hardware circuit architectures. The hardware circuits can be special-purpose processors such as neural network processors, application specific integrated circuits, or hardware accelerators.

[0029] The hardware circuit includes multiple supertiles. Each supertile includes a unified memory for storing inputs to and weights for a neural network layer. Each supertile is configured to execute multiple computational threads based on data retrieved from the supertile's unified memory and instructions received at the supertile via a communication bus coupled to each of the supertiles. In some implementations, each supertile includes multiple computational tiles, each configured to execute one or more computational threads. In some cases, each computational tile is configured to execute one computational thread, such that the supertile can execute multiple computational threads in parallel. In other cases, each computational tile can be configured to execute multiple computational threads, such that a SuperTile executes each of the multiple computational threads in parallel. The computational threads are used to perform calculations and generate outputs for the neural network layers.

[0030] Each supertile includes an arbitration logic unit coupled to a unified memory and to each computational tile or each computational thread that may execute in that supertile. The arbitration logic unit is configured to pass inputs and weights stored in the unified memory to the computational tiles. The arbitration logic unit is also configured to pass outputs generated for a layer to the unified memory of the supertile assigned to receive the output, or to each of one or more supertiles assigned to receive a portion of the output.

[0031] In some implementations, outputs for neural network layers are generated in supertiles based on computations performed in the computation tiles of the supertiles, with inputs and weights for the layer passed to the computation tiles by an arbitration logic unit. In other implementations, one or more layers of a neural network may be divided across multiple supertiles, e.g., a layer may be parallelized across multiple supertiles, with each supertile performing a portion of the processing for the layer. In these implementations, outputs for neural network layers are generated across multiple supertiles as respective sets of output values ​​(e.g., vectors of activation values) that together form the output for the neural network layer.

[0032] 1 is a block diagram of a computing system 100 including exemplary circuitry for a hardware accelerator. In some cases, system 100 is an exemplary computing system for accelerating tensor or neural network computations associated with an artificial deep neural network (DNN), such as an RNN or CNN. For example, system 100 is configured to implement an exemplary artificial neural network (e.g., a CNN) on hardware circuitry 101, such as a dedicated hardware circuit. In some implementations, system 100 is a system-on-chip. For example, the system-on-chip may include hardware circuitry 101 and some (or all) of the other components and devices described herein as being included in system 100.

[0033] The hardware circuit 101 may be a hardware accelerator configured to accelerate the execution and / or performance of a neural network model. For example, the execution of the neural network model may be accelerated relative to the execution of the model on an exemplary general-purpose machine, such as a central processing unit (CPU). Similarly, the performance and execution of the neural network model may be accelerated compared to when the model is implemented on another hardware accelerator (e.g., a graphics processing unit (GPU)) that does not have the improved hardware features and software functionality associated with the techniques described herein.

[0034] A system 100 including the exemplary circuit 101 can include one or more SuperTiles 102. In some implementations, the system 100 includes multiple SuperTiles 102. In the example of FIG. 1 (and FIG. 2 described below), the system 100 is shown as including four SuperTiles 102, but the system 100, and the hardware circuit 101 described herein, may include more or fewer SuperTiles. As described in more detail below, a SuperTile 102 is a discrete, self-contained computing unit of the system 100 (or hardware circuit 101). In some implementations, each SuperTile 102 is a multi-layer neural network (MLN). It is configured to independently perform the computations required by one or more layers of the work (e.g., neural network computations).

[0035] Computations may be required to process data for a machine learning workload or to perform a particular task of that workload. In some implementations, the computational process performed within SuperTiles 102 for one or more neural network layers may include multiplication of data values ​​stored in each element of an input tensor (e.g., inputs or activations) by data values ​​stored in each element of a parameter tensor (e.g., weights). For example, the computation may include multiplying an input or activation value by a weight value over one or more cycles and performing an accumulation of the product over many cycles.

[0036] Each supertile 102 generally includes a respective controller 104, a respective unified memory 106, a respective number of computational tiles (or threads) 108, and a respective arbitration logic unit 110 ("arbitration logic 110").

[0037] Controller 104 is configured to generate control signals 114 for controlling operations occurring within SuperTile 102. For example, control signals 114 may be used to: a) store each received input to a neural network layer in a corresponding location in unified memory 106; and b) store each received weight for a neural network layer in a corresponding location in unified memory 106. Each corresponding memory location storing a respective input or weight is identified by a respective address.

[0038] Controller 104 includes a direct memory access (DMA) module 105 that includes a DMA operation (“DMAOp”) control 105a and a DMAOp tensor traversal unit (TTU) 105b. DMAOp control 105a represents control logic that may be used by controller 104 to: i) manage the writing / storing of data for computation to memory locations in unified memory 106; and ii) manage the reading / retrieving of data for computation from memory locations in unified memory 106. For example, DMAOp control 105a is executed by controller 104 to manage the writing of inputs of input tensors received at supertiles 102 to memory locations in unified memory 106 and the writing of weights of weight tensors received at supertiles 102 to memory locations in unified memory 106.

[0039] DMAOp control 105a is operable to manage traversal operations for execution by DMAOp TTU 105b. In some implementations, the locations or addresses in unified memory 106 to which particular inputs or activations are written or read are generated by DMAOp TTU 105b based on inbound / outbound DMAOp instructions received over communication bus 124 (described below). For example, DMAOp instructions may be processed by DMAOp control 105a to manage the traversal operations performed by DMAOp TTU 105b to generate the locations or addresses in unified memory 106 used to store inputs and weights received over communication bus 124.

[0040] In some cases, inbound and outbound DMAOps may be executed simultaneously. An exemplary outbound DMAOp may include a SuperTile 102 providing activation values ​​for generated layer outputs to adjacent SuperTile 102 in system 100. During the concurrent execution of inbound and outbound DMAOps, any necessary synchronization or arbitration of memory location accesses may be managed through a synchronization flag control scheme managed by controller 104. In some implementations, the controller The controller 104 is operable to manage the synchronization flag control scheme in conjunction with the arbitration logic 11 .

[0041] Control signals 114 generated by controller 104 may also be used to a) cause read arbitration logic 110a to pass one or more inputs obtained from unified memory 106 to arithmetic cells 152 (described below) of a particular computational tile 108n, and b) cause read arbitration logic 110a to pass each set of weights obtained from unified memory 106 to a particular computational tile 108n. In some implementations, arbitration logic 110 passes the inputs and weights to computational tile 108n via input bus 112.

[0042] 1, arbitration logic 110 may be coupled to each computational tile 108n of supertile 102 via a respective input bus 112 and a respective output bus 113. Arbitration logic 110 is configured to retrieve (or read) multiple batches of inputs from memory locations in unified memory 106. Arbitration logic 110 is also configured to store (or write) multiple sets of outputs or output activations provided by each computational tile 108 to memory locations in unified memory 106.

[0043] In some examples, the unified memory 106 may be described as a narrow memory structure operable to store inputs, activations, or gain values ​​to be processed in a neural network layer and to output activations generated by the neural network layer in response to processing the inputs or activations through the neural network layer. The generation and storage of output activations will be described in more detail. The unified memory 106 of each supertile 102 may employ a memory hierarchy that provides addressing arbitration and flexibility that allows traversing multidimensional arrays in any order while also avoiding bank contention for certain memory operations, such as single-cycle read and write operations. In some implementations, the unified memory 106 includes multiple memory banks (e.g., multiple independently arbitrated memory banks), and the arbitration logic 110 is configured to arbitrate read and write accesses to each memory location in each memory bank within the unified memory 106.

[0044] Each batch of inputs passed by the arbitration logic 110 may correspond to a particular computational tile 108n, and the batch of inputs is provided to the particular computational tile 108n via a respective input bus 112 that couples the particular computational tile 108n to the arbitration logic 110. For example, the arbitration logic 110 is configured to load each input of a first batch of inputs onto a first input bus 112 that couples the arbitration logic 110 to a first computational tile 108n in a supertile. The arbitration logic 110 is also configured to load each input of a second, different batch of inputs onto a second, different input bus 112 that couples the arbitration logic 110 to a second, different computational tile 108n in the supertile 102. Alternatively, in some cases, each of multiple batches of inputs may correspond to the same computational tile 108n and be loaded at the same computational tile 108n.

[0045] Arbitration logic 110 is a logical unit or structure of unified memory 106. For example, arbitration logic 110 may be a dedicated memory arbiter used in a shared memory system (e.g., unified memory 106) to determine, for each memory cycle, which control unit (e.g., DMAOp control 105a or TensorOp control 132) is allowed to access the shared memory resources of unified memory 106. For example, in SuperTile 102, different instruction types of DMAOp control 105a and TensorOp control 132 may be configured as independent control threads requesting memory access, and the requests must be arbitrated by arbitration logic 110.

[0046] As described herein, each SuperTile 102 runs k computational threads. In some implementations, each of the k computational threads is a software construct that runs in a respective SuperTile 102, and portions of the k computational threads may be managed or executed by respective computational tiles 108n of the SuperTile 102. A SuperTile 102 may be a superscalar tile or a supervector tile that represents an independent computational unit in which multiple TensorOp pipelines (or threads) run in parallel, i.e., simultaneously. For example, a parameter or variable kNumberComputeThreads (k computational threads) may be used to represent the number of superscalar or supervector tiles that run in parallel, i.e., simultaneously. It may represent the number of parallel TensorOp pipelines in a superscalar tile 102 or a supervector tile 102. A superscalar tile 102 may be an exemplary supertile 102 that operates on scalar input values, and a supervector tile 102 may be an exemplary supertile 102 that operates on vectors of input values.

[0047] In a Supertile 102, each computational thread may correspond to a single computational tile, and the computational tile executes a single computational thread. Alternatively, each computational tile may be configured to execute multiple computational threads. In some implementations, a set of computational tiles 108n may be physically or logically disposed within each Supertile 102 of the system 100. For example, in the system 100 (or hardware circuit 101), the set of computational tiles 108n for each Supertile 102 may be configured in hardware or software. In some implementations, when the computational tiles 108n for each Supertile 102 are configured in software, the Supertile 102 may be configured to execute n computational tiles 108n, where n is an integer greater than or equal to 1. In these implementations, each of the n computational tiles 108n may be configured to execute k computational threads.

[0048] Control signals 114 generated by the controller 104 can also be used to a) cause the write arbitration logic 110b to pass the generated layer output activations to the unified memory 106 for storage in the unified memory 106, and b) cause the supertile 102 to provide the generated layer output activation values ​​to adjacent supertiles.

[0049] System 100 includes an external host / controller 120 coupled to each of SuperTiles 102 via a host interface 122. In some implementations, host interface 122 is coupled between host controller 120 and circuitry for a hardware accelerator (e.g., hardware circuit 101), which may be included in a system-on-chip. Host interface 122 is configured to exchange data communications between host controller 120 and circuitry for the hardware accelerator. In some implementations, host controller 120 is configured to access memory (e.g., external memory) external to the circuitry for the hardware accelerator. The external memory is configured to store data for processing in a neural network implemented in the circuit. For example, the data may be inputs and weights to be processed by one or more layers of the neural network.

[0050] The host interface 122 receives instructions and data values ​​from the external host / controller 120 and provides a respective set of instructions and data values ​​to each of the supertiles 102. In some examples, the data values ​​may be retrieved from an external memory accessible by the host controller 120 and then passed to the supertiles 102 via the host interface 122. The host interface 122 is operable to use an exemplary communication bus accessible by each of the supertiles 102 to pass instructions and data values ​​to the supertiles. In some implementations, the instruction set architecture of the system 100 allows each of the supertiles 102 to issue a respective single instruction. A single instruction may include data values ​​(e.g., inputs and weights), specific data fields, and operational parameters for a workload or a set of tasks within a workload.

[0051] Generally, instructions and data values ​​are provided to one or more devices in system 100 via a communication bus 124 (e.g., an instruction bus or a ring bus). In some cases, SuperTiles 102 receive data and instructions for machine learning tasks via an exemplary communication bus 124 that couples two or more SuperTiles in system 100. For example, communication bus 124 is configured to provide communications coupling SuperTiles 102 of system 100 via a bus data path connecting them to host controller 120 via host interface 122 in an exemplary ring format. The ring format is shown in the example of FIG. 2.

[0052] In some implementations, one or more instructions are initially received by each of the respective controllers 104 in the SuperTile 102 from the host interface 122 and stored in an exemplary instruction memory of the respective controller 104 for later execution by the controller 104. The data may include inputs, activations, gain values, or a combination of each. In some examples, the data is received at the SuperTile 102 and processed at the neural network layer to generate outputs for the neural network layer. In such examples, processing the data at the neural network layer to generate layer outputs includes generating multiple partial outputs (e.g., accumulated values ​​or pre-activation values).

[0053] Each of the computational tiles 108n includes a respective tensor module 130 that includes tensor operation (“TensorOp”) control 132 and TensorOp TTU 134. Each of the respective tensor modules 130 may provide functionality similar to or related to that provided by the DMAOp module 105 of the controller 104. For example, the TensorOp control 132 may represent control logic used by the controller 104 or the computational tile 108n to: i) manage operations for reading / accessing input values ​​assigned to particular elements of an input tensor from corresponding memory locations in the unified memory 106 that store the inputs; and ii) manage the association or assignment of output values ​​(or partial outputs) to particular elements of an output tensor after the output values ​​are generated in response to one or more computational threads executing in the computational tile 108n.

[0054] TensorOp control 130 may be executed by the controller 104 or a computation thread of a computation tile 108n to manage traversal operations for execution by TensorOp TTU 134. For example, TensorOp TTU 134 is operable to execute instructions to access a set of elements along a particular dimension of an N-dimensional or multidimensional tensor (e.g., a 2D input tensor, a 3D weight tensor, or a 4D output tensor). An exemplary N-dimensional tensor may have multiple elements arranged across each of the N dimensions, where N is an integer greater than or equal to 1.

[0055] The TensorOp TTU 134 determines the address of each element in a set of elements along a particular dimension of a tensor (e.g., a 2D weight tensor) so that a computational tile 108n (or computational thread) may access the corresponding memory or register file that stores data for that tensor to read data representing the value of the element along that particular dimension. In some implementations, the program code associated with the TensorOp TTU 134 may include one or more nested loops, and the TensorOp TTU 134 is instructed to access elements of a 2D array / tensor variable within the nested loop according to the nested loop's current index variable value. Based on the current index variable value of the nested loop, TensorOp TTU 134 may determine an offset value representing an offset from the first element of the two-dimensional array variable. For example, the address of a particular element may be an offset address from another element of an N-dimensional tensor.

[0056] Each of the computational tiles 108n includes a wide memory structure 140 that includes multiple local register files 142. In some implementations, the controller 104 is configured to store the set of weights for a particular computational tile 108n in a respective register file of the particular computational tile 108n, where the particular register file 142 is local to the particular computational tile 108n. For example, the controller 104 is configured to store individual weights of the set of layer weights in particular memory locations of the local register file 142 in response to passing the set of layer weights from the unified memory 106 to the particular computational tile 108n.

[0057] Each of the computational tiles 108 n includes a respective computation unit 150 configured to perform arithmetic operations, such as addition and multiplication, using operands corresponding to inputs and weight values ​​passed to the computational tile 108 n. Each of the computational units 150 may include a plurality of arithmetic cells 152. Each arithmetic cell 152 may be a multiply-accumulate cell configured to perform an arithmetic operation (e.g., multiplication) using an input and a weight. For example, the arithmetic operations performed by the computational units 150 generally include multiplying inputs or activations obtained from the unified memory 106 by parameters to generate a set of accumulation values. Parameters for the computations may be obtained from the wide memory structure 140 of the computational tile 108 n, which includes a plurality of local register files 142.

[0058] Each of the computational tiles 108n includes a register array 160 and a nonlinear unit 170 (“NLU 170”). The register array 160 includes a plurality of individual shift registers 162. Each shift register 162 may be a pipelined shift register 162. The pipelined shift registers 162 of the array 160 are used to shift the output values ​​(e.g., accumulation values ​​or partial sums) of the layer to the nonlinear unit 170 (“NLU 170”). The NLU 170 applies a nonlinear activation function to the output values ​​to generate a set of output activations for the layer. The NLU 170 interacts with the write arbitration logic 110b to pass the generated output activations of the layer outputs to the unified memory 106 for storage in the unified memory 106. For example, the output activations may be provided from the NLU 170 to the write arbitration logic 110b via an output activation bus 113.

[0059] In some implementations, the NLU 170 is operable to aggregate multiple partial sums or accumulated values ​​into a final linear output (e.g., a vector of values) based on control signals provided to the NLU 170 from the computational tiles 108n or by the controller 104.

[0060] Figure 2 is a block diagram illustrating an exemplary computational tile architecture of a circuit for a hardware accelerator. The example block diagram of Figure 2 includes a first tile architecture 200 and a second, different tile architecture 210. The first tile architecture 200 represents a tile architecture of an exemplary prior art circuit design of a dedicated hardware circuit, and the second tile architecture 210 represents a new tile architecture of an improved hardware circuit based on the techniques described herein.

[0061] The new tile architecture 210 includes multiple supertiles 102. For context, some prior art approaches that perform neural network computations using individual computation tiles 202 and computation threads 204 are limited in how the computations can be parallelized across the architecture. In contrast to these prior art approaches, the new tile architecture 210 includes multiple supertiles 102 and supertiles 102. The neural network architecture allows for parallelization options within a computational tile 108n of a supertile 102 and across multiple supertiles 102. For example, each supertile 102 may be configured to execute multiple computational threads 214, each of which may execute simultaneously in the supertile 102. In some cases, the simultaneous execution of multiple threads reduces or mitigates processing latency compared to prior art approaches that may require serial execution of two or more computational threads when processing inputs in a layer of a neural network.

[0062] Each of the multiple computational threads 214 executing in a Supertile 102 may be based on data retrieved from the unified memory 106 of the Supertile 102, instructions received at the Supertile 102, control signals 114 generated by the controller 104, or a combination of each. In some implementations, the multiple computational threads executing in each Supertile correspond to one or more tensor operations. In the example of FIG. 2, each Supertile 102 is shown as performing four separate tensor operations, but each Supertile 102 can be configured to perform more or fewer tensor operations.

[0063] In some implementations, for an example computation associated with a neural network layer that uses a 2D input tensor having X and Y dimensions, the external / host controller 120 is operable to execute an input partitioning algorithm to distribute the outputs X and Y across a grid of supertiles 102 (e.g., the novel tile architecture 210). The external / host controller 120 is operable to allocate space in each of the respective unified memories 106 for each supertile 102 to store input activations, halo pixels, and output activations. In the context of an image processing workload, the halo pixels correspond to inputs that are shared between two or more computational tiles 108n. For example, a set of inputs corresponding to the halo pixels may be used in a convolution in which inputs for the edges of an image are shared.

[0064] 2, the first partitioning algorithm 220 includes a loop nest that can be used to express the net (total) work performed by the SuperTile 102. The partitioning algorithm 220 and the loop nest can be represented by portions of program code executed by each TensorOp TTU 134 of different computational tiles 108n in the SuperTile 102. For example, a variation of the partitioning algorithm 220 may be executed by each TensorOp TTU 134 across multiple computational tiles 108n to traverse particular elements along different dimensions of an exemplary 3D input tensor (x, y, zin) to convolve the 3D input tensor with a 2D weight (filter) tensor (kx, ky) to generate a 1D output tensor (zout). This is described in more detail below with reference to FIG. 3.

[0065] 3 illustrates an example tensor 300 (e.g., a 3D input tensor) and a second partitioning algorithm 310 for processing data corresponding to elements of the tensor 300. Based on the novel tile architecture 210 described above, the improved hardware circuit 101 described herein supports multiple approaches and algorithms by which work, such as tasks and computations, can be divided among kNumberComputeThreads for TensorOp threads 304 and 306 executing in a Supertile 102 or across different Supertiles 102. and methods are provided.

[0066] For example, a different combination of approaches for dividing work among each of the computational tiles 108n may include: a) assigning a first set of elements for the X, Y dimensions of the tensor 300 to a first computational tile 108n of a first supertile 102; and b) assigning a second set of elements for the X, Y dimensions of the tensor 300, or for the other dimension of the tensor 300, to a first computational tile 108n of a first supertile 102. The method may include assigning the computed data to a second, different computational tile 108n of the first SuperTile 102.

[0067] Different combinations of approaches may also be used to divide work among each of the multiple supertiles 102 and among a respective plurality of computational tiles 108 n in each supertile 102. For example, one combination of approaches may include i) assigning different sets of elements in the X and Y dimensions of the tensor 300 to at least two computational tiles 108 n of a first supertile 102, and ii) assigning different sets of elements in the X and Y dimensions of the tensor 300 to one or more computational tiles 108 n of a second, different supertile 102.

[0068] If the elements of the X and Y dimensions allocated to a SuperTile 102 are large (e.g., exceed a threshold size of the SRAM in a computational tile 108n), multiple computational threads can work on additional 2D subdivisions of the allocated X and Y dimensions. In some implementations, for image processing workloads, the data of the 2D subdivisions can be processed without requiring explicit exchange of halo pixels 302 across one or more computational threads in a SuperTile 102. In some implementations, input pixels needed by one or more computational threads in a SuperTile 102 first reside in the unified memory 106 of the SuperTile 102 before being passed to the corresponding computational threads.

[0069] As explained above, the circuit architectures and data processing techniques described herein provide different approaches (or combinations of approaches) for optimizing how computations are parallelized across tiles compared to prior art circuit designs for performing neural network computations. In some cases, the optimization can be tied to the utilization of multiply-accumulate cells 152 in a computation unit 150 for different options for partitioning two or more tensor dimensions across supertiles 102 of the improved circuit architecture. By way of example, some common options include partitioning the Z dimension of an input tensor across four supertiles, or partitioning the X, Y dimensions of a 2D tensor across two supertiles.

[0070] For example, multiple approaches may be used to parallelize computations across tiles such that a multiply-accumulate cell 152 of a computation unit 150 can achieve a higher threshold utilization (e.g., 70%) than the utilization of the associated cell in a prior art circuit design. In some cases, the higher threshold utilization for each of the multiple different approaches may be higher than the utilization of the prior art design, even if the prior art design has limited options for how computations may be parallelized across its circuit architecture.

[0071] The approach provided by one or more Supertiles 102 allows for a portion (e.g., some or all) of the input tensors assigned to a Supertile 102 to be further divided among and operated on by different computational tiles 108n within the Supertile 102, a portion (e.g., some or all) of the parameter tensors assigned to a Supertile 102 to be further divided among and operated on by different computational tiles 108n within the Supertile 102, or both. Similarly, this approach allows for the processing in a neural network layer to be divided across two or more Supertiles 102; for example, the neural network layer may be parallelized across multiple Supertiles 102, with each Supertile performing a portion of the processing for that neural network layer. For example, the entire neural network layer may be partitioned across all (or some) of the Supertiles 102. Generally, multiple options for parallelization provide improved implementations of this approach. This can be pursued using circuit architecture.

[0072] Thus, different approaches may be used to allocate work and partition the elements and dimensions of tensor 300, and different combinations of supertiles 102 and computational threads for each supertile 102 may be used to traverse particular elements along different dimensions of N-dimensional tensor 300 to convolve (or perform other operations on) tensor 300 with an N-dimensional weight (filter) tensor to generate an N-dimensional output tensor. Thus, one or more N-dimensional tensors accessible from unified memory 106 and wide memory structure 140 in a single supertile 102 may be traversed based on memory address values ​​processed by respective TensorOp TTUs 134 within the supertile 102.

[0073] The system 100 is configured to determine a partition of addresses among each computational thread of a plurality of computational threads for a given supertile 102. The address partition may be determined based on a particular approach for allocating work and partitioning elements and dimensions of tensors being processed in the system 100. In some implementations, the DMAOp control 105a is operable to determine a mapping of addresses in the partitions for each input in a batch of inputs to be processed through a neural network layer. For example, each batch of inputs may be associated with a different element of the input tensor 300, and each partition of addresses may be assigned to a particular computational tile 108n or computational thread executing in the computational tile 108n.

[0074] FIG. 4 shows a table 400 containing example instructions of an instruction set architecture for one or more supertiles.

[0075] As described above, the instruction set architecture of the system 100 can be configured so that each of the supertiles 102 receives a respective single instruction (or multiple instructions). Each of the single or multiple instructions can include data values ​​(e.g., inputs and weights), specific data fields, and operational parameters for a workload or a set of tasks in a workload. Thus, each of the one or more instructions provided to the supertiles 102 via the communication bus 124 can include multiple parameters or data fields. Each of the data fields can be associated with a particular operation. In some cases, one or more bits of a data field in an instruction can be set to a particular binary value that causes a particular operation to occur in a single computational tile 108 or multiple computational tiles 108.

[0076] Referring now to table 400, a data field for an exemplary tensor operation (“TensorOp”) to be executed in a computational thread of a particular computational tile 108 n indicates the TensorOp pipeline of the target thread 402. In some implementations, based on an instruction received at a supertile 102, multiple data fields may be simultaneously multicast to each of the computational tiles 108 n for each computational thread to be executed in the computational tile 108 n.

[0077] The data field for an exemplary DMA operation (“NarrowToWide DMA”) indicates 404 the wide memory configuration 140 of the target thread that will receive the data retrieved from the unified memory 106. In some implementations, the DMA operation may be performed in the supertile 102 to move data representing each set of weights for a neural network layer from the unified memory 106 (e.g., narrow memory) to the local register file 140 of the wide memory configuration 140. For example, the set of weights is moved to the local register file 142 of the wide memory configuration 140 of the target computational thread. In some implementations, an example operation performed by a target computation thread may include TensorOp TTU 134 obtaining a weight value from local register file 142, passing the weight value from computation tile 108 n to cell 152, and cell 152 using the weight value as an operand for a neural network computation performed to generate an output for the neural network layer.

[0078] The data field for another DMA operation ("RingBusConsumer DMA") is A target thread's wide memory configuration 140 is shown (406) receiving a portion of the data contained in (or included with) the instruction provided to the tile 102. In some implementations, the data field for this DMA operation may correspond to a particular bitmap field in the instruction obtained from the communication bus 124 (e.g., the ring bus). In general, the bitmap may have a particular width defined in terms of bits.

[0079] For example, the header (e.g., a bitmap) of an instruction may indicate to a receiving supertile 102 how the supertile 102 should consume the portion of data associated with the header based on the values ​​of individual bits in the bitmap field of the header. The specific manner in which the supertile 102 is required to consume that portion of data may be the instruction subtype (or instruction subtype). In some implementations, the controller 104 of each receiving supertile 102 examines the header bitmap of an instruction (e.g., a single instruction) and determines that the instruction subtype indicates that the portion of data should be received by the wide memory configuration 140 of the supertile 102. For example, the instruction subtype may indicate which local register file 142 of the target thread should receive the respective set of weights associated with that portion of data.

[0080] The data fields for another example operation ("LoadCoefficientTables") are The instruction indicates 408 the memory of the supertile 102 to load the coefficient table included in (or with) the instruction given to the supertile 102. The data field for this load operation is the same as the bit field for the RingBusConsumer DMA operation described above. The coefficient table may correspond to a particular bitmap field in the instruction that is different from the bitmap field. In some implementations, the coefficient table is used by each of the target threads of the SuperTile 102 to perform neural network computations for the exemplary machine learning workload. In some cases, the coefficient table may be stored across the respective wide memory configurations 140 associated with each computation thread. In other cases, the coefficient table may be stored in some other dedicated memory of the SuperTile 102 that is accessible by each of the k computation threads.

[0081] The data field for the synchronization flag operation ("SyncFlag") indicates the synchronization flag of the target thread (410). In some implementations, the data field for the exemplary synchronization flag operation in the instruction is set only for synchronization flags that are replicated across two or more SuperTiles 102. The data field for the synchronization watcher operation ("SyncWatcher") in a SuperTiles 102 ... Wait for the SyncFlag to be generated and set the "thread_id" field in the instruction for the "SyncFlag" replication instruction. a) ignore or b) wait for the SyncFlag corresponding to the "thread_id" field in the "SyncFlag" replication instruction (412). The data field of the fence operation "TileFence" is "reset_sync_flag_thread_ids" data. The tile fence operation may include a "wait_idle_thread_ids" data field and a "wait_idle_thread_ids" data field 414. These data fields specify whether to reset or wait for synchronization flags in the corresponding computation threads to which the tile fence operation is connected.

[0082] 5 is a flow diagram illustrating an example process 500 for accelerating neural network computations. Process 500 can be implemented or performed using the above-described system 100. The description of process 500 may refer to the above-described computing resources of system 100. In some implementations, the steps or actions of process 500 are enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described herein.

[0083] Referring now to process 500, an example SuperTile 102 of system 100 receives inputs to and weights for a neural network layer (502). For example, the SuperTile 102 may receive the inputs and weights via communication bus 124. In addition to receiving the inputs and weights, the SuperTile may receive one or more instructions for performing neural network calculations for the neural network layer and generating outputs for the neural network layer. A controller of the SuperTile stores the inputs and weights in the SuperTile's unified memory (504). For example, the controller 104 stores the inputs and weights in unified memory 106 based on the instructions received via communication bus 124.

[0084] The arbitration logic unit of the supertile passes one or more of the inputs stored in the unified memory to each computational tile of the plurality of computational tiles in the supertile (506). The arbitration logic unit 110 is coupled to the unified memory 106 and to each computational tile 108n of the plurality of computational tiles 108. In some implementations, the controller 104 is configured to determine a partition of addresses in the unified memory 106 for storing each batch of inputs to be passed to a corresponding computational tile 108n of the supertile 102. For example, each partition of addresses in the unified memory 106 can be assigned to a respective computational tile 108n of the supertile.

[0085] The arbitration logic unit is configured to, for a first partition of addresses, obtain a first batch of inputs from memory locations identified by addresses in the partition of addresses and pass the first batch of inputs to cells 152 of a first computational tile 108n, where the first computational tile 108n is assigned to receive each input in the first batch of inputs based on the determined partition of addresses in the unified memory. In some examples, the set of addresses in the partition of addresses may be for a batch of inputs that form a sample of input features. The sample of input features may include multiple sets of input features, where the multiple sets of input features correspond to a stream of image or audio data.

[0086] The arbitration logic unit passes the respective sets of weights stored in the unified memory to each of the computational tiles (508). The SuperTile 102 executes multiple computational threads in each of the computational tiles within the SuperTile to perform computations and generate outputs for the neural network layer (510). The SuperTile 102 generates outputs for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and the respective sets of weights (512). In some implementations, the neural network layer is an embedding layer of a convolutional neural network, and the output generated by the neural network layer is an embedding output including an embedded feature vector.

[0087] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus.

[0088] Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0089] The term "computing system" encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. Such apparatus may include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, such apparatus may further include code that creates an execution environment for the computer program, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0090] A computer program (which may also be called a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0091] A computer program may, but need not, correspond to a file in a file system. A program may exist in a single file dedicated to that program, or in multiple coordinated files (e.g., one or more modules of code). It may be stored in a file that contains other programs or data (e.g., one or more scripts stored in a markup language document), in a file that contains a program, subprogram, or portion thereof. A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0092] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be realized by, special purpose logic circuitry such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0093] A processor suitable for executing a computer program may be based, by way of example, on a general purpose or special purpose microprocessor or both, or on any kind of central processing unit. Generally, a central processing unit receives instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer further comprises, for example, a It may also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).

[0094] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0095] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, that allows a user to provide input to the computer. Other types of devices may be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0096] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server; a middleware component, e.g., an application server; a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein; or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), e.g., the Internet.

[0097] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0098] Although this specification contains many specific implementation details, these may not limit the scope of any invention. These and other references should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a certain combination, and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0099] Similarly, while operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, or that all of the shown operations need to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A circuit for a hardware accelerator configured to implement a neural network comprising a plurality of neural network layers and to perform computations to generate outputs for the neural network layers, the circuit comprising: a plurality of SuperTiles, each SuperTile of the plurality of SuperTiles comprising: a unified memory configured to store inputs to the neural network layers and a plurality of weights for the neural network layers; a plurality of computational tiles, each computational tile configured to execute a computational thread used to perform the computation to generate the output, each supertile of the plurality of supertiles further comprising: an arbitration logic unit coupled to the unified memory and to each computational tile of the plurality of computational tiles, the arbitration logic unit comprising: passing one or more of the inputs stored in the unified memory to each of the computation tiles; passing a respective set of weights stored in the unified memory to each of the computational tiles; a circuit configured to pass to the unified memory outputs generated for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and the respective sets of weights.

2. a respective controller for each SuperTile, said respective controller configured to generate one or more control signals, said one or more control signals comprising: used to store each of the inputs to the neural network layer in a corresponding location in the unified memory, each of the corresponding locations being identified by a respective address, and the one or more control signals further comprising: used to store each weight of the plurality of weights for the neural network layer in a corresponding location of the unified memory, each of the corresponding locations being identified by a respective address, and the one or more control signals further comprising: The circuit of claim 1 , wherein the arbitration logic causes one or more inputs to be passed to a computational cell of a particular computational tile and a respective set of weights to be passed to the particular computational tile.

3. The controller The circuit of claim 2 , configured to store the respective set of weights for the particular computational tile in a respective register file of the particular computational tile local to the particular computational tile.

4. The controller 4. The circuit of claim 3, configured to determine a partition of addresses within the unified memory for storing each batch of inputs to be passed to a corresponding computational tile of a SuperTile, each partition of addresses being assigned to a respective computational tile of the SuperTile.

5. each address in the partition of addresses corresponds to an input in the batch of inputs that form the sample of input features; the sample of input features includes a plurality of sets of input features; The circuit of claim 4 , wherein the plurality of sets of input features correspond to a stream of image or audio data.

6. The arbitration logic unit for a first section of addresses, obtaining a first batch of inputs from memory locations identified by addresses within said section of addresses; 6. The circuit of claim 4 or 5, configured to pass the first batch of inputs to cells of a first computational tile, the first computational tile being assigned to receive each input in the first batch of inputs based on the determined partitioning of addresses in the unified memory.

7. For each Super Tile, each computational tile of the plurality of computational tiles is configured to execute two or more computational threads in parallel in the computational tile; 10. The circuit of claim 1, wherein each computational tile executes a computational thread to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate a partial output for the neural network layer.

8. For each Super Tile, each computational tile of the plurality of computational tiles performs a portion of the computation to generate the output for the neural network layer in response to executing two or more computational threads in parallel in the computational tile; 8. The circuit of claim 7, configured to generate, in response to performing the portion of the computation, one or more partial outputs used to generate the output for the neural network layer.

9. The circuit comprises: For each respective computational tile of the plurality of computational tiles in a SuperTile: executing two or more computational threads in parallel in the computational tiles; For each respective supertile of the plurality of supertiles:

9. The circuit of claim 1 or 8, configured to execute two or more computational threads assigned to each computational tile in parallel to generate the output for the neural network layer.

10. a first portion of the operations performed using the computational threads corresponds to a first set of tensor operations for traversing one or more dimensions of a first multidimensional tensor; 10. The circuit of claim 1 or 6, wherein the first multidimensional tensor is an input tensor that includes data elements corresponding to the inputs stored in the unified memory.

11. a second portion of the operations performed using the computational threads corresponds to a second set of tensor operations for traversing one or more dimensions of a second multidimensional tensor that is different from the first multidimensional tensor; The circuit of claim 10 , wherein the second multidimensional tensor is a weight tensor that includes data elements corresponding to the plurality of weights stored in the unified memory.

12. 1. A method for performing computations to generate outputs for neural network layers of a neural network using circuitry for a hardware accelerator configured to implement a neural network having a plurality of neural network layers, the method comprising: receiving, at a supertile of a plurality of supertiles, an input to the neural network layer and a plurality of weights for the neural network layer; storing the inputs to the neural network layer and the weights for the neural network layer in a unified memory of the SuperTile; using an arbitration logic unit of the Supertile to pass one or more of the inputs stored in the unified memory to each of a plurality of computational tiles within the Supertile; the arbitration logic unit is coupled to the unified memory and to each computational tile of the plurality of computational tiles, the method further comprising: using the arbitration logic unit of the SuperTile to pass a respective set of weights stored in the unified memory to each of the computation tiles; executing a computation thread in each of the computation tiles within the SuperTile to perform the computation and generate the output for the neural network layer; generating the outputs for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and the respective sets of weights.

13. passing the outputs generated for the neural network layers to the unified memory using the arbitration logic unit; and passing the output generated for the neural network layer to another SuperTile in the circuit using a controller of each of the SuperTile.

14. generating a control signal by the respective controller of the SuperTile; storing each of the inputs to the neural network layer in a corresponding location in the unified memory based on the control signal, each of the corresponding locations being identified by a respective address, the method further comprising: storing each weight of the plurality of weights for the neural network layer in a corresponding location of the unified memory based on the control signal, each of the corresponding locations being identified by a respective address, the method further comprising:

14. The method of claim 13, comprising causing the arbitration logic to pass one or more inputs to a computational cell of a particular computational tile and pass a respective set of weights to the particular computational tile based on the control signal.

15. 15. The method of claim 14, comprising storing the respective set of weights for the particular computational tile in a respective register file of the particular computational tile local to the particular computational tile based on the control signal.

16. 16. The method of claim 15, further comprising: the controller determining a partition of addresses within the unified memory for storing each batch of inputs to be passed to a corresponding computational tile of a supertile, each partition of addresses being assigned to a respective computational tile of the supertile.

17. each address in the partition of addresses corresponds to an input in the batch of inputs that form the sample of input features; the sample of input features includes a plurality of sets of input features; The method of claim 16 , wherein the multiple sets of input features correspond to a stream of image or audio data.

18. For the first section of the address: said arbitration logic unit obtaining a first batch of inputs from memory locations identified by addresses within said partition of addresses; and passing the first batch of inputs to cells of a first computational tile, the first computational tile being assigned to receive each input in the first batch of inputs based on the determined partitioning of addresses in the unified memory.

19. For each Super Tile, executing two or more computational threads in parallel in each computational tile of the plurality of computational tiles; 20. The method of claim 12 or 18, wherein each computational tile executes a computational thread to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate partial outputs for the neural network layer.

20. For each Super Tile, at each computational tile of the plurality of computational tiles, performing a portion of the computation to generate the output for the neural network layer in response to executing the two or more computational threads in parallel at the computational tile; and generating, in response to performing the portion of the computation, one or more partial outputs used to generate the output for the neural network layer.

21. For each respective computational tile of the plurality of computational tiles in a SuperTile: executing two or more computational threads in parallel in the computational tiles; For each respective supertile of the plurality of supertiles: and executing two or more computational threads assigned to each computational tile in parallel to generate the output for the neural network layer.

22. a first portion of the operations performed using the computational threads corresponds to a first set of tensor operations for traversing one or more dimensions of a first multidimensional tensor; 22. The method of claim 12 or 21, wherein the first multidimensional tensor is an input tensor that includes data elements corresponding to the inputs stored in the unified memory.

23. a second portion of the operations performed using the computational threads corresponds to a second set of tensor operations for traversing one or more dimensions of a second multidimensional tensor that is different from the first multidimensional tensor; 23. The method of claim 22, wherein the second multidimensional tensor is a weight tensor that includes data elements corresponding to the plurality of weights stored in the unified memory.

24. A system on a chip, a circuit for a hardware accelerator configured to implement a neural network comprising a plurality of neural network layers and to perform computations to generate outputs for the neural network layers; a host controller configured to access a memory external to the circuitry for the hardware accelerator, the memory configured to store data for processing by the neural network layer, the system-on-chip further comprising: a host interface configured to exchange data communications between the circuitry for the hardware accelerator and the host controller; a plurality of SuperTiles arranged within the circuit, each SuperTile of the plurality of SuperTiles comprising: a unified memory configured to store inputs to the neural network layers and a plurality of weights for the neural network layers, the inputs and the plurality of weights being stored in the memory accessible by the host controller; and each Supertile of the plurality of Supertiles further comprises: a plurality of computational tiles, each computational tile configured to execute a computational thread used to perform the computation to generate the output, each supertile of the plurality of supertiles further comprising: an arbitration logic unit coupled to the unified memory and to each computational tile of the plurality of computational tiles, the arbitration logic unit comprising: passing one or more of the inputs stored in the unified memory to each of the computation tiles; passing a respective set of weights stored in the unified memory to each of the computational tiles; and passing to the unified memory outputs generated for the neural network layers based on computations performed in each of the computation tiles using one or more of the inputs and the respective sets of weights.

Images