Hardware circuitry for accelerating neural network calculation
Patent Information
- Application Number
- JP2024114737
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-12-19
AI Technical Summary
Existing hardware accelerators for neural network computations are limited in their ability to efficiently parallelize operations across computational tiles, leading to suboptimal processing times and resource utilization.
The implementation of a hardware accelerator architecture that includes multiple supertiles, each with unified memory and computational tiles, utilizing arbitration logic to efficiently distribute inputs and weights across these tiles for parallel computation, allowing for simultaneous execution of multiple threads.
This architecture significantly reduces processing time for neural network computations by optimizing parallelization across tiles, achieving higher utilization rates of computational resources, particularly in distributed systems.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] background This specification relates generally to circuits for hardware accelerators used to perform neural network computations. [Background technology]
[0002] A neural network is a machine learning model that uses one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to one or more other layers in the network, e.g., other hidden layers or the output layer of the network. Some of the layers of the network generate outputs from the received input according to the current values of a respective set of parameters. Some neural networks are convolutional neural networks (CNNs) (e.g., used in image processing) or recurrent neural networks (RNNs) (e.g., used in speech and language processing).
[0003] CNNs and RNNs are neural networks that include a respective set of convolutional or recurrent neural network layers. A neural network layer may have an associated set of kernels, which may correspond to parameters or weights that are used to process inputs through the layer to generate corresponding outputs of the layer in order to compute neural network inferences. The kernels may be represented as tensors, i.e., multidimensional arrays, of weights. As an example, a neural network layer in a sequence of layers may process a set of inputs, such as image pixel data or activation values generated by another neural network layer in the sequence of layers. The set of inputs or the set of activation values may also be represented as a tensor. Summary of the Invention [Problem to be solved by the invention]
[0004] overview This document describes an improved hardware circuit that may be used in a hardware accelerator configured to accelerate the computation of an exemplary neural network model, such as the computation of a layer of an artificial neural network. The circuit architecture includes a plurality of supertiles, each supertile configured to execute a plurality of computational threads based on data retrieved from a unified memory of the supertile. The unified memory provides a memory configuration that may be efficiently shared among each of the computational threads such that the computations of each of the computational threads may be performed simultaneously in the supertile.
[0005] In some implementations, the described hardware circuits and processing techniques may be used in an exemplary computing system, such as a small or large distributed system, that includes circuits for multiple dedicated processors (e.g., hardware accelerators) used to perform inference (or training) calculations for an exemplary machine learning workload. The circuit architectures described herein may be integrated into each of the multiple dedicated processors to increase the speed and efficiency with which the multiple dedicated processors perform calculations to perform tasks for various types of machine learning models. [Means for solving the problem]
[0006] One aspect of the subject matter described herein may be implemented in a circuit for a hardware accelerator configured to implement a neural network including multiple neural network layers and to perform computations to generate outputs for the neural network layers. The circuit comprises a plurality of supertiles, each supertile of the plurality of supertiles including a unified memory configured to store inputs to the neural network layers and multiple weights for the neural network layers, and a plurality of computation tiles, each computation tile configured to execute computation threads used to perform the computations to generate outputs, each supertile of the plurality of supertiles further including an arbitration logic unit coupled to the unified memory and each of the plurality of computation tiles. The arbitration logic unit is configured to pass one or more of the inputs stored in the unified memory to each of the computation tiles, pass a respective set of weights stored in the unified memory to each of the computation tiles, and pass to the unified memory an output generated for the neural network layer based on a computation performed in each of the computation tiles using one or more of the inputs and the respective set of weights.
[0007] These and other implementations may each optionally include one or more of the following features: For example, in some implementations, the circuitry includes a respective controller for each supertile, the respective controller configured to generate one or more control signals, the one or more control signals used to store each of the inputs to the neural network layer in a corresponding location of the unified memory, each of the corresponding locations being identified by a respective address, the one or more control signals further used to store each weight of the plurality of weights for the neural network layer in a corresponding location of the unified memory, each of the corresponding locations being identified by a respective address, and the one or more control signals further cause the arbitration logic to pass the one or more inputs to a computational cell of a particular computational tile and pass each set of weights to a particular computational tile.
[0008] In some implementations, the controller is configured to store each set of weights for a particular computational tile in a respective register file of the particular computational tile that is local to the particular computational tile. In some implementations, the controller is configured to determine partitions of addresses within the unified memory for storing each batch of inputs to be passed to a corresponding computational tile of the supertile, each partition of addresses being assigned to a respective computational tile of the supertile.
[0009] In some implementations, each address in the partition of addresses corresponds to an input in a batch of inputs forming a sample of input features, the sample of input features including a plurality of sets of input features, the plurality of sets of input features corresponding to a stream of image or audio data. In some implementations, the arbitration logic unit is configured to, for a first partition of addresses, obtain the first batch of inputs from memory locations identified by addresses in the partition of addresses and pass the first batch of inputs to cells of a first computational tile, the first computational tile being assigned to receive each input in the first batch of inputs based on the determined partition of addresses in the unified memory.
[0010] In some implementations, for each respective supertile, multiple computation tiles are Each computational tile is configured to execute two or more computational threads in parallel in the computational tile, and each computational tile executes the computational threads to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate a partial output for the neural network layer.
[0011] In some implementations, for each respective supertile, each computational tile of the plurality of computational tiles performs a portion of the computation to generate an output for the neural network layer in response to executing two or more computational threads in parallel in the computational tile. In response to executing the unit, the neural network layer is configured to generate one or more partial outputs that are used to generate outputs for the neural network layers.
[0012] In some implementations, the circuitry is configured to, for each respective computational tile of a plurality of computational tiles in the supertile, execute two or more computational threads in parallel in the computational tile, and for each respective supertile of the plurality of supertiles, the circuitry is configured to execute two or more computational threads assigned to each computational tile in parallel to generate output for the neural network layer. In some implementations, a first portion of the operations performed with the computational threads corresponds to a first set of tensor operations for traversing one or more dimensions of a first multidimensional tensor, the first multidimensional tensor being an input tensor that includes data elements corresponding to inputs stored in the unified memory.
[0013] In some implementations, a second portion of the operations performed using the computational threads corresponds to a second set of tensor operations for traversing one or more dimensions of a second multidimensional tensor that is different from the first multidimensional tensor, the second multidimensional tensor being a weight tensor that includes data elements corresponding to a plurality of weights stored in the unified memory.
[0014] One aspect of the subject matter described herein may be embodied in a method for performing computations to generate outputs for neural network layers of a neural network using circuitry for a hardware accelerator configured to implement a neural network including multiple neural network layers. The method includes receiving, at a supertile of multiple supertiles, an input to the neural network layer and multiple weights for the neural network layer, and storing, in a unified memory of the supertile, the input to the neural network layer and the multiple weights for the neural network layer. The method also includes passing, using an arbitration logic unit of the supertile, one or more of the inputs stored in the unified memory to each computational tile of multiple computational tiles in the supertile, the arbitration logic unit being coupled to the unified memory and each computational tile of the multiple computational tiles, the method further including passing, using the arbitration logic unit of the supertile, a respective set of weights stored in the unified memory to each of the computational tiles. The method includes executing computational threads in each of the computational tiles in the supertile to perform computations to generate outputs for the neural network layer, and generating outputs for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and a respective set of weights.
[0015] Each of these and other implementations may optionally include one or more of the following features. For example, in some implementations, the method includes passing the outputs generated for the neural network layers to a unified memory using an arbitration logic unit, and passing the outputs generated for the neural network layers to another supertile in the circuit using a controller of each of the supertiles.
[0016] In some implementations, the method includes generating a control signal by a controller of each of the supertiles, and storing each of the inputs to the neural network layer in a corresponding location of the unified memory based on the control signal, each of the corresponding locations being identified by a respective address, the method further includes storing each weight of the plurality of weights for the neural network layer in a corresponding location of the unified memory based on the control signal, each of the corresponding locations being identified by a respective address, and the method further includes causing the arbitration logic to pass one or more inputs to a computational cell of a particular computational tile and pass each set of weights to a particular computational tile based on the control signal.
[0017] In some implementations, the method includes performing multiple calculations for each respective supertile. and executing each computational thread of two or more computational threads in parallel in each computational tile of the tile, each computational tile executing the computational thread to perform multiplications between one or more inputs to the neural network layer and weights for the neural network layer to generate a partial output for the neural network layer.
[0018] One aspect of the subject matter described herein may be embodied in a system on chip (SoC) comprising: a circuit for a hardware accelerator configured to implement a neural network comprising a plurality of neural network layers and perform calculations to generate outputs for the neural network layers; and a host controller configured to access a memory external to the circuit for the hardware accelerator, the memory configured to store data for processing by the neural network layers; and the system on chip further comprising a host interface configured to exchange data communications between the circuit for the hardware accelerator and the host controller.
[0019] The SoC includes a plurality of supertiles arranged in a circuit. Each supertile of the plurality of supertiles includes a unified memory configured to store inputs to a neural network layer and a plurality of weights for the neural network layer. The inputs and the plurality of weights correspond to data stored in a memory accessible by a host controller. Each supertile includes a plurality of computational tiles, each computational tile configured to execute computational threads used to perform computations and generate outputs. Each supertile of the plurality of supertiles includes an arbitration logic unit coupled to the unified memory and to each computational tile of the plurality of computational tiles.
[0020] The arbitration logic unit is configured to pass one or more of the inputs stored in the unified memory to each of the computational tiles, pass a respective set of weights stored in the unified memory to each of the computational tiles, and pass to the unified memory outputs generated for the neural network layer based on computations performed in each of the computational tiles using the one or more of the inputs and the respective sets of weights.
[0021] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods, encoded on a computer storage device. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof that is installed on the system and that, when operated, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions. Effect of the Invention
[0022] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages: The circuit architectures and data processing techniques described herein can be integrated into an exemplary distributed system to reduce the processing time required to process a set of inputs through layers of a neural network, such as a convolutional or recurrent neural network.
[0023] The circuit architecture and data processing techniques provide a different combination of approaches for optimizing how computations are parallelized across tiles, as compared to prior art circuit designs for performing neural network computations. For example, the described techniques make it possible to optimize how computations are parallelized across tiles for use cases that involve significant reuse of data between computations, such as when activations are reused across different dimensions of a filter and parameters are reused in batches across multiple activations. do.
[0024] The present technology can be used to implement a circuit architecture and software stack that provides one or more supertiles that allow for multiple simultaneous computational threads within the supertile. This architecture enables processing techniques that include determining whether to broadcast and / or slice both parameters (weights) and activations. This determination can be different for different types of workloads to optimize the performance of an exemplary hardware accelerator incorporating this architecture.
[0025] The optimization can be linked to the utilization of the sum-of-multiplies cells in the computational units of the hardware circuit. The utilization can be evaluated with reference to different approaches for partitioning the dimensions of a tensor across supertiles based on the improved circuit architecture, for example, partitioning across the Z dimension of the tensor across four supertiles, or partitioning the X, Y dimensions of the tensor across two supertiles. For example, with the techniques described herein, multiple approaches can be used to parallelize the computation across tiles, such that the sum-of-multiplies cells of the circuit can achieve a threshold utilization (e.g., 70%) higher than the utilization of the cells in prior art circuit designs.
[0026] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0027] [Figure 1] FIG. 1 is a block diagram of a computing system including an example circuit for a hardware accelerator. [Diagram 2] FIG. 1 is a block diagram of an exemplary computational tile architecture of a circuit for a hardware accelerator. [Diagram 3] 1 shows an example tensor and example program code for processing data corresponding to elements of the tensor. [Figure 4] 1 shows a table containing example instructions of an instruction set architecture for one or more supertiles. [Diagram 5] FIG. 1 is a flow diagram illustrating an exemplary process for accelerating neural network computations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0028] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description This specification describes improved hardware circuits and data processing techniques that may be implemented using the improved hardware circuit architectures. The hardware circuits may be special-purpose processors, such as neural network processors, application specific integrated circuits, or hardware accelerators.
[0029] The hardware circuit includes multiple supertiles. Each supertile includes a unified memory for storing inputs to and weights of the neural network layers. Each supertile is configured to execute multiple computational threads based on data retrieved from the supertile's unified memory and instructions received at the supertile via a communication bus coupled to each of the supertiles. In some implementations, each supertile includes multiple computational tiles, each computational tile configured to execute one or more computational threads. In some cases, each computational tile is configured to execute one computational thread such that the supertile can execute multiple computational threads in parallel. In other cases, each computational tile can be configured to execute multiple computational threads, such that a SuperTile executes each of the multiple computational threads in parallel. The computational threads are used to perform calculations and generate outputs for the neural network layers.
[0030] Each supertile includes an arbitration logic unit coupled to the unified memory and to each computational tile or each computational thread that may be executed in that supertile. The arbitration logic unit is configured to pass inputs and weights stored in the unified memory to the computational tiles. The arbitration logic unit is also configured to pass outputs generated for a layer to the unified memory of a supertile assigned to receive the output, or to each of one or more supertiles assigned to receive a portion of the output.
[0031] In some implementations, outputs for neural network layers are generated in supertiles based on computations performed in the computation tiles of the supertiles with inputs and weights for the layer passed to the computation tiles by the arbitration logic unit. In other implementations, one or more layers of a neural network may be split across multiple supertiles, e.g., a layer may be parallelized across multiple supertiles such that each supertile performs a portion of the processing for the layer. In these implementations, outputs for neural network layers are generated across multiple supertiles as respective sets of output values (e.g., vectors of activation values) that together form the output for the neural network layer.
[0032] 1 is a block diagram of a computing system 100 including an example circuit for a hardware accelerator. In some cases, the system 100 is an example computing system for accelerating tensor or neural network computations associated with an artificial deep neural network (DNN), such as an RNN or a CNN. For example, the system 100 is configured to implement an example artificial neural network (e.g., a CNN) on a hardware circuit 101, such as a dedicated hardware circuit. In some implementations, the system 100 is a system-on-chip. For example, the system-on-chip may include the hardware circuit 101 and some (or all) of the other components and devices described herein as being included in the system 100.
[0033] The hardware circuit 101 may be a hardware accelerator configured to accelerate the execution and / or performance of a neural network model. For example, the execution of the neural network model may be accelerated relative to the execution of the model on an exemplary general-purpose machine, such as a central processing unit (CPU). Similarly, the performance and execution of the neural network model may be accelerated compared to when the model is implemented on another hardware accelerator (e.g., a graphics processing unit (GPU)) that does not have the improved hardware features and software functions associated with the techniques described herein.
[0034] A system 100 including the exemplary circuit 101 can include one or more supertiles 102. In some implementations, the system 100 includes multiple supertiles 102. In the example of FIG. 1 (and FIG. 2 described below), the system 100 is shown as including four supertiles 102, but the system 100, and the hardware circuit 101 described herein, may include more or fewer supertiles. As described in more detail below, a supertile 102 is a discrete, self-contained computing unit of the system 100 (or hardware circuit 101). In some implementations, each supertile 102 is a multi-layer neural network (MLN). It is configured to independently perform the computations required by one or more layers of the work (e.g., neural network computations).
[0035] Computations may be required to process data for a machine learning workload or to perform a particular task of that workload. In some implementations, the computation process performed within Supertile 102 for one or more neural network layers may include multiplication of data values stored in respective elements of an input tensor (e.g., inputs or activations) with data values stored in respective elements of a parameter tensor (e.g., weights). For example, the computation may include multiplying the input or activation values by weight values over one or more cycles and performing an accumulation of products over many cycles.
[0036] Each supertile 102 generally includes a respective controller 104, a respective unified memory 106, a respective number of computational tiles (or threads) 108, and a respective arbitration logic unit 110 ("arbitration logic 110").
[0037] The controller 104 is configured to generate control signals 114 for controlling operations occurring within the SuperTile 102. For example, the control signals 114 may be used to: a) store each of the received inputs to a neural network layer in a corresponding location in the unified memory 106, and b) store each of the received weights for a neural network layer in a corresponding location in the unified memory 106. Each of the corresponding memory locations that store a respective input or weight is identified by a respective address.
[0038] The controller 104 includes a direct memory access (DMA) module 105 that includes a DMA operation ("DMAOp") control 105a and a DMAOp tensor traversal unit (TTU) 105b. The DMAOp control 105a represents control logic that may be used by the controller 104 to i) manage writing / storing data for a computation to memory locations in the unified memory 106, and ii) manage reading / retrieving data for a computation from memory locations in the unified memory 106. For example, the DMAOp control 105a is executed by the controller 104 to manage writing inputs of input tensors received at the supertiles 102 to memory locations in the unified memory 106, and weights of weight tensors received at the supertiles 102 to memory locations in the unified memory 106.
[0039] DMAOp control 105a is operable to manage traversal operations for execution by DMAOp TTU 105b. In some implementations, the locations or addresses of unified memory 106 to which particular inputs or activations are written or read are generated by DMAOp TTU 105b based on inbound / outbound DMAOp instructions received over communication bus 124 (described below). For example, DMAOp instructions may be processed by DMAOp control 105a to manage the traversal operations performed by DMAOp TTU 105b to generate the locations or addresses of unified memory 106 used to store the inputs and weights received over communication bus 124.
[0040] In some cases, inbound and outbound DMAOps may be executed simultaneously. An exemplary outbound DMAOp may include a supertile 102 providing activation values of generated layer outputs to adjacent supertiles 102 of the system 100. During the simultaneous execution of inbound and outbound DMAOps, any necessary synchronization or arbitration of memory location accesses may be managed through a synchronization flag control scheme managed by the controller 104. In some implementations, the controller 104 may be configured to synchronize the DMAOps with the DMAOps. The controller 104 is operable to manage the synchronization flag control scheme in conjunction with the arbitration logic 11 .
[0041] Control signals 114 generated by the controller 104 may also be used to a) cause the read arbitration logic 110a to pass one or more inputs obtained from the unified memory 106 to the computational cells 152 (described below) of a particular computational tile 108n, and b) cause the read arbitration logic 110a to pass each set of weights obtained from the unified memory 106 to a particular computational tile 108n. In some implementations, the arbitration logic 110 passes the inputs and weights to the computational tile 108n via an input bus 112.
[0042] 1, the arbitration logic 110 may be coupled to each computational tile 108n of the supertile 102 via a respective input bus 112 and a respective output bus 113. The arbitration logic 110 is configured to retrieve (or read) multiple batches of inputs from memory locations of the unified memory 106. The arbitration logic 110 is also configured to store (or write) multiple sets of outputs or output activations provided by each computational tile 108 to memory locations of the unified memory 106.
[0043] In some examples, the unified memory 106 may be described as a narrow memory structure operable to store inputs, activations, or gain values to be processed in a neural network layer and to output activations generated by the neural network layer in response to processing the inputs or activations through the neural network layer. The generation and storage of output activations will be described in more detail. The unified memory 106 of each supertile 102 may employ a memory hierarchy that provides addressing arbitration and flexibility that allows traversing multi-dimensional arrays in any order, while also avoiding bank contention for certain memory operations, such as single-cycle read and write operations. In some implementations, the unified memory 106 includes multiple memory banks (e.g., multiple independently arbitrated memory banks), and the arbitration logic 110 is configured to arbitrate read and write accesses to each memory location of each memory bank in the unified memory 106.
[0044] Each batch of inputs passed by the arbitration logic 110 may correspond to a particular computational tile 108n, and the batch of inputs is provided to the particular computational tile 108n via a respective input bus 112 that couples the particular computational tile 108n to the arbitration logic 110. For example, the arbitration logic 110 is configured to load each input of a first batch of inputs onto a first input bus 112 that couples the arbitration logic 110 to a first computational tile 108n in a supertile. The arbitration logic 110 is also configured to load each input of a second, different batch of inputs onto a second, different input bus 112 that couples the arbitration logic 110 to a second, different computational tile 108n in the supertile 102. Alternatively, in some cases, each of the multiple batches of inputs may correspond to and be loaded at the same computational tile 108n.
[0045] The arbitration logic 110 is a logical unit or structure of the unified memory 106. For example, the arbitration logic 110 may be a dedicated memory arbiter used in a shared memory system (e.g., the unified memory 106) to determine which control unit (e.g., the DMAOp control 105a or the TensorOp control 132) is allowed to access the shared memory resources of the unified memory 106 for each memory cycle. For example, in a supertile 102, the different instruction types of the DMAOp control 105a and the TensorOp control 132 may be configured as independent control threads requesting memory access, and the requests need to be arbitrated by the arbitration logic 110.
[0046] As described herein, each SuperTile 102 executes k computational threads. In some implementations, each of the k computational threads is a software construct that runs in a respective supertile 102, and portions of the k computational threads may be managed or executed by respective computational tiles 108n of the supertile 102. A supertile 102 may be a superscalar tile or a supervector tile that represents an independent computational unit in which multiple TensorOp pipelines (or threads) are executed in parallel, i.e., simultaneously. For example, a parameter or variable kNumberComputeThreads (k computational threads) may be a supertile 102 that represents the number of TensorOp pipelines (or threads) that are executed in parallel, i.e., simultaneously. It may represent the number of parallel TensorOp pipelines in a perscalar tile 102 or a supervector tile 102. A superscalar tile 102 may be an exemplary supertile 102 that operates on scalar input values, and a supervector tile 102 may be an exemplary supertile 102 that operates on vectors of input values.
[0047] In a supertile 102, each computational thread may correspond to a single computational tile, and the computational tile executes a single computational thread. Alternatively, each computational tile may be configured to execute multiple computational threads. In some implementations, a set of computational tiles 108n may be physically or logically arranged within each supertile 102 of the system 100. For example, in the system 100 (or hardware circuit 101), the set of computational tiles 108n for each supertile 102 may be configured in hardware or software. In some implementations, when the computational tiles 108n for each supertile 102 are configured in software, the supertile 102 may be configured to execute n computational tiles 108, where n is an integer equal to or greater than 1. In these implementations, each of the n computational tiles 108n may be configured to execute k computational threads.
[0048] Control signals 114 generated by the controller 104 can also be used to a) cause the write arbitration logic 110b to pass the generated layer output activation to the unified memory 106 for storage in the unified memory 106, and b) cause the supertile 102 to provide the generated layer output activation value to an adjacent supertile.
[0049] The system 100 includes an external host / controller 120 coupled to each of the supertiles 102 via a host interface 122. In some implementations, the host interface 122 is coupled between the host controller 120 and a circuit for a hardware accelerator (e.g., hardware circuit 101) that may be included in a system-on-chip. The host interface 122 is configured to exchange data communications between the host controller 120 and the circuit for the hardware accelerator. In some implementations, the host controller 120 is configured to access a memory that is external to the circuit for the hardware accelerator (e.g., an external memory). The external memory is configured to store data for processing in a neural network implemented in the circuit. For example, the data may be inputs and weights to be processed by one or more layers of the neural network.
[0050] The host interface 122 receives instructions and data values from an external host / controller 120 and provides a respective set of instructions and data values to each of the supertiles 102. In some examples, the data values may be retrieved from an external memory accessible by the host controller 120 and then passed to the supertiles 102 via the host interface 122. The host interface 122 is operable to use an exemplary communication bus accessible by each of the supertiles 102 to pass instructions and data values to the supertiles. In some implementations, the instruction set architecture of the system 100 may be such that each of the supertiles 102 receives its own single instruction. A single instruction may include data values (e.g., inputs and weights), specific data fields, and operational parameters for a workload or a set of tasks within a workload.
[0051] Generally, instructions and data values are provided to one or more devices in the system 100 via a communication bus 124 (e.g., an instruction bus or a ring bus). In some cases, a supertile 102 receives data and instructions for a machine learning task via an exemplary communication bus 124 that couples two or more supertiles in the system 100. For example, the communication bus 124 is configured to provide communications coupling the supertiles 102 of the system 100 via a bus data path that connects the supertiles 102 of the system 100 to the host controller 120 via the host interface 122 in an exemplary ring format. The ring format is illustrated in the example of FIG.
[0052] In some implementations, one or more instructions are received by each of the respective controllers 104 in the supertile 102 from the host interface 122 at an initial time and stored in an exemplary instruction memory of each controller 104 for later execution by the controller 104. The data can include inputs, activations, gain values, or a combination of each. In some examples, the data is received at the supertile 102 and processed at the neural network layer to generate outputs for the neural network layer. In such examples, processing the data at the neural network layer to generate layer outputs includes generating multiple partial outputs (e.g., accumulated values or pre-activation values).
[0053] Each of the computational tiles 108n includes a respective tensor module 130 that includes a tensor operation ("TensorOp") control 132 and a TensorOp TTU 134. Each of the respective tensor modules 130 may provide functionality similar to or related to that provided by the DMAOp module 105 of the controller 104. For example, the TensorOp control 132 may represent control logic used by the controller 104 or computational tile 108n to: i) manage operations for reading / accessing input values assigned to particular elements of an input tensor from corresponding memory locations in the unified memory 106 that stores the inputs; and ii) manage the association or assignment of output values (or partial outputs) to particular elements of an output tensor after the output values are generated in response to one or more computational threads executing in the computational tile 108n.
[0054] TensorOp control 130 may be executed by the controller 104 or a computation thread of a computation tile 108n to manage traversal operations for execution by TensorOp TTU 134. For example, TensorOp TTU 134 is operable to execute instructions to access a set of elements along a particular dimension of an N-dimensional or multidimensional tensor (e.g., a 2D input tensor, a 3D weight tensor, or a 4D output tensor). An exemplary N-dimensional tensor may have multiple elements arranged across each of the N dimensions, where N is an integer greater than or equal to 1.
[0055] The TensorOp TTU 134 determines the address of each element in a set of elements along a particular dimension of a tensor (e.g., a 2D weight tensor) so that a computational tile 108n (or computational thread) may access a corresponding memory or register file that stores data for that tensor to read data representing the value of the element along that particular dimension. In some implementations, the program code associated with the TensorOp TTU 134 may include one or more nested loops, and the TensorOp TTU 134 is instructed to access an element of a two-dimensional array / tensor variable in the nested loop according to a current index variable value of the nested loop. Based on the current index variable value of the nested loop, the TensorOp TTU 134 may determine an offset value that represents an offset from a first element of a two-dimensional array variable. For example, the address of a particular element may be an address offset from another element of an N-dimensional tensor.
[0056] Each of the computational tiles 108n includes a wide memory structure 140 that includes multiple local register files 142. In some implementations, the controller 104 is configured to store the set of weights for a particular computational tile 108n in a respective register file of the particular computational tile 108n, where the particular register file 142 is local to the particular computational tile 108n. For example, the controller 104 is configured to store individual weights of the set of layer weights in particular memory locations of the local register file 142 in response to passing the set of layer weights from the unified memory 106 to the particular computational tile 108n.
[0057] Each of the computational tiles 108n includes a respective computational unit 150 configured to perform arithmetic operations, such as addition and multiplication, using operands corresponding to inputs and weight values passed to the computational tile 108n. Each of the computational units 150 may include a number of arithmetic cells 152. Each arithmetic cell 152 may be a multiply-accumulate cell configured to perform an arithmetic operation (e.g., multiplication) using inputs and weights. For example, the arithmetic operations performed by the computational units 150 generally include multiplying inputs or activations obtained from the unified memory 106 by parameters to generate a set of accumulation values. Parameters for the computations may be obtained from the wide memory structure 140 of the computational tile 108n, which includes a number of local register files 142.
[0058] Each of the computational tiles 108n includes a register array 160 and a nonlinear unit 170 ("NLU 170"). The register array 160 includes a plurality of individual shift registers 162. Each shift register 162 may be a pipelined shift register 162. The pipelined shift registers 162 of the array 160 are used to shift the output values (e.g., accumulation values or partial sums) of the layer to a nonlinear unit 170 ("NLU 170"). The NLU 170 applies a nonlinear activation function to the output values to generate a set of output activations for the layer. The NLU 170 interacts with the write arbitration logic 110b to pass the generated output activations of the layer outputs to the unified memory 106 for storage in the unified memory 106. For example, the output activations may be provided from the NLU 170 to the write arbitration logic 110b via an output activation bus 113.
[0059] In some implementations, the NLU 170 is operable to aggregate multiple partial sums or accumulated values into a final linear output (e.g., a vector of values) based on control signals provided to the NLU 170 from the computational tiles 108n or by the controller 104.
[0060] Figure 2 is a block diagram illustrating an exemplary computational tile architecture of a circuit for a hardware accelerator. The block diagram in the example of Figure 2 includes a first tile architecture 200 and a second, different tile architecture 210. The first tile architecture 200 represents an exemplary prior art circuit design tile architecture of a dedicated hardware circuit, and the second tile architecture 210 represents a new tile architecture of an improved hardware circuit based on the techniques described in this document.
[0061] The new tile architecture 210 includes multiple supertiles 102. For context, some prior art approaches that perform neural network computations using individual computation tiles 202 and computation threads 204 were limited in how the computations could be parallelized across the architecture. In contrast to these prior art approaches, the new tile architecture 210 includes multiple supertiles 102 and performs neural network computations using individual computation tiles 202 and computation threads 204. The neural network architecture allows for parallelization options within computational tiles 108n of a supertile 102 and across multiple supertiles 102. For example, each supertile 102 may be configured to execute multiple computational threads 214, each of which may execute simultaneously in a supertile 102. In some cases, the simultaneous execution of multiple threads reduces or mitigates processing latency as compared to prior art approaches that may require serial execution of two or more computational threads when processing inputs in a layer of a neural network.
[0062] Each of the multiple computational threads 214 executing in a supertile 102 may be based on data retrieved from the unified memory 106 of the supertile 102, instructions received at the supertile 102, control signals 114 generated by the controller 104, or a combination of each. In some implementations, the multiple computational threads executing in each supertile correspond to one or more tensor operations. In the example of FIG. 2, each supertile 102 is shown as performing four separate tensor operations, but each supertile 102 can be configured to perform more or fewer tensor operations.
[0063] In some implementations, for an exemplary computation associated with a neural network layer that uses a 2D input tensor having X, Y dimensions, the external / host controller 120 is operable to execute an input partitioning algorithm to distribute the outputs X, Y across a grid of supertiles 102 (e.g., the novel tile architecture 210). The external / host controller 120 is operable to allocate space in each of the respective unified memories 106 for each supertile 102 to store input activations, halo pixels, and output activations. In the context of an image processing workload, the halo pixels correspond to inputs that are shared between two or more computational tiles 108n. For example, a set of inputs corresponding to the halo pixels may be used in a convolution in which inputs for edges of an image are shared.
[0064] In the example of FIG. 2, the first partitioning algorithm 220 includes a loop nest that can be used to express the net (total) work done by the supertile 102. The partitioning algorithm 220 and the loop nest can be represented by a portion of program code executed by each TensorOp TTU 134 of different computation tiles 108n in the supertile 102. For example, a variation of the partitioning algorithm 220 may be executed by each TensorOp TTU 134 across multiple computation tiles 108n to traverse specific elements along different dimensions of an exemplary 3D input tensor (x, y, zin) to convolve the 3D input tensor (x, y, zin) with a 2D weight (filter) tensor (kx, ky) to generate a 1D output tensor (zout). This is described in more detail below with reference to FIG. 3.
[0065] 3 illustrates an example tensor 300 (e.g., a 3D input tensor) and a second partitioning algorithm 310 for processing data corresponding to elements of the tensor 300. Based on the novel tile architecture 210 described above, the improved hardware circuitry 101 described herein provides multiple approaches and algorithms by which work such as tasks and computations may be divided among kNumberComputeThreads for TensorOp threads 304 and 306 executing in a supertile 102 or across different supertiles 102. The present invention provides a method and
[0066] For example, a different combination of approaches for dividing work among each of the computational tiles 108n may include: a) assigning a first set of elements for the X,Y dimensions of the tensor 300 to a first computational tile 108n of a first supertile 102; and b) assigning a second set of elements for the X,Y dimensions of the tensor 300 or for the other dimension of the tensor 300 to a first computational tile 108n of a first supertile 102. The method may include assigning the computed number to a second, different computational tile 108 n of the first SuperTile 102 .
[0067] Different combinations of approaches can also be used to divide work among each of the multiple supertiles 102 and among a respective number of computational tiles 108n in each supertile 102. For example, one combination of approaches may include i) assigning different sets of elements in the X, Y dimensions of the tensor 300 to at least two computational tiles 108n of a first supertile 102, and ii) assigning different sets of elements in the X, Y dimensions of the tensor 300 to one or more computational tiles 108n of a second, different supertile 102.
[0068] If the elements of the X, Y dimensions allocated to a supertile 102 are large (e.g., exceeding a threshold size of the SRAM in a computational tile 108n), multiple computational threads can work on additional 2D subpartitions of the allocated X, Y dimensions. In some implementations, for image processing workloads, the data of the 2D subpartitions can be processed without requiring explicit exchange of halo pixels 302 across one or more computational threads in a supertile 102. In some implementations, input pixels required by one or more computational threads in a supertile 102 first reside in the unified memory 106 of the supertile 102 and are then passed to the corresponding computational threads.
[0069] As explained above, the circuit architectures and data processing techniques described herein provide different approaches (or combinations of approaches) for optimizing how computations are parallelized across tiles, as compared to prior art circuit designs for performing neural network computations. In some cases, the optimization can be tied to the utilization of multiply-accumulate cells 152 in the computation unit 150 for different options for partitioning two or more tensor dimensions across supertiles 102 of the improved circuit architecture. As examples, some common options include partitioning the Zin dimension of an input tensor across four supertiles, or partitioning the X, Y dimensions of a 2D tensor across two supertiles.
[0070] For example, multiple approaches may be used to parallelize computations across tiles such that sum-of-products cells 152 of computational units 150 can achieve a higher threshold utilization (e.g., 70%) than the utilization of associated cells in prior art circuit designs. In some cases, the higher threshold utilization for each of the multiple different approaches may be higher than the utilization of the prior art design, even if the prior art design has limited options for how computations may be parallelized across its circuit architecture.
[0071] The approach provided by one or more supertiles 102 allows for a portion (e.g., some or all) of the input tensors assigned to a supertile 102 to be further divided among and operated on by different computational tiles 108n within the supertile 102, or a portion (e.g., some or all) of the parameter tensors assigned to a supertile 102 to be further divided among and operated on by different computational tiles 108n within the supertile 102, or both. Similarly, this approach allows for the processing at a neural network layer to be divided across two or more supertiles 102, e.g., the neural network layer may be parallelized across multiple supertiles 102, with each supertile performing a portion of the processing for that neural network layer. For example, the entire neural network layer may be partitioned across all (or some) of the supertiles 102. In general, multiple options for parallelization are provided to improve upon this approach. This can be achieved using circuit architecture.
[0072] Thus, different approaches may be used to allocate work and partition elements and dimensions of tensor 300 to traverse particular elements along different dimensions of N-dimensional tensor 300 using different combinations of supertiles 102 and computational threads for each supertile 102 to convolve tensor 300 with an N-dimensional weight (filter) tensor (or perform other operations) to generate an N-dimensional output tensor. Thus, one or more N-dimensional tensors accessible from unified memory 106 and wide memory structure 140 in a single supertile 102 may be traversed based on memory address values processed by respective TensorOp TTUs 134 within the supertile 102.
[0073] The system 100 is configured to determine a partition of addresses between each computational thread of a plurality of computational threads for a given supertile 102. The address partition may be determined based on a particular approach for allocating work and partitioning elements and dimensions of tensors to be processed in the system 100. In some implementations, the DMAOp control 105a is operable to determine a mapping of addresses in the partitions for each input in a batch of inputs to be processed through a neural network layer. For example, each batch of inputs may be associated with a different element of the input tensor 300, and each partition of addresses may be assigned to a particular computational tile 108n or computational thread executing in the computational tile 108n.
[0074] FIG. 4 shows a table 400 that includes example instructions of an instruction set architecture for one or more supertiles.
[0075] As described above, the instruction set architecture of the system 100 can be configured such that each of the supertiles 102 receives a respective single instruction (or multiple instructions). Each of the single or multiple instructions can include data values (e.g., inputs and weights), specific data fields, and operational parameters for a workload or a set of tasks in a workload. Thus, each of the one or more instructions provided to the supertiles 102 via the communication bus 124 can include multiple parameters or data fields. Each of the data fields can be associated with a particular operation. In some cases, one or more bits of a data field in an instruction can be set to a particular binary value that causes a particular operation to occur in a single computational tile 108 or multiple computational tiles 108.
[0076] Referring now to table 400, a data field for an exemplary tensor operation ("TensorOp") to be executed in a computational thread of a particular computational tile 108n indicates the TensorOp pipeline of the target thread 402. In some implementations, based on an instruction received at the supertile 102, multiple data fields may be simultaneously multicast to each of the computational tiles 108n for a respective computational thread to be executed in the computational tile 108n.
[0077] A data field for an exemplary DMA operation ("NarrowToWide DMA") indicates (404) the wide memory configuration 140 of the target thread that will receive the data retrieved from the unified memory 106. In some implementations, the DMA operation may be performed in the supertile 102 to move data representing each set of weights for a neural network layer from the unified memory 106 (e.g., a narrow memory) to the local register file 140 of the wide memory configuration 140. For example, the set of weights is moved to the local register file 142 of the wide memory configuration 140 of the target computational thread. In some implementations, an example operation performed by a target computation thread may include a TensorOp TTU 134 obtaining a weight value from a local register file 142, passing the weight value from a computation tile 108 n to a cell 152, and the cell 152 using the weight value as an operand for a neural network computation performed to generate an output for a neural network layer.
[0078] The data field for another DMA operation ("RingBusConsumer DMA") is A wide memory configuration 140 of a target thread is shown receiving a portion of the data contained in (or included with) an instruction provided to the tile 102 (406). In some implementations, the data field for this DMA operation may correspond to a particular bitmap field in an instruction obtained from the communication bus 124 (e.g., a ring bus). In general, the bitmap may have a particular width defined in terms of bits.
[0079] For example, the header (e.g., a bitmap) of an instruction may indicate to a receiving supertile 102 how the supertile 102 should consume the portion of the data associated with the header based on the values of the individual bits of the bitmap field of the header. The specific manner in which the supertile 102 is required to consume that portion of the data may be an instruction subtype (or subtype of the instruction). In some implementations, the controller 104 of each of the receiving supertiles 102 examines the header bitmap of an instruction (e.g., a single instruction) and determines that the subtype of the instruction indicates that the portion of the data should be received by the wide memory configuration 140 of the supertile 102. For example, the instruction subtype may indicate the local register file 142 of the target thread to receive the respective set of weights associated with the portion of the data.
[0080] The data fields for another exemplary operation ("LoadCoefficientTables") are The memory of the supertile 102 is pointed to (408) for loading the coefficient table contained in (or with) the instruction given to the supertile 102. The data field for this load operation is the bitmap for the RingBusConsumer DMA operation described above. The coefficient table may correspond to a particular bitmap field in the instruction that is different from the k bitmap field. In some implementations, the coefficient table is used by each of the target threads of the supertile 102 to perform the neural network computations for the example machine learning workload. In some cases, the coefficient table may be stored across the respective wide memory configurations 140 associated with each computation thread. In other cases, the coefficient table may be stored in some other dedicated memory of the supertile 102 accessible by each of the k computation threads.
[0081] A data field for a synchronization flag operation ("SyncFlag") indicates the synchronization flag of the target thread (410). In some implementations, the data field for the exemplary synchronization flag operation in the instruction is set only for synchronization flags that are replicated across two or more supertiles 102. The data field for a synchronization watcher operation ("SyncWatcher") in a supertile 102 indicates the synchronization flag of the target thread (410). Wait for the SyncFlag to be generated and set the "thread_id" field in the instruction for the "SyncFlag" duplication instruction. a) ignore or b) wait for the SyncFlag corresponding to the "thread_id" field in the "SyncFlag" replication command (412). The data field of the fence operation "TileFence" is set to "reset_sync_flag_thread_ids". The tile fence operation may include a "wait_idle_thread_ids" data field and a "wait_idle_thread_ids" data field (414), which specify whether to reset or wait for synchronization flags in the corresponding computation threads to which the tile fence operation is connected.
[0082] 5 is a flow diagram illustrating an example process 500 for accelerating neural network computations. Process 500 can be implemented or performed using the system 100 described above. The description of process 500 may refer to the computing resources described above of system 100. In some implementations, steps or actions of process 500 are enabled by programmed firmware or software instructions executable by one or more processors of the devices and resources described herein.
[0083] Referring now to process 500, an example SuperTile 102 of system 100 receives inputs to and weights for a neural network layer (502). For example, the SuperTile 102 may receive the inputs and weights via a communication bus 124. In addition to receiving the inputs and weights, the SuperTile may receive one or more instructions for performing neural network calculations for the neural network layer and generating outputs for the neural network layer. A controller of the SuperTile stores the inputs and weights in a unified memory of the SuperTile (504). For example, the controller 104 stores the inputs and weights in the unified memory 106 based on the instructions received via the communication bus 124.
[0084] The arbitration logic unit of the supertile passes one or more of the inputs stored in the unified memory to each computational tile of the plurality of computational tiles in the supertile (506). The arbitration logic unit 110 is coupled to the unified memory 106 and to each computational tile 108n of the plurality of computational tiles 108. In some implementations, the controller 104 is configured to determine a partition of addresses in the unified memory 106 for storing each batch of inputs to be passed to a corresponding computational tile 108n of the supertile 102. For example, each partition of addresses of the unified memory 106 can be assigned to a respective computational tile 108n of the supertile.
[0085] The arbitration logic unit is configured to obtain, for a first partition of addresses, a first batch of inputs from memory locations identified by addresses in the partition of addresses and pass the first batch of inputs to cells 152 of a first computational tile 108n, which is assigned to receive each input in the first batch of inputs based on the determined partition of addresses in the unified memory. In some examples, the set of addresses in the partition of addresses may be for a batch of inputs that form a sample of input features. The sample of input features may include multiple sets of input features, where the multiple sets of input features correspond to a stream of image or audio data.
[0086] The arbitration logic unit passes the respective sets of weights stored in the unified memory to each of the computational tiles (508). The supertile 102 executes multiple computational threads in each of the computational tiles in the supertile to perform computations and generate outputs for the neural network layer (510). The supertile 102 generates outputs for the neural network layer based on computations performed in each of the computational tiles using one or more of the inputs and the respective sets of weights (512). In some implementations, the neural network layer is an embedding layer of a convolutional neural network, and the output generated by the neural network layer is an embedding output that includes an embedded feature vector.
[0087] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, or in a combination of one or more of these, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus.
[0088] Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.
[0089] The term "computing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. Such apparatus may include special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, such apparatus may further include code that creates an execution environment for the computer program, such as code that constitutes, for example, processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.
[0090] A computer program (which may also be called a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0091] A computer program may, but need not, correspond to a file in a file system. A program may exist in a single file dedicated to that program, or in multiple coordinated files (e.g., one or more modules of code). The program may be stored in a file that contains a program, subprogram, or portion thereof, in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document). A computer program may be deployed to be executed on one computer, or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.
[0092] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be realized by, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).
[0093] A processor suitable for executing a computer program may be based, by way of example, on a general purpose or special purpose microprocessor or both, or on any kind of central processing unit. Typically, a central processing unit receives instructions and data from a read-only memory or a random access memory or both. Some elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer further comprises, for example, a A computer may include one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, and / or be operatively coupled to receive data from or transfer data to the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive).
[0094] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0095] To provide for user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., an LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and pointing device, e.g., a mouse or trackball, by which a user can provide input to the computer. Other types of devices may be used to provide for user interaction, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.
[0096] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, a middleware component, e.g., an application server, a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.
[0097] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
[0098] Although the specification contains many specific implementation details, these may not limit the scope of any invention. The following descriptions should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operative in a combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.
[0099] Similarly, although operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, to achieve desirable results, or that all of the shown operations need to be performed. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system modules and components in the above-described embodiments does not require such separation in all embodiments, and that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.
[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
Claims
1. An integrated circuit configured to realize a neural network having a supertile and a plurality of neural network layers, the supertile and the integrated circuit comprising: a first computational tile configured to execute a first computational thread used to generate an output for a neural network layer; a second computational tile configured to execute a second computational thread used to generate the outputs for the neural network layer; and a memory shared between the first computational tile and the second computational tile and configured to store inputs to the neural network layer; an arbitration unit configured to determine when the first computational thread and the second computational thread are allowed to access the memory to retrieve respective portions of inputs used by the first computational tile or the second computational tile to generate the outputs for the neural network layer.
2. The integrated circuit of claim 1, wherein the arbitration unit is configured to pass respective portions of an input to the first computational tile and the second computational tile in response to determining when the first computational thread and the second computational thread are allowed to access the memory.
3. The integrated circuit of claim 2, wherein the arbitration unit is configured to pass neural network layer outputs between the memory and the first computational tile and the second computational tile.
4. An integrated circuit as described in any one of claims 1 to 3, further comprising a controller configured to partition dimensions of a multidimensional tensor across the first and second computational tiles of the supertile and across the first and second computational threads of the first and second computational tiles.
5. The integrated circuit of claim 4, wherein the first computational tile and the second computational tile of the supertile are configured to process element data values along 2D subdivisions of X, Y tensor dimensions assigned to the first computational tile and the second computational tile.
6. i) the controller is configured to determine a partition of addresses in the memory for the supertiles; ii) each address in the range of addresses is to a memory location storing one of the inputs to the neural network layer; iii) the input to the neural network layer comprises a set of input features; 6. The integrated circuit of claim 5, wherein iv) the set of input features corresponds to a stream of image or audio data.
7. i) the first computational tile of the supertile is configured to execute a first plurality of computational threads in parallel to process a first subset of input features in batches; 7. The integrated circuit of claim 6, wherein ii) the second computational tile of the supertile is configured to execute a second plurality of computational threads in parallel.
8. i) the first computational tile is configured to, in response to executing the first plurality of computational threads in parallel, perform a computation using a first portion of inputs of an input tensor and a first portion of weights of a weight tensor; 10. The integrated circuit of claim 7, further configured to: ii) generate a first portion of the outputs for the neural network layer based on the computations performed by executing the first plurality of computational threads in parallel.
9. i) the second computational tile is configured to perform a computation using a second portion of the inputs of the input tensor and a second portion of the weights of the weight tensor in response to executing the second plurality of computational threads in parallel; 10. The integrated circuit of claim 8, further configured to: ii) generate a second portion of the output for the neural network layer based on the computations performed by executing the second plurality of computational threads in parallel.
10. A method performed using an integrated circuit, the integrated circuit configured to realize a neural network comprising supertiles and comprising a plurality of neural network layers, the method comprising: obtaining an input to a neural network layer of the plurality of neural network layers in the supertile; storing the input in a shared memory of the supertile, the memory being shared between a first and a second computational tile of the supertile, the method further comprising: an arbitration unit of the supertile determining when each computational thread of the first or second computational tile of the supertile is permitted to access the memory to retrieve a respective portion of the input; executing in parallel a first computational thread in the first computational tile and a second computational thread in the second computational tile based on a memory access decision of the arbitration unit; and executing the first computational thread and the second computational thread in parallel to post-process the respective portions of the input to generate an output for the neural network layer.
11. The method of claim 10, wherein in response to the arbitration unit determining when the first and second computational threads are permitted to access the shared memory, passing respective portions of an input to the first and second computational tiles; The method of claim 10 , further comprising the arbitration unit passing a neural network layer output between the memory and the first and second computational tiles.
12. The method of claim 10 or 11, further comprising the integrated circuit controller partitioning dimensions of a multidimensional tensor across the first and second computational tiles of the supertile and across the first and second computational threads of the first and second computational tiles.
13. The method of claim 12, wherein the first computational tile and the second computational tile of the supertile are configured to process element data values along 2D subdivisions of X, Y tensor dimensions assigned to the first computational tile and the second computational tile.
14. The method of claim 13, further comprising the controller determining, in the shared memory, a partition of addresses for a plurality of computational tiles of the supertile.
15. i) each address in a range of addresses is to a memory location storing one of the inputs to the neural network layer; ii) the input to the neural network layer comprises a set of input features; The method of claim 14 , wherein iii) the set of input features corresponds to a stream of image or audio data.
16. The method of claim 15, wherein the first computational tile of the supertile executes a first plurality of computational threads in parallel to process a first subset of input features at the neural network layer; 16. The method of claim 15, further comprising: the second computational tile of the supertile executing a second plurality of computational threads in parallel to process a second subset of input features at the neural network layer.
17. The method of claim 16, further comprising: performing, in response to the first plurality of computational threads executing in parallel, a computation using a first portion of inputs of an input tensor and a first portion of weights of a weight tensor; 17. The method of claim 16, further comprising: the supertile generating a first portion of the outputs for the neural network layer based on the computations performed by executing the first plurality of computational threads in parallel.
18. The method of claim 17, further comprising: performing, in response to executing the second plurality of computational threads in parallel, a computation using a second portion of the inputs of the input tensor and a second portion of the weights of the weight tensor, the second computational tile performing a computation using the second portion of the inputs of the input tensor and a second portion of the weights of the weight tensor; and generating a second portion of the output for the neural network layer based on the computations performed by executing the second plurality of computational threads in parallel.
19. i) the inputs to the neural network layer are represented as multidimensional tensors represented by the input tensors; ii) the first portion of the input tensor is included in the input to the neural network layer; 20. The method of claim 18, wherein iii) the second portion of the input tensor is included in the input to the neural network layer.
20. A computer program executable by a processing device to cause the performance of a method according to any one of claims 10 to 19.
