Neural network accelerator with on-chip resident parameters

Abstract

A hardware compute system for performing computations of a neural network layer by distributing tensor computations across a plurality of compute tiles.SOLUTION: The computing system 100 includes a controller 102, a host interface 108, an input / output (I / O) link 110, a plurality of tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a data bus identified in a bus map 118. The controller includes a data memory 104, an instruction memory 106, and a processor that executes one or more instructions encoded in a computer-readable storage medium. The instruction memory stores one or more machine-readable instructions that are executable by the processor of the controller. The controller executes one or more instructions related to tensor computations in the system, including instructions stored in the instruction memory.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 U.S.C. §119(e) of the filing date of U.S. patent application Ser. No. 62 / 544,171, entitled "Neural Network Accelerator with Parameters Resident on Chip," filed Aug. 11, 2017, the entire contents of which are incorporated herein by reference.

[0002] background This specification generally relates to neural network (NN) computation tiles for computing deep neural network ("DNN") layers. Summary of the Invention

[0003] overview In general, one innovative aspect of the subject matter described in this specification can be embodied in storing neural network parameters on an accelerator. Neural networks differ from typical computational workloads in that their working set, i.e., the total amount of storage required for the entire computation, is practically limited. Roughly speaking, this working set typically corresponds to a number of parameters, ranging from hundreds of thousands to billions. This amount of storage is consistent with existing hardware storage technology.

[0004] Despite these facts, current accelerators include local storage for parameters that are passed through, i.e., parameters do not reside permanently on-chip; rather, parameters flow from external memory with each new inference.

[0005] Therefore, external memory bandwidth is a key limitation of all neural network (NN) accelerators. The embodiments described herein replace temporary local storage for parameters with on-chip storage for parameters. That is, the embodiments keep all parameters of the NN resident within the accelerator and do not flush them from external memory.

[0006] The advantages of storing parameters on-chip include overcoming the performance limitations of NN accelerators, dramatically facilitating an increase in the number of multiply-accumulate ("MAC") operators since performance limitations are overcome, and providing a lower-power neural network accelerator since external memory accesses typically require at least an order of magnitude more energy than local memory accesses.

[0007] In certain embodiments, the accelerator comprises a computation unit, the computation unit comprising a first memory bank for storing input activations or output activations and a second memory bank for storing neural network parameters used in performing the computation, the second memory bank configured to store a sufficient amount of the neural network parameters on the computation unit to enable a latency below a particular level and a throughput above a particular level for a given NN model and architecture, the computation unit further comprising at least one cell including at least one MAC operator that receives the parameters from the second memory bank and performs the computation, and a first traversal unit in data communication with at least the first memory bank, the first traversal unit The accelerator is configured to provide control signals to one memory bank to provide input activations to a data bus accessible by the MAC operator. The accelerator performs one or more computations related to at least one element of the data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between the input activations received from the data bus and parameters received from the second memory bank. If the parameter storage is sufficient to hold all neural network parameters, then the accelerator performance is not determined by memory bandwidth. In that case, it is possible to provide all MACs with parameters every cycle.

[0008] Another innovative aspect of the subject matter described herein may be embodied in a computer-implemented method for accelerating tensor computations, the computer-implemented method including sending, by a first memory bank, a first input activation in response to the first memory bank receiving a control signal, the first input activation being sent over a data bus, the method further including receiving, by at least one MAC operator, one or more parameters from a second memory bank for storing neural network parameters used in performing the computation, the second memory bank storing a sufficient amount of the neural network parameters on a computation unit to enable a latency below a particular threshold and a throughput above a particular threshold for a given NN model and architecture, the method further including performing, by the MAC operator, one or more computations related to at least one element of a data array, the one or more computations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank.

[0009] Another innovative aspect of the subject matter described herein may be embodied in a method for accelerating computations, the method including, prior to execution, loading neural network weight parameters into a neural network accelerator and, at execution time, processing inputs to the accelerator without substantially accessing the neural network weight parameters external to the accelerator. If the number of neural network weight parameters is too large to fit on a single accelerator die, the method may further include loading the neural network weight parameters into multiple densely connected accelerator dies and processing inputs to the accelerator dies without substantially accessing the neural network weight parameters external to the densely connected accelerator dies.

[0010] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages: Using registers to track memory address values ​​allows a program to iterate through deeply nested loops with a single instruction. Tensors accessible from narrow and wide memory units in a single computational tile are traversed based on memory address values ​​retrieved from registers. The memory address values ​​correspond to elements of the tensor. Tensor computations occur in individual computational tiles based on the execution of deep loop nests. Computations can be distributed across multiple tiles. Computational efficiency is improved and accelerated based on distributing tensor computations of a multi-layer neural network across several computational tiles. Tensors can be traversed and tensor computations can be performed with fewer instructions.

[0011] The embodiments described herein affect the operation and design of neural network (NN) accelerators. The embodiments address one of the significant limitations of neural network accelerator design: the combination of low latency and high throughput. By having parameters reside on-chip, low latency can be achieved given high throughput and specific NN models and architectures. is dramatically reduced. The accelerator does not require high memory bandwidth and energy is lower.

[0012] Tiling as described herein provides compiled locality of reference. For example, placing a fully connected model next to SRAM increases internal bandwidth compared to a cached model. The embodiments described herein run faster than convolutional neural network accelerators. Certain embodiments have many more operators. To supply the operators, the accelerator requires more internal bandwidth. To address this, the architecture must distribute memory and aggregate parameters on the accelerator.

[0013] The subject matter described herein can also be implemented in particular embodiments to realize other advantages. For example, by employing a memory hierarchy that combines narrow, low-bandwidth memory with wide, high-bandwidth memory, high utilization of MAC operators can be achieved for DNN layers of very different dimensionality and locality of reference. The narrow, low-bandwidth memory allows for flexible addressing to traverse multidimensional arrays in any order.

[0014] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof that is installed on the system and that, when operated, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0015] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of an example computing system. [Figure 2] 1 illustrates an exemplary neural network computational tile. [Figure 3] 1 illustrates an exemplary tensor traversal unit (TTU) structure. [Figure 4] 1 illustrates an example architecture that includes a narrow memory unit that provides input activations to one or more multiply-accumulate (MAC) operators. [Figure 5] 5 illustrates an exemplary architecture including an output bus that provides output activation to the narrow memory units of FIGS. 2 and 4. [Figure 6] 3 is an exemplary flowchart of a process for performing tensor computations using the neural network computational tile of FIG. 2. [Figure 7] 1 is another embodiment of a computing system. [Figure 8] 1 is another embodiment of a neural network computation tile. [Figure 9]1 is an exemplary flowchart of a process for accelerating computations by loading neural network weight parameters into a neural network accelerator. DETAILED DESCRIPTION OF THE INVENTION

[0017] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description The subject matter described in this specification relates to a method for accelerating computations. includes, prior to execution, loading neural network weight parameters into a neural network accelerator, and, during execution, processing inputs to the accelerator without substantially accessing neural network weight parameters external to the accelerator. If the number of neural network weight parameters is too large to fit on a single accelerator die, the method may further include loading the neural network weight parameters into multiple densely connected accelerator dies, and processing inputs to the accelerator dies without substantially accessing neural network weight parameters external to the densely connected accelerator dies.

[0018] The subject matter described herein also relates to a hardware computing system including multiple computing units configured to accelerate machine learning inference workloads of neural network layers. Each computing unit of the hardware computing system is self-contained and can independently perform the computations required by a given layer of a multi-layer neural network. This specification generally relates to neural network (NN) computing tiles for computing deep neural network ("DNN") layers with on-chip resident parameters to enable latency below a certain level and throughput above a certain level for a given NN model and architecture.

[0019] A neural network with multiple layers can be used to compute an inference. For example, given an input, the neural network can compute an inference for that input. The neural network computes this inference by processing the input through each layer of the neural network. In particular, each layer of a neural network has its own set of weights. Each layer receives an input and processes the input according to the set of weights for that layer to generate an output.

[0020] Thus, to compute an inference from a received input, a neural network receives the input and processes it through each neural network layer to generate an inference, with the output from one neural network layer being provided as the input to the next neural network layer. The data input or output for a neural network layer, e.g., the input to the neural network or the output of the layer below it in the sequence, can be referred to as the activations of that layer.

[0021] In some implementations, the layers of a neural network are arranged in a sequence. In other implementations, the layers are arranged in a directed graph, meaning that any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be configured so that the output of one layer can be sent back as input to a previous layer.

[0022] The hardware computing system described herein can perform neural network layer computations by distributing tensor computations across multiple computation tiles. The computation process performed within the neural network layer can include multiplication of input tensors including input activations and parameter tensors including weights. The computation includes multiplying the input activations by the weights in one or more cycles and performing an accumulation of the products over many cycles.

[0023] Tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. Generally, processing is performed by computation tiles to perform tensor computations by processing nested loops to traverse N-dimensional tensors. In one exemplary computation process, each loop traverses an N-dimensional tensor. A tensor traversal unit may be responsible for traversing a particular dimension of a tensor. For a given tensor construct, a computational tile may need access to elements of that tensor to perform multiple dot product calculations associated with that tensor. A computation occurs when input activations, provided by narrow memory structures, are multiplied by parameters or weights, provided by wide memory structures. Because tensors are stored in memory, a set of tensor indices may require conversion to a set of memory addresses. In general, a tensor traversal unit in a computational tile performs control operations that provide the indices for each dimension associated with a tensor and the order in which the index elements are traversed and the computation is performed. The tensor computation is complete when the multiplication results are written to an output bus and stored in memory.

[0024] FIG. 1 illustrates a block diagram of an exemplary computing system 100 for accelerating tensor computations associated with deep neural networks (DNNs). System 100 generally includes a controller 102, a host interface 108, an input / output (I / O) link 110, a plurality of tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a data bus (shown for clarity but not included in system 100) identified in a bus map 118. Controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. Instruction memory 106 can store one or more machine-readable instructions executable by the one or more processors of controller 102. Data memory 104 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within system 100.

[0025] The controller 102 is configured to execute one or more instructions related to tensor computations in the system 100, including instructions stored in the instruction memory 106. In some implementations, the data memory 104 and the instruction memory 106 are volatile memory units or units. In other implementations, the data memory 104 and the instruction memory 106 are non-volatile memory units or units. The data memory 104 and the instruction memory 106 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In various implementations, the controller 102 may be referred to as or be called a core manager 102.

[0026] As shown, host interface 108 is coupled to I / O link 110, controller 102, and classifier portion 116. Host interface 108 receives instructions and data parameters from I / O link 110 and provides instructions and parameters to controller 102. Generally, instructions may be provided to one or more devices in system 100 via instruction bus 124 (described below), and parameters may be provided to one or more devices in system 100 via ring bus 128 (described below). In some implementations, instructions are initially received by controller 102 from host interface 118 and stored in instruction memory 106 for later execution by controller 102.

[0027] The classifier portion 116 is similarly coupled to the controller 102 and to tile 7 of the second tile set 114. In some implementations, the classifier portion 116 is implemented as a separate tile within the system 100. In alternative implementations, the classifier portion 116 is disposed or located within the controller 102 as a subcircuit or subdevice of the controller 102. The classifier portion 116 generally comprises an accumulated tile received as the output of the fully connected layer. The fully connected layer is configured to perform one or more functions on the pre-activation values ​​obtained by the fully connected layer. The fully connected layer may be divided across tiles in tile sets 112 and 114. Thus, each tile is configured to generate a subset of pre-activation values ​​(i.e., linear outputs) that can be stored in the tile's memory unit. A classification result bus 120 provides a data path from the classifier portion 116 to the controller 102. Data including post-function values ​​(i.e., results) are provided from the classifier portion 116 to the controller 102 via the classification result bus 120.

[0028] The bus map 118 illustrates data buses providing one or more interconnected data communication paths between tiles in the first tile set 112 and the second tile set 114. The bus map 118 provides a notational explanation for identifying the classification result bus 120, the CSR / master bus 122, the instruction bus 124, the mesh bus 126, and the ring bus 128, as shown in FIG. 1 . Generally, tiles are core components within the accelerator architecture of the system 100 and are the focal point for tensor computations occurring within the system. Each tile is an individual computational unit that correlates with other tiles in the system and can accelerate computations across one or more layers of a multi-layer neural network. While tiles within the tile sets 112, 114 can share the execution of tensor computations associated with a given instruction, individual computational units are self-contained computational components configured to perform a subset of tensor computations independently of other corresponding tiles within the tile sets 112, 114.

[0029] The CSR bus 122 is a single-master, multiple-slave bus that allows the controller 102 to send one or more instructions to set program configurations and read status registers associated with one or more tiles. The CSR bus 122 can be connected in a single daisy-chain configuration with one master bus segment and multiple slave bus segments. As shown in FIG. 1 , the CSR bus 122 provides communication coupling the tiles of the tile sets 112, 114 and the controller 102 via a bus data path that connects them in a ring to the host interface 110. In some implementations, the host interface 110 is the single master of the CSR bus ring, and the entire CSR bus address space is memory-mapped into memory space within the host interface 110.

[0030] The CSR bus 122 may be used by the host interface 110 to perform one or more operations, including, for example, programming memory buffer pointers within the controller 102 to enable the controller 102 to begin fetching instructions from the instruction memory 106, updating / programming various tile settings (e.g., coefficient tables for polynomial approximation calculations) that remain static for one or more calculations, and / or loading / reloading firmware for the classifier portion 116. In one example, a firmware reload may include a new function to be applied to the linear output (i.e., pre-activation values). Thus, every slave with access to the CSR bus 122 will have a distinct node identifier (node ​​ID) associated with and identifying that slave. The node ID is part of the instruction address and is used, inspected, or otherwise consulted by the CSR slave (i.e., the controller 102, tiles 112, 114, and classifier 116) to determine whether a CSR packet is addressed to the slave.

[0031] In some implementations, one or more commands can be sent by the host interface 102 through the controller 102. A command may be, for example, 32 bits wide, with the first 7 bits containing header information indicating the command address / destination that is to receive and execute the command. The first 7 bits of the header may contain a data parameter that represents a particular node ID. Thus, a slave on the CSR bus ring (e.g., each tile) can inspect the command header to determine whether a request by the master (host interface 110) is a header. If the node ID in the header does not indicate that the destination is a test tile, the test tile copies the incoming CSR command packet to the CSR bus input connected to the next tile for testing by the next tile.

[0032] The instruction bus 124 provides communications that originate from the controller 102 and couple the tiles in the tile sets 112, 114 via bus data paths that, similar to the CSR bus 122, connect the tiles in a ring back to the controller 102. In one implementation, the controller 102 broadcasts one or more instructions via the instruction bus 124. The instructions broadcast by the controller 102 may differ from the instructions provided via the CSR bus 122. However, the manner in which tiles receive and / or consume or execute instructions received via the bus 124 may be similar to the process for executing instructions received via the CSR bus 122.

[0033] In one example, the header of an instruction (i.e., a bitmap) indicates to a receiving tile that it needs to consume a particular instruction based on the bitmap associated with that instruction. The bitmap may have a particular width defined in terms of bits. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of instruction bus 124 may be configured to be smaller than the size / width of the instruction. Thus, in such a configuration, the transmission of instructions occurs over several cycles, and the bus stops of instruction bus 124 have decoders to place the instructions received at that tile into the appropriate target instruction buffer associated with that tile.

[0034] As described further below, tiles within tile sets 112, 114 are generally configured to support two broad categories of instructions. The two broad categories are also referred to as instruction types. The instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, a DMAOp instruction has one or more specializations that are allowed to occur simultaneously. The one or more specializations may be referred to as DMAOp instruction subtypes or opcodes. In some cases, every unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a particular tile.

[0035] At a particular one of the tiles 112, 114, the bus stop associated with the instruction bus 124 examines the header bitmap to determine the instruction type / subtype. The instruction may be received by the tile and subsequently written to the tile's instruction buffer prior to execution of the instruction by the tile. The tile's instruction buffer to which the instruction is written may be determined by the instruction's type and subtype indicator / fields. The instruction buffer may include a first-in, first-out (FIFO) control scheme that prioritizes consumption of one or more associated instructions. Therefore, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which they arrive on the instruction bus.

[0036] The different instruction buffers within a tile are TensorOp instruction buffers and DMAOp instruction buffers. As noted above, instruction types include TensorOp instructions and DMAOp instructions. With respect to DMAOp instructions, instruction subtypes (indicating the "write to" buffer location) include: 1) mesh inbound instruction buffer; 2) mesh outbound instruction buffer; 3) narrow-wide DMA instruction buffer; 4) wide-narrow DMA instruction buffer; and 5) ring bus DMA instruction buffer. These buffer locations are described in more detail below with reference to FIG. 2. Wide and narrow designations are used throughout this specification and generally refer to the approximate width size (bits / bytes) of one or more memory units. As used herein, "narrow" refers to a buffer that is less than 16 bits each. "Wide" may refer to one or more memory units each having a size or width between 16 bits and 64 bits.

[0037] The mesh bus 126 provides a data communication path that is different from the CSR bus 122, the instruction bus 124, and the ring bus 128 (described below). As shown in FIG. 1, the mesh bus 126 provides a communication path that couples or connects each tile to its corresponding neighboring tiles in both the X and Y dimensions. In various implementations, the mesh bus 126 can be used to transport input activation amounts between one or more narrow memory units in adjacent tiles. As shown, the mesh bus 126 does not allow input activation data to be directly transferred to non-adjacent tiles.

[0038] In various implementations, the mesh bus 126 and the various tiles connected via the mesh bus 126 may have the following configurations: The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge, non-corner tiles have four inbound ports and four outbound ports. Generally, in an example NxN tile layout, edge tiles are tiles with no more than three neighboring tiles, and corner tiles are tiles with two neighboring tiles. With regard to the dataflow methodology over the mesh bus 126, generally, all input activations arriving via the mesh bus 126 for a particular tile must be committed to one or more narrow memory units of that tile. Furthermore, for tile configurations with fewer than four inbound ports, a DMAOp instruction may write zero values ​​to locations in the tile's narrow memory instead of waiting for data on nonexistent input ports. Similarly, for tile configurations with fewer than four outbound ports, DMAOp instructions do not perform narrow memory reads and port writes associated with transfers to non-existent ports.

[0039] In some implementations, the location or address of the narrow memory unit(s) to which a particular input activation will be written or read will be generated by a tensor traversal unit (hereinafter "TTU") based on inbound / outbound DMAOps provided via mesh bus 126. The inbound and outbound DMAOps may be executed simultaneously, with any necessary synchronization being managed by a synchronization flag control scheme managed by controller 102. The TTU is described in further detail below with reference to Figures 2 and 3.

[0040] The ring bus 128 provides communications that originate from the controller 102 and couple through bus data paths that connect the tiles 112, 114 in a ring-like fashion back to the controller 102, similar to the CSR bus 122 and the instruction bus 124. In various implementations, the ring bus 128 generally connects or couples all of the wide memory units (described in more detail below with reference to FIG. 2) within all of the tiles 112, 114. Thus, the payload width of the ring bus 128 corresponds to the width of the wide memory units located within each tile of the tile set 112, 114. As mentioned above, the ring bus 128 also includes a bitmap header that indicates which tiles should consume payload data, including instructions or parameters, communicated over the ring bus 128.

[0041] With respect to data (i.e., payload) received at a particular tile over ring bus 128, in response to receiving the information, each tile zeros (i.e., clears) the location data indicated in the bitmap header specific to the receiving tile before forwarding the data to another tile. Thus, the header bitmap indicates the location of the tile that is to receive the payload. When a tile has no remaining bits set data indicating a particular tile to be processed, the transfer of payloads to another tile will stop. Payload data generally refers to activations and weights used by one or more tiles during tensor computations performed based on execution of deeply nested loops.

[0042] In some implementations, the controller 102 may be described as part of the ring bus 128. In one example, for a DMAOp instruction executed within a particular tile, the controller 102 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 102 may also commit the payload data to one or more wide memory units of the tile if required by an instruction in the bitmap header. The addresses of the one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.

[0043] In various implementations, each tile in tile set 112, 114 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over ring bus 128 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives data, writes it to one or more wide memory units within the tile, and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data over ring bus 128, there is typically only one producer / master of data on ring bus 128 at any given time. The DMAOp instruction execution order (e.g., a FIFO control scheme) among all tiles will ensure that there is only one producer / master of data on ring bus 128 at a given time.

[0044] In some implementations, the controller 102 uses a synchronization flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 128 at a given time. In one example, each write by a tile to a ring output will trigger an increment of a corresponding synchronization flag count. The controller 102 can examine the payload data to determine the number of data chunks or segments that comprise the payload. The controller 102 then monitors execution by a tile to ensure that the expected number of data segments is transferred and / or consumed by that tile before another tile executes in master mode.

[0045] An exception to ensuring that there is only one producer / master of data on ring bus 128 at a given time occurs when there are local multicast groups connected via ring bus 128 that do not have overlapping regions on ring bus 128. For example, Tile 0 (master) may multicast (i.e., generate data) to a tile in the Tile 0-Tile 3 grouping, and Tile 4 (master) may do the same for a tile in the Tile 4-Tile 7 grouping. A key requirement of this dual-master multicast method is that different multicast groups cannot see each other's data packets, as this could result in packet duplication and lead to one or more data calculation errors.

[0046] 1, the controller 102 provides a communication data path that couples or connects the tiles in the tile sets 112, 114 to the I / O 110 and includes several core functions. The core functions of the controller 102 generally include providing one or more I / O input activations to the tiles in the tile sets 112, 114, transmitting one or more I / O input activations to the tiles in the tile sets 112, 114, and transmitting one or more I / O input activations to the tiles in the tile sets 112, 114 from the I / O 110. or providing multiple input activations and parameters to the tiles, providing one or more instructions received from I / O 110 to the tiles, sending I / O output activations to host interface 108, and acting as ring stops for CSR bus 122 and ring bus 128. As described in more detail below, first tileset 112 and second tileset 114 each include multiple tiles used to perform one or more tensor computations that are performed based on a deep loop nest consisting of an inner loop and an outer loop.

[0047] The system 100 generally operates as follows: The host interface 108 provides the controller 102 with one or more instructions that define the direct memory access operations (DMAOps) to occur for a given computation. Descriptors associated with the instructions provided to the controller 102 will contain information needed by the controller to facilitate large-scale dot product computations involving multidimensional data arrays (tensors). Generally, the controller 102 receives input activations, tile instructions, and model parameters (i.e., weights) from the host interface 108 to perform tensor computations for a given layer of the neural network. The controller 102 can then multicast the instruction(s) to the tiles 112, 114 in a dataflow manner defined by the instructions. As described above, a tile that consumes an instruction can then initiate broadcasting new / subsequent instructions to another tile based on the bitmap data in the instruction header.

[0048] In terms of data flow, input activations and parameters are sent to tiles in the tile set 112, 114 via the ring bus 128. Each of the tiles 112, 114 will store a subset of the input activations needed to compute the subset of output activations assigned to that particular tile. A DMAOp instruction to a tile moves the input activations from wide memory to narrow memory. A computation within a tile begins when the necessary input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) become available to the tile. A computation occurring within a tile ends when the MAC operator (described below) within the tile has completed all dot product operations defined by the instruction set and the pre-activation function has been applied to the results of the multiplication operations (i.e., the output activations).

[0049] The results of one or more tensor computations include writing the output activations of the computational layer to the narrow memory unit(s) of the tile performing the computation. Some tensor computations result in the transfer of output edge activations to neighboring tiles via the mesh bus 126. Transferring output edge activations to neighboring tiles is required to calculate output activations for subsequent layers when the computation spans multiple layers. Once the computations for all layers are complete, a DMAOp moves the final activations to the classifier tile 116 via the ring bus 128. The controller 102 then reads the final activations from the classifier tile 116 and executes a DMAOp to move the final activations to the host interface 108. In some implementations, the classifier portion 116 performs the computations for the output layer (i.e., the last layer) of the NN. In other implementations, the output layer of the NN is a classifier layer, a recurrent layer, or one of another layer types commonly associated with neural networks.

[0050] 2 illustrates an exemplary neural network (NN) computational tile 200. In general, the exemplary tile 200 may correspond to any tile within the first tile set 112 and the second tile set 114 described above with reference to FIG. 1. In various implementations, the computational tiles 200 may also be referred to as or be referred to as computational units 200. Each computational tile 200 is a self-contained computational unit configured to execute instructions independently of other corresponding tiles within the tile sets 112, 114. As briefly described above, each computational tile 200 may execute two types of instructions: TensorOp instructions and D Execute MAOp instructions. Generally, each instruction type involves computational operations related to deep loop nests, and therefore each instruction type will generally be executed over multiple time epochs to ensure completion of all loop iterations.

[0051] As discussed in more detail below, different instruction types are executed by independent control units within the computational tile 200 that synchronize on data via synchronization flag control managed within the computational tile 200. The synchronization flag control manages concurrency between the execution of different instruction types within the computational tile 200. Each computational operation associated with each instruction type is executed in strict issue order (i.e., first-in-first-out). For two instruction types, TensorOP and DMAOp, there are no ordering guarantees between these different instruction types, and each type is treated as a separate control thread by the computational tile 200.

[0052] With respect to the data flow configuration, the computational tile 200 generally includes a data path 202 and a data path 205, each of which provides a communication path for data flow into and out of the computational tile 200. As described above, the system 100 includes three different data bus structures laid out in a ring configuration: the CSR bus 122, the instruction bus 124, and the ring bus 128. With reference to FIG. 2 , the data path 205 corresponds to the instruction bus 124, and the data path 202 generally corresponds to one of the CSR bus 122 and the ring bus 128. As shown, the data path 202 includes a ring output 203 that provides an output path for data exiting the computational tile 200, and a ring input 204 that provides an input path for data entering the computational tile 200.

[0053] Compute tile 200 further includes TensorOp control 206, which includes TensorOp tensor traversal units (TTUs) 226, and DMAOp control 208, which includes DMAOpTTUs 228. TensorOp control 206 generally manages writes to and reads from TensorOpTTU registers 232, managing traverse operations for execution by TensorOpTTU 226. Similarly, DMAOp control 208 generally manages writes to and reads from DMAOpTTU registers 234, managing traverse operations for execution by DMAOpTTU 228. TTU registers 232 include an instruction buffer for storing one or more instructions, including operations to be performed by TensorOpTTU 226 upon execution of the instruction by TensorOp control 206. Similarly, the TTU registers 234 include an instruction buffer for storing one or more instructions containing operations to be performed by the TTU 208 upon execution of the instruction by the DMAOp control 208. As explained further below, the TTUs are used by the computation tiles 200 to traverse array elements of one or more tensors that typically reside in the narrow memory 210 and the wide memory 212.

[0054] In some implementations, an instruction for execution by a computational tile 200 arrives at the tile via data path 205 (i.e., part of instruction bus 124). The computational tile 200 examines a header bitmap to determine the instruction type (TensorOp or DMAOp) and instruction subtype (read operation or write operation). Instructions received by the computational tile 200 are then written to a particular instruction buffer depending on the instruction type. Generally, instructions are received and stored (i.e., written to a buffer) prior to execution of the instructions by components of the computational tile 200. As shown in FIG. 2, the instruction buffers (i.e., TensorOpTTU register 232 and DMAOpTTU register 234) can each include a first-in, first-out (FIFO) control scheme to prioritize consumption (execution) of one or more associated instructions.

[0055] As briefly mentioned above, tensors are multidimensional geometric objects, and exemplary multidimensional Original geometric objects include matrices and data arrays. Algorithms involving deeply nested loops may be executed by the computational tiles 200 to perform tensor computations by iterating through one or more nested loops to traverse an N-dimensional tensor. In one exemplary computational process, each loop in a loop nest may be responsible for traversing a particular dimension of the N-dimensional tensor. As described herein, the TensorOp control 206 generally manages one or more tensor operations that drive the sequence of traversing and accessing dimensional elements of a particular tensor construct to complete the computation defined by the deeply nested loop.

[0056] Computational tile 200 further includes narrow memory 210 and wide memory 212. The narrow and wide designations generally refer to the width size (bits / bytes) of the memory units of narrow memory 210 and wide memory 212. In some implementations, narrow memory 210 includes memory units each having a size or width of less than 16 bits, and wide memory 212 includes memory units each having a size or width of less than 32 bits. Generally, computational tile 200 receives input activations via data path 205, and DMA control 208 performs operations to write the input activations to narrow memory 210. Similarly, computational tile 200 receives parameters (weights) via data path 202, and DMA control 208 performs operations to write the parameters to wide memory 212. In some implementations, narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine which control unit (e.g., TensorOp control 206 or DMAOp control 208) is allowed to access the shared memory unit of narrow memory 210 for each memory cycle.

[0057] The computational tile 200 further includes an input activation bus 216 and a MAC array 214 including multiple cells, each including a MAC operator 215 and a sum register 220. Generally, the MAC array 214 performs tensor computations, including arithmetic operations involving dot product calculations, using the MAC operators 215 and sum registers 220 across multiple cells. The input activation bus 216 provides a data path along which input activations are provided by the narrow memory 210, one for each access by each MAC operator 215 of the MAC array 214. Thus, based on the broadcast of the input activations one by one, a single MAC operator 215 of a particular cell will each receive the input activation. The arithmetic operations performed by the MAC operators of the MAC array 214 generally involve multiplying the input activations provided by the narrow memory 210 with parameters accessed from the wide memory 212 to generate a single output activation value.

[0058] During an arithmetic operation, partial sums may be accumulated and stored in corresponding, e.g., sum registers 220, or written to wide memory 212 and re-accessed by specific cells of MAC array 214 to complete a subsequent multiplication operation. A tensor computation can be described as having a first and a second part. The first part is completed when the multiplication operation produces an output activation, e.g., by completing the multiplication of the input activations with a parameter to produce the output activation. The second part involves the application of a nonlinear function to the output activation, and the second part is completed when the output activation is written to narrow memory 210 after the function is applied.

[0059] Computational tile 200 further includes an output activation bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 indicating core attributes of the components within computational tile 200. While reference map 230 is shown for clarity, it is not included in computational tile 200. Core attributes include whether a particular component is a unit, storage device, operator, controller, or data path. Generally, upon completion of the first portion of the tensor computation, output activations are provided from MAC array 214 to NLU 222 via output activation bus 218. NLU 2 After arriving at 22, data specifying an activation function, received via activation pipeline 224, is applied to the output activations, which are then written to narrow memory 210. In some implementations, output activation bus 218 includes at least one pipelined shift register 236, and completing the second part of the tensor computation includes shifting the output activations toward narrow memory 210 using the shift register 236 of activation bus 218.

[0060] For example, for a dot product calculation of two multidimensional data arrays for a single computational tile 200, the MAC array 214 provides robust single instruction, multiple data (SIMD) capabilities. SIMD generally means that all parallel units (multiple MAC operators 215) share the same instruction (based on deep loop nesting), but each MAC operator 215 executes the instruction on different data elements. In one basic example, adding arrays [1,2,3,4] and [5,6,7,8] element-by-element to obtain array [6,8,10,12] in one cycle typically requires four arithmetic units to perform the operation on each element. By using SIMD, the four units share the same instruction (e.g., "add") and can perform the calculation in parallel. Thus, the system 100 and computational tile 200 provide improved acceleration and parallelism in tensor computations over conventional methods.

[0061] In one example, and as described in more detail below, a single instruction may be provided by the controller 102 to multiple computational tiles 200 (see tile sets 112, 114 in FIG. 1 ) for consumption by multiple MAC arrays 214. In general, a neural network layer may include multiple output neurons, which may be partitioned such that tensor computations associated with a subset of the output neurons can be assigned to a particular tile of the tile set 112, 114. Each tile of the tile set 112, 114 may then perform the associated tensor computations on a different group of neurons for a given layer. Thus, the computational tiles 200 may provide at least two forms of parallelism: 1) one form involving partitioning output activations (corresponding to a subset of output neurons) among multiple tiles of the tile set 112, 114; and 2) another form involving simultaneous computation (using a single instruction) of multiple subsets of output neurons based on the partitioning among tiles of the tile set 112, 114.

[0062] FIG. 3 shows an exemplary tensor traversal unit (TTU) structure 300 containing four tensors to track, each with a depth of eight. The TTU 300 generally includes a counter tensor 302, a stride tensor 304, an initial tensor 306, and a limit tensor 308. The TTU 300 further includes an adder bank 310 and a tensor address index 312. As mentioned above, tensors are multidimensional geometric objects, and to access elements of a tensor, an index for each dimension must be given. Because tensors are stored in narrow memory 210 and wide memory 212, a set of tensor indices must be converted to a set of memory addresses. In some implementations, the conversion of an index to a memory address is performed by linearly combining the memory address with the indices and reflecting the address via the tensor address index 312.

[0063] There is a TTU for each control thread, and there is a control thread for each instruction type (TensorOp and DMAOp) in the computation tile 200. Thus, as described above, there are two sets of TTUs in the computation tile 200: 1) TensorOpTTU 226; and 2) DMAOpTTU 228. In various implementations, the TensorOp control 206 assigns a TensorOpTTU counter 302, a control The limit 308 causes the stride value 304 to be loaded and does not change the register value before the instruction is retired. Each of the two TTUs will need to generate addresses for the following memory address ports in the computational tile 200: 1) a wide memory 212 address port, and 2) a narrow memory 210 with four independently arbitrated banks presented as four address ports.

[0064] As mentioned above, in some implementations, narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine, for each memory cycle, which control unit (e.g., TensorOp control 206 or DMAOp control 208) is permitted to access the shared memory resources of narrow memory 210. In one example, different instruction types (TensorOp and DMAOp) are independent control threads requesting memory access that must be arbitrated. When a particular control thread commits a tensor element to memory, that control thread increments a counter 302 of tensor references committed to memory.

[0065] In one example, when TensorOp control 206 executes an instruction to access a particular element of a tensor, TTU 300 can determine the address of the particular element of the tensor, and control 206 can access storage, such as narrow memory 210, to retrieve data representing the activation value of the particular element. In some implementations, a program can include nested loops, and control 206 can execute instructions to access elements of a two-dimensional array variable within the nested loop according to a current index variable value associated with the nested loop.

[0066] The TTU 300 may simultaneously maintain traversal state for up to X number of TTU rows for a given tensor(s). Each tensor simultaneously resident in the TTU 300 occupies a dedicated hardware tensor control descriptor. The hardware control descriptor can consist of X number of TTU counters 302 per row position, stride 304, and limit registers 308 supporting tensors with up to X number of TTU counters per row dimension. In some implementations, the number of rows and number of counters per row may vary.

[0067] For a given position register, the final memory address is calculated from an addition operation involving adding the position registers together. The base address is incorporated into counter 302. One or more adders are shared for tensor references that reside in the same memory. In one implementation, since there can only be a single load / store on any given port in a cycle, the function of loop nesting control is to ensure that multiple tensor references that reside in the same narrow or wide memory do not have their counters incremented in any given cycle. The use of registers to calculate memory access address values, including determining offset values, is described in more detail in patent application Ser. No. 15 / 014,265, entitled "Matrix Processing Apparatus," filed February 3, 2016, which is expressly incorporated herein by reference in its entirety.

[0068] The following provides template parameters that may be used to instantiate a specialized TTU 300: 1) X number of TTU rows; 2) X number of TTU counters per row; 3) X number of TTU adder units; 4) For each TTU row, indicate the shared adder reference; and 5) For each counter, indicate X counter size [TTU][row][depth]. All TTU registers are architecturally visible. The address of the particular tensor element that needs to be accessed for a computation (i.e., tensor address 312) is the result of the counter addition. When an increment signal is issued from the controlling thread to a TTU row, TTU 300 performs a single cycle operation, incrementing the innermost dimension by that dimension's stride 304. Increment and propagate the rollover through all depths.

[0069] Generally, the TTU 300 determines state associated with one or more tensors. The state may include loop bound values, a current loop index variable value, a dimension multiplier for calculating memory address values, and / or a program counter value for processing branch loop bounds. The TTU 300 may include one or more tensor state elements and an arithmetic logic unit. Each of the tensor state elements may be a storage element, such as a register or any other suitable storage circuit. In some implementations, the tensor state elements may be physically or logically organized into distinct groups.

[0070] FIG. 4 illustrates an exemplary architecture including a narrow memory 210 that broadcasts activations 404 to one or more multiply-accumulate (MAC) operators via an input bus 216. A shift register 404 provides a shifting function whereby activations 404 are sent out one at a time onto the input bus 216 for receipt by one or more MAC operators 215 within the MAC cell 410. In general, a MAC cell 410 including a MAC operator 215 can be defined as a computational cell that computes partial sums and, in some implementations, is configured to write the partial sum data to an output bus 218. As shown, the cell 410 may consist of one or more MAC operators. In one implementation, the number of MAC operators 215 within the MAC cell 410 is referred to as the issue width of the cell. As an example, a dual-issue cell refers to a cell with two MAC operators that can compute the multiplication of two activation values ​​(from the narrow memory 210) by two parameters (from the wide memory 212) and perform an addition between the results of the two multipliers and the current partial sum.

[0071] As mentioned above, input bus 216 is a broadcast bus that provides input activations to MAC operators 215 of a linear unit (i.e., MAC array 214). In some implementations, the same input is shared among all MAC operators 215. The width of input bus 216 must be wide enough to supply the broadcast inputs to a corresponding number of cells for a given MAC array 214. Consider the following example to illustrate the structure of input bus 216. When the number of cells in the linear unit equals four and the activation width equals eight bits, input bus 216 can be configured to provide up to four input activations per cycle. In this example, all cells in MAC array 214 will access only one of the four broadcast activations.

[0072] Based on the TensorOp field settings of instructions received by the computational tile 200, cells of the MAC array 214 may need to perform a computation using the same input activations. This may be referred to as Zout partitioning within a cell of the MAC array 214. Similarly, Zin partitioning within a cell occurs when cells of the MAC array 214 require different activations to perform a computation. In the former case, a single input activation is replicated four times, and four activations read from the narrow memory 210 are broadcast over four cycles. In the latter case, a read of the narrow memory 210 is required every cycle. In the above example, the TensorOp control 206 orchestrates this broadcast method based on the execution of instructions received from the controller 102.

[0073] 5 shows an exemplary architecture including an output bus 218 for providing output activations to the narrow memory units 210 of FIGS. 2 and 4. Generally, each MAC cell 215 of the MAC array 214 in a computational tile 200 computes a different output activation. However, if the output feature depth for the output feature array is less than the number of MAC cells 215 in the computational tile 200, the cells may be grouped to form one or more cell groups. All MAC cells 215 in a cell group compute the same output (i.e., for the output feature map), but each cell computes a subset of the output activations, corresponding to a subset of the Z dimension. 238 provides a control signal to NLU 222 that aggregates these partial sums into a final linear output.

[0074] As mentioned above, the output bus 218 is a pipelined shift register. In various implementations, once the first portion of the tensor computation is finished and the TensorOp control 206 indicates (by executing an instruction) that the partial sums need to be written out, there will be parallel loads of partial sums provided to the output bus 218. The number of parallel loads will correspond to the number of MAC cells in the computation tile 200. The TensorOp control 206 then causes the partial sum amounts to be shifted out and sent through the nonlinear pipeline. In some implementations, there may be situations where not all MAC cells in a tile are actually utilized to perform the computation. In such situations, not all partial sums shifted onto the output bus will be valid. In this example, the TensorOp control 206 may provide control signals to the MAC array 214 to indicate the number of valid cells to be shifted out. The parallel load amount loaded onto the output bus 218 still corresponds to the number of MAC cells in the computation tile, but only valid values ​​will be shifted out and committed to the narrow memory 210.

[0075] FIG. 6 is an example flowchart of a process 600 for performing tensor computations using a neural network (NN) computational tile, such as the computational tile 200 of FIG. 2. Process 600 begins at block 602 by loading enough parameters into an on-chip secondary memory to enable latency below a particular level and throughput above a particular level for a given NN model and architecture. Throughput is the maximum performance achieved in the presence of a large number of requests / inferences. Latency is the minimum time it takes to compute one request. Process 600 continues at block 604, where the narrow memory 210 of the computational tile 200 sends (i.e., broadcasts) activations one by one onto the input activation data bus 216. The activation values ​​are stored in the narrow memory 210. The narrow memory 210 may be a collection of static random access memory (SRAM) banks that allow addressing specific memory locations to access input quantities. The activations read from memory 210 are broadcast via input activation bus 216 to linear cells of a MAC array 214 (i.e., a linear unit) that includes multiple MAC operators 215 and sum registers 220. At block 606 of process 600, the MAC operators 215 of the computational tile 200 each receive two inputs: one input (the activation) is received from input activation bus 216; the other input (the parameter) is received from wide memory 212. Thus, the activations supply one of the inputs of each MAC operator 215, and each MAC operator 215 in a cell of MAC array 214 obtains its second multiplier input from wide memory 212.

[0076] At block 608 of process 600, the MAC array 214 of the computational tile 200 performs tensor computations, including dot product computations, based on elements of the data array structure accessed from memory. The wide memory 212 may have a width in bits equal to the width of the linear unit (e.g., 32 bits). Thus, the linear unit (LU) is a SIMD vector arithmetic logic unit (ALU) unit that receives data from the vector memory (i.e., the wide memory 212). In some implementations, the MAC operator 215 may also obtain an accumulator input (partial sum) from the wide memory 212. In some implementations, there is time sharing for the wide memory 212 ports for reading and / or writing for two different operands (parameters and partial sums). Generally, to optimize area, the wide memory 212 may have a limited number of ports. As a result, it is possible to simultaneously read an operand (e.g., parameter) from the wide memory 212 and write an operand (e.g., If a particular operand (e.g., a partial sum) needs to be written, the pipeline associated with that particular operand may stall.

[0077] In block 610, the computational cell (having the MAC operator 215 and sum register 220) of the computational tile 200 generates at least one output activation based on the multiplication operation performed by the MAC / computational cell. The result of the MAC cell operation includes either a partial sum that is written back to memory (during a partial sum arithmetic operation) or an output activation that is sent to the output bus 218. The NLU 222 of the computational tile 200 may apply a nonlinear activation function to the output activations and write the activations to the narrow memory 210. In some implementations, the output bus 218 is a shift register that can accumulate parallel loads of results / output activations from the MAC operator 215 and shift them out one at a time for the application of the nonlinear function and write operation to the narrow memory 210 of the same tile.

[0078] The embodiments described in this specification take advantage of two key insights: 1) the bottleneck of most existing neural network (NN) accelerators is the memory bandwidth required to load the NN weights (also known as parameters); and 2) even if the number of parameters in a generative model is large, i.e., a few KB to a few GB, as opposed to a few MB to a few hundred MB for most models, these numbers are within the realm of what can be achieved with hardware including on-chip memory, e.g., using memory distributed among tiles.

[0079] Adding a large memory, e.g., a cache or scratchpad, on the die to contain all the parameters is not enough. The goal of overcoming memory bandwidth limitations is to scale out the performance of the architecture, which means increasing the number of operators, usually MACs. However, to achieve high performance, we must be able to deliver these operators along with the parameters every cycle. It is also important to understand "performance" not only as throughput but also as latency. This is true for many user-facing applications.

[0080] In other words, in a neural network (NN) accelerator, loading parameters into one layer at a time is very costly. If parameters can be preloaded on-chip, they only need to be loaded for runtime activation. In other words, embodiments include large on-chip memory.

[0081] The embodiments described herein affect the operation and design of neural network (NN) accelerators. The embodiments address one of the significant limitations of neural network accelerator design: the combination of low latency and high throughput. We assume that the NN accelerator represents the entire neural network. The input is some data, e.g., a small image or audio. The NN accelerator executes layers one by one. The cost in performance and energy is loading the parameters of each layer one by one. The NN accelerator loads the parameters of one layer, performs the calculation, saves the output of the layer, and then loads the parameters of the next layer. This process is where most of the memory bandwidth is consumed.

[0082] By having the parameters reside on-chip, latency is dramatically reduced, given high throughput and the specific NN model and architecture. The accelerator only needs to load a few bytes of audio, after which it can be very fast. The accelerator does not require high memory bandwidth and energy is lower.

[0083] The von Neumann model, where memory is loaded into the CPU from memory, is a common architecture. Such a traditional von Neumann-like architecture, where memory resides on one end of the die and computational operators reside on the other end of the die, is impractical, if not impossible, for a large number of operators, implying a large number of wires to transfer data from memory (or memory banks) to the operators. Instead, embodiments of the present invention take advantage of the memory locality property of neural network computations and organize the architecture in a tiled configuration with memory distributed across tiles (as shown in Figures 2 and 8).

[0084] Because NNs are large, but not gigantic, one can approach effectively fitting all parameters for one or a few NNs on a single chip. NN accelerators are moving to self-contained architectures. With tiled architectures, memory can be partitioned within the chip. Instead of having one large SRAM in one corner of the chip, embodiments allocate appropriate SRAM per tile, also avoiding on-chip bandwidth issues. In certain embodiments, a wide memory for each tile contains the parameters, and a wide ring (of roughly the same width as the wide memory) provides the wide memory with high bandwidth. The embodiments described in this specification contemplate architectural variations. Depending on the characteristics of the NN layer, embodiments can have at least two NN architectures:

[0085] In neural networks that are mostly composed of fully connected layers, parameter reuse between layers is rare. Neurons in a fully connected layer have full connections to all activations in the previous layer. Consider a fully connected neural network where parameters are not reused (consider the absence of batch processing, e.g., real-time applications). If all parameters are not contained in wide memory, they must be fetched from external memory over a ring bus. In that case, the performance of the overall design is limited by the external memory bandwidth. If all parameters reside in wide memory, all operators can be supplied with parameters every cycle without needing to access external memory, maximizing performance. Instead of utilizing external memory only to fetch parameters, the embodiments described herein keep the parameters resident in wide memory.

[0086] As an example, consider a model with 50M parameters in the fully connected layer (or, for simplicity's sake, 50MB). Consider an accelerator with 16384 MACs running at 1GHz. Consider the input to the model to be 16KB. All values ​​are reasonable for the current application. The maximum performance corresponds to running the model in 50×10^6 / (16384×10^9) = 3.05×10^-6 seconds. This corresponds to a memory bandwidth of (50×10^6 + 16,384) / (3.05×10^-6) = 16.40TB / s. As a point of comparison, typical DRAM chips are around 10GB / s, while state-of-the-art high-bandwidth memory (HBM) is around 256GB / s.

[0087] Convolutional neural networks pass parameters from one tile to another. For neural networks consisting mostly of convolutional layers, where parameters are reused between neurons (also known as activations), the memory bandwidth requirements are not very high, but are generally higher than typical external memories. The ring bandwidth may be sufficient to load the parameters into the tiles, but it is connected to a large on-die memory as wide as the ring. In other words, for each inference, the tile needs to access / load all the parameters of the model. This is true for any model. The only difference with fully connected layers in neural networks is that each parameter is used only once during one inference. For convolutional layers, parameters can be reused multiple times within the layer. Used.

[0088] As an example, consider a model with 50M parameters in its convolutional layers. Some of these layers may be very small, while others may be large, resulting in variable parameter reuse. A reasonable average across models is a maximum of 100 reuses per parameter. Thus, using the same theory as above, the bandwidth requirement drops to 16.40TB / s / 100 ~ 164GB / s. However, the bandwidth requirement for cost-effective DRAM remains high. However, in the above architecture, a wide ring of 164 x 8 = 1312 bits connected to a large memory of the same width can serve the tiles at a reasonable speed.

[0089] Consider a first embodiment where memory is sufficient to contain all parameters for layers of a fully connected model. If all tiles operate simultaneously on a layer, the parameters must be distributed among the tiles. The embodiment splits the output neurons / activations of each layer among the tiles. At runtime, each tile processes a subset of the layer, computes the corresponding partial sums, and passes them on to its neighbors. That is, the partial sums go around the ring, and after going around all the tiles, the tile produces the final sum.

[0090] A second embodiment involves preloading / caching the same subset of (currently used) parameters for all tiles, since the tiles utilize the same parameters simultaneously. At runtime, the parameters (subset) that are not partial activation sums are rotated around the ring.

[0091] The number of tiles is a scale factor. Robust scaling can be achieved by improving latency and throughput without increasing memory requirements using the embodiments described herein. However, processing tiles together to increase computational power increases memory requirements and the number of activations required. Batch processing scaling is difficult without increasing memory bandwidth over traditional memory options. Batch processing is often associated with real-time applications and has associated latency and throughput requirements.

[0092] It is worth noting that having parameters in cache is different from having them in wide memory as part of the tiling. Arranging the tiles as described herein provides compiled-style locality of reference. For example, placing a fully connected model next to SRAM provides greater internal bandwidth compared to a cached model.

[0093] The embodiments described herein operate faster than conventional neural network accelerators. Certain embodiments have many more operators, and the accelerator requires more internal bandwidth to supply the operators. To address this, the architecture must distribute memory and centralize parameters on the accelerator.

[0094] Currently, the largest chips are approximately 650 square millimeters. Therefore, there is a limit to the amount of SRAM that can be packed on a chip. Embodiments include utilizing higher density memory, including utilizing 3D stacking, within given space constraints.

[0095] The embodiments described in this specification apply to the inference (post-training) mode and the training mode.

[0096] An additional consideration is another level of hierarchy: there is the memory hierarchy, which usually includes the register file. There is also a file hierarchy. Parameters are loaded into registers and the registers are reused, allowing for increased memory bandwidth. There is memory bandwidth in the register file and in the memory. This means there is further cost savings, i.e., less wiring from memory to the register file and from the register file to the computation. The embodiments described herein can reduce the wiring costs associated with the register file. Parameters are addressed by consuming them directly from SRAM, i.e., the memory feeds the ALU directly.

[0097] FIG. 7 shows a block diagram of another embodiment of a computing system 700 for accelerating tensor computations associated with deep neural networks (DNNs). The system 700 generally includes a controller / uncore (UNCORE) 702, a memory interface 708, and multiple tiles, including a first tile set 112 and a second tile set 714. The controller 702 generally includes a data memory 704, an instruction memory 706, and at least one processor configured to execute one or more instructions encoded on a computer-readable storage medium. The instruction memory 706 can store one or more machine-readable instructions executable by the one or more processors of the controller 702. The data memory 704 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within the system 700.

[0098] The controller 702 is configured to execute one or more instructions related to tensor computations in the system 700, including instructions stored in an instruction memory 706. In some implementations, the data memory 704 and the instruction memory 706 are volatile memory units or units. In other implementations, the data memory 704 and the instruction memory 706 are non-volatile memory units or units. The data memory 704 and the instruction memory 706 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In various implementations, the controller 702 may be referred to as or be called a core manager 702.

[0099] The memory interface 708 receives instructions and data parameters from the I / O link and provides the instructions and parameters to the controller 702. Generally, instructions may be provided to one or more devices in the system 100 via an instruction bus (an instruction bus between the controller and the tiles is not shown), and parameters may be provided to one or more devices in the system 700 via a ring bus 728. In some implementations, instructions are initially received by the controller 702 from the host interface 708 and stored in the instruction memory 706 for later execution by the controller 702.

[0100] The ring bus 728 provides communications that originate from the controller 702 and couple via bus data paths that connect the tiles 712, 714 in a ring back to the controller 702. In various implementations, the ring bus 728 generally connects or couples all of the wide memory units in all of the tiles 712, 714. Thus, the payload width of the ring bus 728 corresponds to the width of the wide memory units located within each tile of the tile set 712, 714. As mentioned above, the ring bus 728 also includes a bitmap header that indicates which tiles should consume payload data, including commands or parameters, communicated over the ring bus 728.

[0101] Data (i.e., payload) received at a particular tile via ring bus 728 Regarding the bitmap, in response to receiving the information, each tile zeros (i.e., clears) the position data indicated in the bitmap header specific to the receiving tile before forwarding the data to another tile. Thus, when the header bitmap has no remaining bits set data indicating the particular tile that is to receive the payload, forwarding of the payload to another tile will stop. Payload data generally refers to the activations and weights used by one or more tiles during tensor computations that are performed based on the execution of deeply nested loops.

[0102] In some implementations, the controller 702 may be described as part of the ring bus 728. In one example, for a DMAOp instruction executed within a particular tile, the controller 702 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 702 may also commit the payload data to one or more wide memory units of the tile if required by an instruction in the bitmap header. The addresses of the one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.

[0103] In various implementations, each tile in the tile sets 712, 714 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over the ring bus 728 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives data, writes it to one or more wide memory units within the tile, and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data over the ring bus 728, there is typically only one producer / master of data on the ring bus 728 at any given time. The DMAOp instruction execution order (e.g., a FIFO control scheme) among all tiles will ensure that there is only one producer / master of data on the ring bus 728 at a given time.

[0104] In some implementations, the controller 702 uses a synchronization flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 728 at a given time. In one example, each write by a tile to a ring output will trigger an increment of a corresponding synchronization flag count. The controller 702 can examine the payload data to determine the number of data chunks or segments that comprise the payload. The controller 702 then monitors execution by a tile to ensure that the expected number of data segments is transferred and / or consumed by that tile before another tile executes in master mode.

[0105] An exception to ensuring that there is only one producer / master of data on the ring bus 728 at a given time occurs when there are local multicast groups connected via the ring bus 728 that do not have overlapping areas on the ring bus 728. A key requirement of this dual-master multicast method is to ensure that different multicast groups cannot see each other's data packets, as this could result in packet duplication and cause one or more data calculation errors.

[0106] In contrast to Figure 1, the tiles in Figure 7 form a rectangular pattern themselves, while being connected in a zigzag or serpentine fashion by a ring bus. In the illustrated embodiment, the accelerators are made up of 8 and 16 tiles, respectively. In alternative embodiments, the accelerators may contain even more tiles.

[0107] FIG. 8 is a simplified diagram of the computational tile of FIG. 9 is an example flowchart of a process 900 for performing tensor computations using a neural network (NN) computational tile, such as computational tile 200 of FIG. 2. Process 900 begins at block 902, e.g., by loading NN weight parameters into an NN accelerator prior to execution. Process 900 continues at block 904 with processing inputs to the accelerator without accessing neural network weight parameters substantially external to the accelerator. At block 906, the process generates at least one output activation based on processing the inputs to the accelerator.

[0108] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Alternatively, or in addition, the program instructions may be encoded on an artificially generated, propagated signal, such as, for example, a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0109] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be realized by, special purpose logic circuitry such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0110] A processor suitable for executing a computer program may be based, by way of example, on a general-purpose or special-purpose microprocessor, or both, or on any type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operably coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices.

[0111] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0112] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of possible features specific to particular embodiments of particular inventions. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination, and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0113] Similarly, while operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, or that all of the shown operations need to be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. An accelerator for accelerating tensor computations, comprising: a. a computation unit; b. a first memory bank for storing at least one of the input activations or the output activations; and c) a second memory bank for storing neural network parameters used in performing calculations, the second memory bank configured to store a sufficient amount of the neural network parameters on the computation unit to enable, for a given NN model and architecture, latency below a particular level and throughput above a particular level, the accelerator further comprising: d. at least one cell containing at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs a calculation; e. a first traversal unit in data communication with at least said first memory bank, said first traversal unit configured to provide control signals to said first memory bank to provide input activations to a data bus accessible by said MAC operator; f. the accelerator performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between the input activation received from the data bus and a parameter received from the second memory bank.

2. The accelerator of claim 1 , wherein the second memory bank is configured to store more than 100,000 parameters.

3. The accelerator of claim 1 , wherein the second memory bank is configured to store more than 1,000,000 parameters.

4. The accelerator of claim 1 , wherein the second memory bank is configured to store greater than 100,000,000 parameters.

5. The accelerator of claim 1 , wherein the second memory bank includes SRAM.

6. The accelerator includes a plurality of computational units, each of which: a. a first memory bank for storing at least one of input activations or output activations; and b. a second memory bank for storing neural network parameters used in performing the calculations, said second memory bank configured to store a sufficient amount of said neural network parameters on said computation unit to enable, for a given NN model and architecture, latency below a particular level and throughput above a particular level, each computation unit further comprising: c. at least one cell containing at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs a calculation; d. a first traversal unit in data communication with at least said first memory bank, said first traversal unit configured to provide control signals to said first memory bank to provide input activations to a data bus accessible by said MAC operator; e. the computation unit performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and dependent, in part, on the input activations received from the data bus and the second memory bank. The accelerator of claim 1 , further comprising a multiplication operation with a parameter received from

7. The accelerator of claim 1 , wherein the second memory bank includes 3D SRAM.

8. 1. A computer-implemented method for accelerating tensor computations, comprising: a) sending a first input activation by a first memory bank in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit, the method further comprising: receiving one or more parameters from a second memory bank for storing neural network parameters used in performing calculations by said at least one cell, said second memory bank storing a sufficient amount of said neural network parameters on said computation unit to enable, for a given NN model and architecture, a latency below a particular threshold with a throughput above a particular threshold, said at least one cell including at least one multiply-accumulate ("MAC") operator, said method further comprising: c) performing, by said MAC operator, one or more calculations relating to at least one element of a data array, said one or more calculations including, in part, a multiplication operation between at least said first input activation accessed from said data bus and at least one parameter received from said second memory bank.

9. The method of claim 8 , wherein the second memory bank is configured to store greater than 100,000 parameters.

10. The method of claim 8 , wherein the second memory bank is configured to store greater than 1,000,000 parameters.

11. The method of claim 8 , wherein the second memory bank comprises SRAM.

12. The method of claim 8 , wherein the second memory bank comprises 3D SRAM.

13. 9. The method of claim 8, further comprising loading the neural network parameters into the second memory for use in performing calculations.

14. 1. A computer-implemented method for accelerating tensor computations, comprising: a. sending a first input activation by a first memory bank in response to said first memory bank receiving a control signal, said first input activation being sent over a data bus, said method further comprising: b. receiving one or more parameters from a second memory bank for storing neural network parameters used in performing calculations by at least one multiply-accumulate ("MAC") operator, said second memory bank storing a sufficient amount of said neural network parameters on said computation unit to enable a latency below a certain threshold and a throughput above a certain threshold for a given NN model and architecture, said method further comprising: c) performing, by said MAC operator, one or more calculations relating to at least one element of a data array, said one or more calculations comprising, in part, a multiplication operation between at least said first input activation accessed from said data bus and at least one parameter received from said second memory bank.

15. The method of claim 14 , wherein the second memory bank is configured to store greater than 100,000 parameters.

16. The method of claim 14 , wherein the second memory bank is configured to store greater than 1,000,000 parameters.

17. 15. The method of claim 14, wherein the second memory bank is configured to store greater than 100,000,000 parameters.

18. 15. The method of claim 14, wherein the second memory bank comprises SRAM.

19. The method of claim 14 , wherein the second memory bank comprises 3D SRAM.

20. 15. The method of claim 14, further comprising loading the neural network parameters into the second memory for use in performing calculations.

Images