Neural network accelerator using on-chip resident parameter
By storing neural network parameters on-chip, the neural network accelerators overcome memory bandwidth limitations, achieving improved performance and reduced energy consumption.
Patent Information
- Application Number
- JP2024144252
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2017-08-11
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2038-08-09
AI Technical Summary
Current neural network accelerators face limitations due to high memory bandwidth requirements and energy consumption, as they rely on external memory for parameter storage, which is inefficient and costly in terms of performance and energy.
Storing neural network parameters on-chip within the accelerator, eliminating the need for frequent external memory access by keeping all parameters resident on the chip, thereby reducing memory bandwidth demands and energy consumption.
This approach enhances performance by allowing all multiply-accumulate operators to be supplied with parameters every cycle, reducing delays and energy usage, while maintaining high throughput and low latency.
Smart Images

Figure 2024164192000001_ABST
Abstract
Description
[Background technology]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 U.S.C. §119(e) of the filing date of U.S. patent application Ser. No. 62 / 544,171, filed Aug. 11, 2017, for “Neural Network Accelerator with Parameters Resident on Chip,” the entire contents of which are incorporated herein by reference.
[0002] background This specification generally relates to neural network (NN) computation tiles for computing deep neural network ("DNN") layers. Summary of the Invention
[0003] overview In general, one innovative aspect of the subject matter described in this specification can be implemented in storing the parameters of a neural network on an accelerator. A neural network differs from a typical computational workload in that its working set, i.e., the total amount of storage required for the entire computation, is actually limited. Roughly speaking, this working set corresponds to a number of parameters, usually hundreds of thousands to billions. This amount of storage is consistent with existing hardware storage technology.
[0004] Despite these facts, current accelerators include local storage for parameters that are passed through, i.e., parameters do not reside permanently on-chip; rather, parameters flow from external memory with each new inference.
[0005] Therefore, external memory bandwidth is a key limitation of all neural network (NN) accelerators. The embodiments described herein replace temporary local storage for parameters with on-chip storage for parameters. That is, the embodiments keep all parameters of the NN resident within the accelerator and do not flush them from external memory.
[0006] Advantages of storing parameters on-chip include overcoming the performance limitations of NN accelerators; dramatically facilitating an increase in the number of multiply-accumulate ("MAC") operators since the performance limitations are overcome; and providing a lower-power neural network accelerator since external memory accesses typically require at least an order of magnitude more energy than local memory accesses.
[0007] In a particular embodiment, the accelerator comprises a computation unit comprising a first memory bank for storing input activations or output activations and a second memory bank for storing neural network parameters used in performing the computation, the second memory bank configured to store a sufficient amount of the neural network parameters on the computation unit to enable a latency below a particular level and a throughput above a particular level for a given NN model and architecture, the computation unit further comprising at least one cell including at least one MAC operator that receives the parameters from the second memory bank and performs the computation, and a first traversal unit in data communication with at least the first memory bank, the first traversal unit being configured to: The accelerator is configured to provide control signals to one memory bank to provide input activations to a data bus accessible by the MAC operator. The accelerator performs one or more computations related to at least one element of the data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between the input activations received from the data bus and the parameters received from the second memory bank. If the parameter storage is sufficient to hold all the neural network parameters, the performance of the accelerator is not determined by the memory bandwidth. In that case, it is possible to provide all the MACs with the parameters every cycle.
[0008] Another innovative aspect of the subject matter described herein may be embodied in a computer-implemented method for accelerating tensor computations, the computer-implemented method including sending a first input activation by a first memory bank in response to the first memory bank receiving a control signal, the first input activation being sent by a data bus, the method further including receiving, by at least one MAC operator, one or more parameters from a second memory bank for storing neural network parameters used in performing the computation, the second memory bank storing a sufficient amount of the neural network parameters on the computation unit to enable a latency below a particular threshold and a throughput above a particular threshold for a given NN model and architecture, the method further including performing, by the MAC operator, one or more computations related to at least one element of a data array, the one or more computations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and the at least one parameter received from the second memory bank.
[0009] Another innovative aspect of the subject matter described herein may be embodied in a method for accelerating computations that includes, prior to execution, loading neural network weight parameters into a neural network accelerator, and, at execution, processing inputs to the accelerator without substantially accessing the neural network weight parameters external to the accelerator. If the number of neural network weight parameters is too large to fit on a single accelerator die, the method may further include loading the neural network weight parameters into multiple densely connected accelerator dies, and processing inputs to the accelerator dies without substantially accessing the neural network weight parameters external to the densely connected accelerator dies.
[0010] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages: Using registers to track memory address values allows a program to iterate through deeply nested loops with one instruction. Tensors accessible from narrow and wide memory units in a single computational tile are traversed based on memory address values retrieved from registers. The memory address values correspond to elements of the tensor. Tensor computations occur in individual computational tiles based on the execution of deep loop nests. Computations can be distributed across multiple tiles. Computational efficiency is improved and accelerated based on distributing tensor computations of a multi-layer neural network across several computational tiles. Tensors can be traversed and tensor computations can be performed with fewer instructions.
[0011] The embodiments described herein impact the operation and design of Neural Network (NN) accelerators. The embodiments address one of the prominent limitations of neural network accelerator design: the combination of low latency and high throughput. By having parameters reside on-chip, it is possible to achieve low latency given the high throughput as well as specific NN models and architectures. is dramatically reduced. The accelerator does not require high memory bandwidth and the energy is lower.
[0012] Tiling as described herein provides compiled locality. For example, placing the fully connected model next to the SRAM provides more internal bandwidth compared to cache models. The embodiments described herein run faster than convolutional neural network accelerators. Certain embodiments have many more operators. To supply the operators, the accelerator requires more internal bandwidth. To address this, the architecture must distribute memory and aggregate parameters on the accelerator.
[0013] The subject matter described in this specification can also be implemented in certain embodiments to achieve other advantages. For example, by employing a memory hierarchy that combines narrow, low-bandwidth memory with high-bandwidth, wide memory, high utilization of MAC operators can be achieved for DNN layers of very different dimensions and locality of reference. The narrow, low-bandwidth memory allows for flexible addressing to traverse multidimensional arrays in any order.
[0014] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or combinations thereof that are installed on the systems and that, when operated, cause the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the systems to perform the actions.
[0015] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0016] [Figure 1] FIG. 1 is a block diagram of an example computing system. [Diagram 2] 1 illustrates an exemplary neural network computation tile. [Diagram 3] 1 illustrates an exemplary tensor traversal unit (TTU) structure. [Figure 4] 1 illustrates an example architecture that includes a narrow memory unit that provides input activations to one or more multiply-accumulate (MAC) operators. [Diagram 5] 5 illustrates an exemplary architecture including an output bus that provides output activation to the narrow memory units of FIGS. [Figure 6] 3 is an example flow chart of a process for performing tensor computations using the neural network computational tile of FIG. 2. [Figure 7] 1 is another embodiment of a computing system. [Figure 8] 1 is another embodiment of a neural network computation tile. [Figure 9]1 is an example flowchart of a process for accelerating computations by loading neural network weight parameters into a neural network accelerator. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description The subject matter described in this specification relates to a method for accelerating computations. includes loading the neural network weight parameters into the neural network accelerator prior to execution, and processing inputs to the accelerator at execution without substantially accessing the neural network weight parameters external to the accelerator. If the number of neural network weight parameters is too large to fit on a single accelerator die, the method may further include loading the neural network weight parameters into multiple densely connected accelerator dies, and processing inputs to the accelerator dies without substantially accessing the neural network weight parameters external to the densely connected accelerator dies.
[0018] The subject matter described herein also relates to a hardware computing system including a plurality of computing units configured to accelerate machine learning inference workloads of neural network layers. Each computing unit of the hardware computing system is self-contained and can independently perform the computations required by a given layer of a multi-layer neural network. This specification generally relates to a neural network (NN) computing tile for computing deep neural network ("DNN") layers with on-chip resident parameters to enable latency below a certain level and throughput above a certain level for a given NN model and architecture.
[0019] A neural network with multiple layers can be used to compute an inference. For example, given an input, a neural network can compute an inference for that input. The neural network computes this inference by processing the input through each layer of the neural network. In particular, each layer of the neural network has its own set of weights. Each layer receives an input and processes the input according to the set of weights for that layer to generate an output.
[0020] Thus, to compute an inference from a received input, a neural network receives the input and processes it through each neural network layer to generate an inference, with the output from one neural network layer being provided as the input to the next neural network layer. The data inputs or outputs for a neural network layer, e.g., the inputs to the neural network, or the outputs of the layer below it in the sequence, can be referred to as the activations of that layer.
[0021] In some implementations, the layers of a neural network are arranged in a sequence. In other implementations, the layers are arranged in a directed graph, meaning that any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be configured such that the output of one layer can be sent back as input to a previous layer.
[0022] The hardware computing system described herein can perform the computation of a neural network layer by distributing tensor computations across multiple computation tiles. The computation process performed within the neural network layer can include multiplication of input tensors including input activations with parameter tensors including weights. The computation includes multiplying the input activations with the weights in one or more cycles and performing an accumulation of the products over many cycles.
[0023] Tensors are multi-dimensional geometric objects, and exemplary multi-dimensional geometric objects include matrices and data arrays. In general, processing is performed by computation tiles to perform tensor computations by processing nested loops to traverse an N-dimensional tensor. In one exemplary computation process, each loop traverses an N-dimensional tensor. A tensor traversal unit may be responsible for traversing a particular dimension of a tensor. For a given tensor construct, a computational tile may need access to elements of that tensor to perform multiple dot product calculations associated with that tensor. A computation occurs when input activations provided by narrow memory structures are multiplied with parameters or weights provided by wide memory structures. Since tensors are stored in memory, a set of tensor indices may require conversion to a set of memory addresses. In general, a tensor traversal unit of a computational tile performs control operations that give the indexes of each dimension associated with a tensor and the order in which the index elements are traversed to perform the computation. The tensor computation is completed when the multiplication results are written to an output bus and stored in memory.
[0024] FIG. 1 illustrates a block diagram of an exemplary computing system 100 for accelerating tensor computations associated with deep neural networks (DNNs). The system 100 generally includes a controller 102, a host interface 108, an input / output (I / O) link 110, a number of tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a data bus (shown for clarity but not included in the system 100) identified in a bus map 118. The controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. The instruction memory 106 can store one or more machine-readable instructions executable by the one or more processors of the controller 102. The data memory 104 may be any of a variety of data storage media for storing and subsequently accessing various data associated with computations occurring within the system 100.
[0025] The controller 102 is configured to execute one or more instructions related to tensor computations in the system 100, including instructions stored in the instruction memory 106. In some implementations, the data memory 104 and the instruction memory 106 are volatile memory unit(s). In some other implementations, the data memory 104 and the instruction memory 106 are non-volatile memory unit(s). The data memory 104 and the instruction memory 106 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In various implementations, the controller 102 may be referred to or called as a core manager 102.
[0026] As shown, host interface 108 is coupled to I / O link 110, controller 102, and classifier portion 116. Host interface 108 receives instructions and data parameters from I / O link 110 and provides instructions and parameters to controller 102. In general, instructions may be provided to one or more devices in system 100 via instruction bus 124 (described below), and parameters may be provided to one or more devices in system 100 via ring bus 128 (described below). In some implementations, instructions are initially received by controller 102 from host interface 118 and stored in instruction memory 106 for later execution by controller 102.
[0027] The classifier portion 116 is similarly coupled to the controller 102 and to tile 7 of the second tile set 114. In some implementations, the classifier portion 116 is implemented as a separate tile within the system 100. In alternative implementations, the classifier portion 116 is disposed or located within the controller 102 as a subcircuit or subdevice of the controller 102. The classifier portion 116 generally comprises an accumulated eigenvalue that is received as the output of a fully connected layer. The fully connected layer is configured to perform one or more functions on the pre-activation values obtained by the pre-activation process. The fully connected layer may be divided across tiles in tile sets 112 and 114. Thus, each tile is configured to generate a subset of pre-activation values (i.e., linear outputs) that may be stored in the tile's memory unit. The classification result bus 120 provides a data path from the classifier portion 116 to the controller 102. Data including post-function values (i.e., results) are provided from the classifier portion 116 to the controller 102 via the classification result bus 120.
[0028] The bus map 118 illustrates a data bus providing one or more interconnected data communication paths between tiles of the first tile set 112 and the second tile set 114. The bus map 118 provides an explanation of the symbols used to identify the classification result bus 120, the CSR / master bus 122, the instruction bus 124, the mesh bus 126, and the ring bus 128 as shown in FIG. 1. In general, tiles are the core components within the accelerator architecture of the system 100 and are the focal points for tensor computations occurring within the system. Each tile is an individual computational unit that is correlated with other tiles in the system and can accelerate computations across one or more layers of a multi-layer neural network. While tiles within the tile sets 112, 114 can share the execution of tensor computations associated with a given instruction, individual computational units are self-contained computational components that are configured to perform a subset of tensor computations independently of other corresponding tiles within the tile sets 112, 114.
[0029] The CSR bus 122 is a single-master, multiple-slave bus that allows the controller 102 to send one or more instructions that set program configurations and read status registers associated with one or more tiles. The CSR bus 122 can be connected in a single daisy-chain configuration with one master bus segment and multiple slave bus segments. As shown in FIG. 1, the CSR bus 122 provides communication coupling the tiles of the tile sets 112, 114 and the controller 102 via a bus data path that connects them in a ring to the host interface 110. In some implementations, the host interface 110 is the single master of the CSR bus ring, and the entire CSR bus address space is memory-mapped into a memory space within the host interface 110.
[0030] The CSR bus 122 may be used by the host interface 110 to perform one or more operations including, for example, programming memory buffer pointers in the controller 102 to allow the controller 102 to begin fetching instructions from the instruction memory 106, updating / programming various tile settings (e.g., coefficient tables for polynomial approximation calculations) that remain static for one or more calculations, and / or loading / reloading firmware for the classifier portion 116. In one example, a firmware reload may include a new function to be applied to the linear output (i.e., pre-activation values). Thus, every slave that has access to the CSR bus 122 will have a distinct node identifier (node ID) that is tied to and identifies that slave. The node ID is part of the instruction address and is used, inspected, or otherwise consulted by the CSR slave (i.e., the controller 102, tiles 112, 114, and classifier 116) to determine whether a CSR packet is addressed to the slave.
[0031] In some implementations, one or more commands can be sent by the host interface 102 through the controller 102. A command can be, for example, 32 bits wide, with the first 7 bits including header information indicating the command address / destination that is to receive and execute the command. The first 7 bits of the header can include a data parameter that represents a particular node ID. Thus, a slave on the CSR bus ring (e.g., each tile) can inspect the command header to determine whether a request by the master (host interface 110) is a header. The inspection tile can determine whether the incoming CSR command packet is addressed to a tile that inspects the header. If the node ID in the header does not indicate that the destination is an inspection tile, the inspection tile copies the incoming CSR command packet to a CSR bus input connected to the next tile for inspection by the next tile.
[0032] The instruction bus 124 provides communications that originate from the controller 102 and couple the tiles in the tile sets 112, 114 over a bus data path that, similar to the CSR bus 122, connects the tiles in a ring back to the controller 102. In one implementation, the controller 102 broadcasts one or more instructions over the instruction bus 124. The instructions broadcast by the controller 102 may differ from the instructions provided over the CSR bus 122. However, the manner in which tiles receive and / or consume or execute instructions received over the bus 124 may be similar to the process for executing instructions received over the CSR bus 122.
[0033] In one example, the header of the instruction (i.e., a bitmap) indicates to the receiving tile that it needs to consume a particular instruction based on the bitmap associated with that instruction. The bitmap may have a particular width defined in terms of bits. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of the instruction bus 124 may be configured to be smaller than the size / width of the instruction. Thus, in such a configuration, the transmission of instructions occurs over several cycles, and the bus stops of the instruction bus 124 have decoders to place the instructions received at that tile in the appropriate target instruction buffer associated with that tile.
[0034] As described further below, tiles in a tile set 112, 114 are generally configured to support two broad categories of instructions. The two broad categories are also referred to as instruction types. The instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, a DMAOp instruction has one or more specializations that are allowed to be simultaneous. The one or more specializations may be referred to as DMAOp instruction subtypes or opcodes. In some cases, every unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a particular tile.
[0035] At a particular one of the tiles 112, 114, the bus stop associated with the instruction bus 124 consults the header bitmap to determine the instruction type / subtype. The instruction may be received by the tile and subsequently written to the tile's instruction buffer prior to execution of the instruction by the tile. The tile's instruction buffer to which the instruction is written may be determined by the instruction's type and subtype indicators / fields. The instruction buffer may include a first-in, first-out (FIFO) control scheme that prioritizes consumption of one or more associated instructions. Thus, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which they arrived on the instruction bus.
[0036] The different instruction buffers within a tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As noted above, instruction types include TensorOp instructions and DMAOp instructions. With respect to DMAOp instructions, instruction subtypes (indicating "write to" buffer locations) include: 1) mesh inbound instruction buffer; 2) mesh outbound instruction buffer; 3) narrow-wide DMA instruction buffer; 4) wide-narrow DMA instruction buffer; and 5) ring bus DMA instruction buffer. These buffer locations are described in more detail below with reference to FIG. 2. Wide and narrow designations are used throughout this specification and generally refer to the approximate width size (in bits / bytes) of one or more memory units. As used herein, "narrow" refers to a buffer that is less than 16 bits each. "Wide" may refer to one or more memory units having a size or width between 16 and 64 bits, and "wide" may refer to one or more memory units each having a size or width between 16 and 64 bits.
[0037] The mesh bus 126 provides a data communication path that is distinct from the CSR bus 122, the instruction bus 124, and the ring bus 128 (described below). As shown in FIG. 1, the mesh bus 126 provides a communication path that couples or connects each tile to its corresponding neighboring tiles in both the X and Y dimensions. In various implementations, the mesh bus 126 can be used to transport input activation amounts between one or more narrow memory units in adjacent tiles. As shown, the mesh bus 126 does not allow input activation data to be directly transferred to non-adjacent tiles.
[0038] In various implementations, the mesh bus 126 and the various tiles connected via the mesh bus 126 may have the following configurations: The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge, non-corner tiles have four inbound ports and four outbound ports. In general, in an example N×N tile layout, edge tiles are tiles that have only three neighboring tiles and corner tiles are tiles that have two neighboring tiles. With regard to the data flow methodology over the mesh bus 126, in general, all input activations arriving over the mesh bus 126 for a particular tile must be committed to one or more narrow memory units of that tile. Additionally, for tile configurations with fewer than four inbound ports, a DMAOp instruction may write zero values to locations in the narrow memory of the tile instead of waiting for data on nonexistent input ports. Similarly, for tile configurations with fewer than four outbound ports, DMAOp instructions do not perform narrow memory reads and port writes associated with transfers to non-existent ports.
[0039] In some implementations, the location or address of the narrow memory unit(s) to which a particular input activation will be written or read will be generated by a tensor traversal unit (hereinafter "TTU") based on inbound / outbound DMAOps provided over mesh bus 126. The inbound and outbound DMAOps may be executed simultaneously, with any necessary synchronization being managed by a synchronization flag control scheme managed by controller 102. The TTU is described in further detail below with reference to Figures 2 and 3.
[0040] The ring bus 128 provides communication that originates from the controller 102 and couples the tiles 112, 114 over bus data paths that connect them in a ring back to the controller 102, similar to the CSR bus 122 and the instruction bus 124. In various implementations, the ring bus 128 generally connects or couples all of the wide memory units (described in more detail below with reference to FIG. 2) within all of the tiles 112, 114. Thus, the payload width of the ring bus 128 corresponds to the width of the wide memory units located within each tile of the tile set 112, 114. As mentioned above, the ring bus 128 also includes a bitmap header that indicates which tiles should consume payload data, including instructions or parameters, communicated over the ring bus 128.
[0041] With respect to data (i.e., payload) received at a particular tile over the ring bus 128, in response to receiving the information, each tile zeroes (i.e., clears) the position data indicated in the bitmap header specific to the receiving tile before forwarding the data to another tile. Thus, the header bitmap indicates the location of the tile that is to receive the payload. When a tile does not have any remaining bits set data indicating a particular tile to be processed, the transfer of payload to another tile will stop. The payload data generally refers to the activations and weights used by one or more tiles during tensor computations that are performed based on the execution of deeply nested loops.
[0042] In some implementations, the controller 102 may be described as being part of the ring bus 128. In one example, for a DMAOp instruction executed within a particular tile, the controller 102 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 102 may also commit the payload data to one or more wide memory units of the tile if required by an instruction in the bitmap header. The address of the one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.
[0043] In various implementations, each tile in the tile set 112, 114 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over the ring bus 128 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives data, writes it to one or more wide memory units in the tile, and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data over the ring bus 128, there is typically only one producer / master of data on the ring bus 128 at any given time. The DMAOp command execution order (e.g., FIFO control scheme) in all tiles will ensure that there is only one producer / master of data on the ring bus 128 at a given time.
[0044] In some implementations, the controller 102 uses a sync flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 128 at a given time. In one example, each write by a tile to a ring output will trigger an increment of a corresponding sync flag count. The controller 102 can examine the payload data to determine the number of data chunks or segments that contain payload. The controller 102 then monitors the execution by a tile to ensure that the expected number of data segments are transferred and / or consumed by that tile before another tile executes in master mode.
[0045] An exception to ensuring that there is only one producer / master of data on ring bus 128 at a given time occurs when there are local multicast groups connected via ring bus 128 that do not have overlapping areas on ring bus 128. For example, tile 0 (master) may multicast (i.e., generate data) to a tile in the grouping of tile 0-tile 3, and tile 4 (master) may do the same for a tile in the grouping of tile 4-tile 7. An important requirement of this dual master multicast method is that different multicast groups cannot see each other's data packets, as this could result in packet duplication and cause one or more data calculation errors.
[0046] 1, the controller 102 provides a communication data path that couples or connects the tiles in the tile sets 112, 114 to the I / O 110, and includes several core functions. The core functions of the controller 102 generally include providing one or more I / O input activations to the tiles in the tile sets 112, 114, transmitting ... receiving one or more I / O input activations from the I / O 110, and transmitting the received I / O activations to the tiles in the tile sets 112, 114. or supplying a plurality of input activations and parameters to the tiles, supplying one or more instructions received from I / O 110 to the tiles, sending I / O output activations to the host interface 108, and acting as a ring stop for the CSR bus 122 and the ring bus 128. As described in more detail below, the first tile set 112 and the second tile set 114 each include a plurality of tiles used to perform one or more tensor computations that are performed based on a deep loop nest consisting of an inner loop and an outer loop.
[0047] The system 100 generally operates as follows: The host interface 108 provides the controller 102 with one or more instructions that define the direct memory access operations (DMAOps) that will occur for a given computation. The descriptors associated with the instructions provided to the controller 102 will contain information required by the controller to facilitate large-scale dot product computations associated with multi-dimensional data arrays (tensors). In general, the controller 102 receives input activations, tile instructions, and model parameters (i.e., weights) from the host interface 108 to perform tensor computations for a given layer of the neural network. The controller 102 can then multicast the instruction(s) to the tiles 112, 114 in a dataflow manner defined by the instructions. As described above, a tile that consumes an instruction can then initiate broadcasting new / subsequent instructions to another tile based on the bitmap data in the instruction header.
[0048] In terms of data flow, input activations and parameters are sent to tiles of the tile set 112, 114 via the ring bus 128. Each of the tiles 112, 114 will store a subset of the input activations required to compute the subset of output activations assigned to that particular tile. DMAOp instructions to the tile move the input activations from wide memory to narrow memory. A computation within a tile begins when the required input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) become available to the tile. A computation occurring within a tile ends when the MAC operator (described below) within the tile completes all dot product operations defined by the instruction set and the pre-activation function is applied to the results of the multiplication operations (i.e., the output activations).
[0049] The results of one or more tensor computations include writing the output activations of the computational layer to the narrow memory unit(s) of the tile performing the computation. For some tensor computations, there will be a transfer of output edge activations to neighboring tiles via the mesh bus 126. Transfer of output edge activations to neighboring tiles is required to compute output activations for subsequent layers when the computation spans multiple layers. Once the computations for all layers are complete, a DMAOp moves the final activations to the classifier tile 116 via the ring bus 128. The controller 102 then reads the final activations from the classifier tile 116 and executes a DMAOp to move the final activations to the host interface 108. In some implementations, the classifier portion 116 performs the computation of the output layer (i.e., the last layer) of the NN. In other implementations, the output layer of the NN is one of a classifier layer, a recurrent layer, or another layer type commonly associated with neural networks.
[0050] 2 illustrates an exemplary neural network (NN) computational tile 200. In general, the exemplary tile 200 may correspond to any tile in the first tile set 112 and the second tile set 114 described above with reference to FIG. 1. In various implementations, the computational tiles 200 may also be referred to or referred to as computational units 200. Each computational tile 200 is a self-contained computational unit configured to execute instructions independently of other corresponding tiles in the tile sets 112, 114. As briefly described above, each computational tile 200 may execute two types of instructions: TensorOp instructions and D Execute MAOp instructions. Typically, each instruction type involves computational operations that are related to deep loop nests, and therefore each instruction type will typically be executed over multiple time epochs to ensure completion of all loop iterations.
[0051] As discussed in more detail below, different instruction types are executed by independent control units within the computational tile 200 that synchronize on data via synchronization flag control managed within the computational tile 200. The synchronization flag control manages concurrency between the execution of different instruction types within the computational tile 200. Each computational operation associated with each instruction type is executed in strict issue order (i.e., first in, first out). For two instruction types, TensorOP and DMAOp, there are no ordering guarantees between these different instruction types and each type is treated as a separate control thread by the computational tile 200.
[0052] With respect to the data flow configuration, the computational tile 200 generally includes data path 202 and data path 205, each of which provides a communication path for data flow into and out of the computational tile 200. As described above, the system 100 includes three different data bus structures laid out in a ring configuration: the CSR bus 122, the instruction bus 124, and the ring bus 128. With reference to FIG. 2, the data path 205 corresponds to the instruction bus 124, and the data path 202 generally corresponds to one of the CSR bus 122 and the ring bus 128. As shown, the data path 202 includes a ring output 203 that provides an output path for data exiting the computational tile 200, and a ring input 204 that provides an input path for data entering the computational tile 200.
[0053] The computational tile 200 further includes a TensorOp control 206 that includes a TensorOp tensor traversal unit (TTU) 226, and a DMAOp control 208 that includes a DMAOpTTU 228. The TensorOp control 206 generally manages writes to and reads from TensorOpTTU registers 232 to manage traverse operations for execution by the TensorOpTTU 226. Similarly, the DMAOp control 208 generally manages writes to and reads from DMAOpTTU registers 234 to manage traverse operations for execution by the DMAOpTTU 228. The TTU registers 232 include an instruction buffer for storing one or more instructions that include operations to be performed by the TensorOpTTU 226 upon execution of the instruction by the TensorOp control 206. Similarly, the TTU registers 234 include an instruction buffer for storing one or more instructions that include operations to be performed by the TTU 208 upon execution of the instruction by the DMAOp control 208. As described further below, the TTUs are used by the computation tiles 200 to traverse array elements of one or more tensors that typically reside in the narrow memory 210 and the wide memory 212.
[0054] In some implementations, an instruction for execution by the computational tile 200 arrives at the tile via data path 205 (i.e., part of the instruction bus 124). The computational tile 200 consults a header bitmap to determine the instruction type (TensorOp or DMAOp) and the instruction subtype (read or write operation). The instruction received by the computational tile 200 is then written to a particular instruction buffer depending on the instruction type. In general, an instruction is received and stored (i.e., written to a buffer) prior to execution of the instruction by components of the computational tile 200. As shown in FIG. 2, the instruction buffers (i.e., TensorOpTTU register 232 and DMAOpTTU register 234) may each include a first-in-first-out (FIFO) control scheme that prioritizes consumption (execution) of one or more associated instructions.
[0055] As briefly mentioned above, tensors are multidimensional geometric objects, and an example multidimensional The original geometric objects include matrices and data arrays. Algorithms involving deeply nested loops may be executed by the computational tiles 200 to perform tensor computations by iterating through one or more nested loops to traverse an N-dimensional tensor. In one exemplary computational process, each loop of a loop nest may be responsible for traversing a particular dimension of an N-dimensional tensor. As described herein, the TensorOp control 206 generally manages one or more tensor operations that drive the sequence of traversing and accessing dimensional elements of a particular tensor construct to complete a computation defined by a deeply nested loop.
[0056] Computational tile 200 further includes narrow memory 210 and wide memory 212. The narrow and wide designations generally refer to the size (bits / bytes) of the width of the memory units of narrow memory 210 and wide memory 212. In some implementations, narrow memory 210 includes memory units each having a size or width of less than 16 bits, and wide memory 212 includes memory units each having a size or width of less than 32 bits. In general, computational tile 200 receives input activations via data path 205, and DMA control 208 performs operations to write the input activations to narrow memory 210. Similarly, computational tile 200 receives parameters (weights) via data path 202, and DMA control 208 performs operations to write the parameters to wide memory 212. In some implementations, narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine which control device (e.g., TensorOp control 206 or DMAOp control 208) is allowed to access the shared memory units of narrow memory 210 for each memory cycle.
[0057] The computational tile 200 further includes an input activation bus 216 and a MAC array 214 including multiple cells each including a MAC operator 215 and a sum register 220. Generally, the MAC array 214 performs tensor computations, including arithmetic operations involving dot product computations, using the MAC operators 215 and sum registers 220 across multiple cells. The input activation bus 216 provides a data path along which input activations are provided by the narrow memory 210, one for each access by each MAC operator 215 of the MAC array 214. Thus, based on the broadcast of the input activations one by one, a single MAC operator 215 of a particular cell will each receive an input activation. The arithmetic operations performed by the MAC operators of the MAC array 214 generally include multiplying the input activations provided by the narrow memory 210 with parameters accessed from the wide memory 212 to generate a single output activation value.
[0058] During an arithmetic operation, partial sums may be accumulated and stored in corresponding, e.g., sum registers 220 or written to wide memory 212 and re-accessed by specific cells of MAC array 214 to complete a subsequent multiplication operation. A tensor computation may be described as having a first part and a second part. The first part is completed when the multiplication operation produces an output activation, e.g., by completing multiplication of an input activation with a parameter to produce an output activation. The second part includes application of a nonlinear function to the output activation, and the second part is completed when the output activation is written to narrow memory 210 after application of the function.
[0059] Computation tile 200 further includes an output activation bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 that indicates core attributes of the components within computation tile 200. Reference map 230 is shown for clarity but is not included in computation tile 200. Core attributes include whether a particular component is a unit, a storage device, an operator, a control device, or a data path. In general, when a first portion of a tensor computation is completed, output activations are provided from MAC array 214 to NLU 222 via output activation bus 218. NLU 2 After arriving at 22, data specifying an activation function, received via an activation pipeline 224, is applied to the output activations, which are then written to the narrow memory 210. In some implementations, the output activation bus 218 includes at least one pipelined shift register 236, and completing the second portion of the tensor computation includes shifting the output activations toward the narrow memory 210 using the shift registers 236 of the activation bus 218.
[0060] For example, for a dot product calculation of two multi-dimensional data arrays for a single computational tile 200, the MAC array 214 provides robust single instruction multiple data (SIMD) capabilities. SIMD generally means that all parallel units (multiple MAC operators 215) share the same instruction (based on deep loop nesting), but each MAC operator 215 executes the instruction on different data elements. In one basic example, element-wise addition of arrays [1,2,3,4] and [5,6,7,8] to obtain array [6,8,10,12] in one cycle would typically require four arithmetic units to perform the operation on each element. By using SIMD, the four units share the same instruction (e.g., "add") and can perform the calculation in parallel. Thus, the system 100 and computational tile 200 provide improved acceleration and parallelism over traditional methods in tensor computation.
[0061] In one example, and as described in more detail below, a single instruction may be provided by the controller 102 to multiple computational tiles 200 (see tilesets 112, 114 in FIG. 1 ) for consumption by multiple MAC arrays 214. In general, a neural network layer may include multiple output neurons, which may be partitioned such that tensor computations associated with a subset of the output neurons may be assigned to a particular tile of the tilesets 112, 114. Each tile of the tilesets 112, 114 may then perform associated tensor computations on a different group of neurons for a given layer. Thus, the computational tiles 200 may provide at least two forms of parallelism: 1) one form involves partitioning output activations (corresponding to a subset of output neurons) among multiple tiles of the tilesets 112, 114; 2) another form involves simultaneous computation (with a single instruction) of multiple subsets of output neurons based on the partitioning among tiles of the tilesets 112, 114.
[0062] FIG. 3 illustrates an exemplary tensor traversal unit (TTU) structure 300 that includes four tensors to track, each having a depth of eight. The TTU 300 generally includes a counter tensor 302, a stride tensor 304, an initial tensor 306, and a limit tensor 308. The TTU 300 further includes an adder bank 310 and a tensor address index 312. As mentioned above, tensors are multi-dimensional geometric objects, and to access an element of a tensor, an index for each dimension must be given. Since tensors are stored in narrow memory 210 and wide memory 212, a set of tensor indices must be converted to a set of memory addresses. In some implementations, the conversion of the indices to memory addresses is done by making the memory address a linear combination of the indices and reflecting the address through the tensor address index 312.
[0063] There is a TTU for each control thread, and there is a control thread for each instruction type in the computation tile 200 (TensorOP and DMAOp). Thus, as described above, there are two sets of TTUs in the computation tile 200: 1) TensorOpTTU 226; and 2) DMAOpTTU 228. In various implementations, the TensorOp control 206 assigns a TensorOpTTU counter 302, a control The limit 308 causes the stride value 304 to be loaded and does not change the register value before the instruction is retired. Each of the two TTUs will need to generate addresses for the following memory address ports in the computational tile 200: 1) the wide memory 212 address port, and 2) the narrow memory 210 with four independent arbitrated banks presented as four address ports.
[0064] As mentioned above, in some implementations, the narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine which control unit (e.g., TensorOp control 206 or DMAOp control 208) is permitted to access the shared memory resources of the narrow memory 210 for each memory cycle. In one example, different instruction types (TensorOp and DMAOp) are independent control threads requesting memory access that must be arbitrated. When a particular control thread commits a tensor element to memory, that control thread increments a counter 302 of tensor references committed to memory.
[0065] In one example, when the TensorOp control 206 executes an instruction to access a particular element of a tensor, the TTU 300 can determine the address of the particular element of the tensor, and the control 206 can access storage, such as the narrow memory 210, to retrieve data representing the activation value of the particular element. In some implementations, a program can include nested loops, and the control 206 can execute instructions to access an element of a two-dimensional array variable within the nested loop according to a current index variable value associated with the nested loop.
[0066] The TTU 300 may hold traversal state for up to X number of TTU rows simultaneously for a given tensor(s). Each tensor that resides simultaneously in the TTU 300 occupies a dedicated hardware tensor control descriptor. The hardware control descriptor may consist of X number of TTU counters 302 per row position, stride 304, and limit registers 308 that support tensors with up to X number of TTU counters per row dimension. In some implementations, the number of rows and number of counters per row may vary.
[0067] For a given position register, the final memory address is calculated from an addition operation involving adding the position registers together. The base address is incorporated into the counter 302. One or more adders are shared for tensor references that reside in the same memory. In one implementation, since there can only be a single load / store on any given port in a cycle, it is the function of the loop nest control to ensure that multiple tensor references that reside in the same narrow or wide memory do not have their counters incremented in any given cycle. The use of registers to calculate memory access address values, including determining offset values, is described in more detail in patent application Ser. No. 15 / 014,265, entitled "Matrix Processing Apparatus," filed Feb. 3, 2016, which is expressly incorporated by reference in its entirety.
[0068] The following gives the template parameters that may be used to instantiate a specialized TTU 300: 1) X number of TTU rows; 2) X number of TTU counters per row; 3) X number of TTU adder units; 4) for each TTU row, indicate the shared adder reference; and 5) for each counter, indicate X counter size [TTU][row][depth]. All TTU registers are architecturally visible. The address of the particular tensor element that needs to be accessed for a computation (i.e., tensor address 312) is the result of the counter addition. When an increment signal is issued from the controlling thread to a row of TTUs, TTU 300 performs a single cycle operation and increments the innermost dimension by that dimension's stride 304. Increment and propagate the rollover through all depths.
[0069] In general, the TTU 300 determines state associated with one or more tensors. The state may include loop bound values, a current loop index variable value, a dimension multiplier for computing memory address values, and / or a program counter value for processing branch loop bounds. The TTU 300 may include one or more tensor state elements and an arithmetic logic unit. Each of the tensor state elements may be a storage element, such as a register or any other suitable storage circuit. In some implementations, the tensor state elements may be physically or logically organized into distinct groups.
[0070] FIG. 4 illustrates an exemplary architecture including a narrow memory 210 that broadcasts activations 404 to one or more multiply-accumulate (MAC) operators via an input bus 216. A shift register 404 provides a shifting function where activations 404 are sent out one at a time to the input bus 216 for receipt by one or more MAC operators 215 in the MAC cell 410. In general, the MAC cell 410 including the MAC operator 215 can be defined as a computation cell that computes partial sums and, in some implementations, is configured to write the partial sum data to an output bus 218. As shown, the cell 410 may consist of one or more MAC operators. In one implementation, the number of MAC operators 215 in the MAC cell 410 is referred to as the issue width of the cell. As an example, a dual issue cell refers to a cell with two MAC operators that can compute the multiplication of two activation values (from the narrow memory 210) with two parameters (from the wide memory 212) and perform an addition between the results of the two multipliers and the current partial sum.
[0071] As mentioned above, the input bus 216 is a broadcast bus that provides input activations to the MAC operators 215 of the linear unit (i.e., the MAC array 214). In some implementations, the same input is shared between all the MAC operators 215. The width of the input bus 216 must be wide enough to provide the broadcast inputs to a corresponding number of cells for a given MAC array 214. Consider the following example to illustrate the structure of the input bus 216. When the number of cells in the linear unit is equal to 4 and the activation width is equal to 8 bits, the input bus 216 can be configured to provide up to four input activations per cycle. In this example, all cells in the MAC array 214 will access only one of the four activations that are broadcast.
[0072] Based on the TensorOp field settings of the instructions received by the computational tile 200, the cells of the MAC array 214 may need to perform a computation using the same input activation. This may be referred to as Zout partitioning within the cells of the MAC array 214. Similarly, Zin partitioning within a cell occurs when the cells of the MAC array 214 need different activations to perform a computation. In the former case, a single input activation is replicated four times and four activations read from the narrow memory 210 are broadcast over four cycles. In the latter case, a read of the narrow memory 210 is required every cycle. In the above example, the TensorOp control 206 orchestrates this broadcast method based on the execution of the instructions received from the controller 102.
[0073] 5 shows an exemplary architecture including an output bus 218 for providing output activations to the narrow memory units 210 of FIGS. 2 and 4. In general, each MAC cell 215 of the MAC array 214 in a computational tile 200 computes a different output activation. However, if the output feature depth for an output feature array is less than the number of MAC cells 215 in the computational tile 200, the cells may be grouped to form one or more cell groups. All MAC cells 215 in a cell group compute the same output (i.e., for an output feature map), but each cell computes a subset of the outputs, corresponding to a subset of the Zin dimensions. 238 provides control signals to NLU 222 that are used to compute partial sums of the linear inputs of NLU 222. In some implementations, NLU 222 only computes subsets of the linear inputs of NLU 222. As a result, the output of MAC cell 215 is now a partial sum rather than a final linear output. In some implementations, NLU 222 aggregates these partial sums into a final linear output based on control signals provided to NLU 222 by NLU control 238.
[0074] As mentioned above, the output bus 218 is a pipelined shift register. In various implementations, when the first part of the tensor computation is finished and the TensorOp control 206 indicates that the partial sums need to be written out (by executing an instruction), there will be parallel loads of partial sums provided to the output bus 218. The number of parallel loads will correspond to the number of MAC cells in the computation tile 200. The TensorOp control 206 then causes the partial sum amounts to be shifted out and sent through the non-linear pipeline. In some implementations, there may be situations where not all MAC cells in the tile are actually utilized to perform the computation. In such situations, not all partial sums shifted to the output bus will be valid. In this example, the TensorOp control 206 may provide control signals to the MAC array 214 to indicate the number of valid cells to be shifted out. The amount of parallel loads loaded to the output bus 218 still corresponds to the number of MAC cells in the computation tile, but only the valid values will be shifted out and committed to the narrow memory 210.
[0075] FIG. 6 is an exemplary flow chart of a process 600 for performing tensor computations using a neural network (NN) computational tile, such as the computational tile 200 of FIG. 2. The process 600 begins at block 602 by loading sufficient parameters into an on-chip second memory to allow for latency below a particular level and throughput above a particular level for a given NN model and architecture. Throughput is the maximum performance achieved in the presence of a large number of requests / inferences. Latency is the minimum time it takes to compute one request. The process 600 continues at block 604, where the narrow memory 210 of the computational tile 200 sends (i.e., broadcasts) activations one by one onto the input activation data bus 216. The activation values are stored in the narrow memory 210. The narrow memory 210 may be a collection of static random access memory (SRAM) banks that allow addressing of specific memory locations to access input quantities. The activations read from memory 210 are broadcast via input activation bus 216 to linear cells of a MAC array 214 (i.e., a linear unit) that includes multiple MAC operators 215 and sum registers 220. At block 606 of process 600, the MAC operators 215 of the computational tile 200 each receive two inputs: one input (the activation) is received from the input activation bus 216; the other input (the parameter) is received from the wide memory 212. Thus, the activations provide one of the inputs of each MAC operator 215, and each MAC operator 215 in a cell of the MAC array 214 gets their second multiplier input from the wide memory 212.
[0076] At block 608 of process 600, the MAC array 214 of the computational tile 200 performs tensor computations, including dot product computations, based on elements of the data array structure accessed from memory. The wide memory 212 may have a width in bits equal to the width of the linear unit (e.g., 32 bits). Thus, the linear unit (LU) is a SIMD vector arithmetic logic unit (ALU) unit that receives data from the vector memory (i.e., the wide memory 212). In some implementations, the MAC operator 215 may also get an accumulator input (partial sum) from the wide memory 212. In some implementations, there is time sharing for the wide memory 212 ports for reading and / or writing for two different operands (parameters and partial sums). In general, to optimize area, the wide memory 212 may have a limited number of ports. As a result, it is not necessary to read an operand (e.g., parameter) from the wide memory 212 and simultaneously write an operand (e.g., If a particular operand needs to be written (e.g., a partial sum), the pipeline associated with that particular operand may stall.
[0077] At block 610, the computational cells of the computational tile 200 (having MAC operators 215 and sum registers 220) generate at least one output activation based on the multiplication operation performed by the MAC / computational cell. The result of the MAC cell operation includes either a partial sum that is written back to memory (during partial sum arithmetic operations) or an output activation that is sent to the output bus 218. The NLU 222 of the computational tile 200 may apply a non-linear activation function to the output activations and write the activations to the narrow memory 210. In some implementations, the output bus 218 is a shift register that can accumulate parallel loads of results / output activations from the MAC operators 215 and shift them out one at a time for application of the non-linear function and write operations to the narrow memory 210 of the same tile.
[0078] The embodiments described in this specification exploit two perspectives: 1) the bottleneck of most existing Neural Network (NN) accelerators is the memory bandwidth required to load the NN weights (also known as parameters); 2) even if the number of parameters in the generative model is large, i.e., a few KB to a few GB, as opposed to a few MB to a few hundred MB for most models, these numbers are within the realm of what can be achieved with hardware including on-chip memory, e.g., with memory distributed among tiles.
[0079] Adding a large memory, e.g. a cache or scratchpad, on the die to contain all the parameters is not enough. The goal of overcoming the memory bandwidth limitations is to scale out the performance of the architecture, which means increasing the number of operators, usually MACs. But to achieve high performance, we must be able to supply these operators, along with the parameters, every cycle. It is also important to understand "performance" not only as throughput but also as latency. This is true for many user-facing applications.
[0080] In other words, in a Neural Network (NN) accelerator, loading parameters into one layer at a time is very costly. If the parameters can be preloaded on-chip, they only need to be loaded for runtime activation. In other words, the embodiment has a large on-chip memory.
[0081] The embodiments described in this specification affect the operation and design of Neural Network (NN) accelerators. The embodiments address one of the prominent limitations of neural network accelerator design: the combination of low latency and high throughput. We assume that the NN accelerator represents the entire neural network. The input is some data, e.g., a small image or audio. The NN accelerator executes layers one by one. What is costly in performance and energy is loading the parameters of the layers one by one. The NN accelerator loads the parameters of a layer, performs the calculation, holds the output of the layer, and then loads the parameters of the next layer. This process is where most of the memory bandwidth is consumed.
[0082] By having the parameters reside on-chip, latency is dramatically reduced given the high throughput and the specific NN model and architecture. The accelerator only needs to load a few bytes of audio, after which it can be very fast. The accelerator does not require high memory bandwidth and the energy goes down.
[0083] The von Neumann model, where memory is loaded from memory to the CPU, is a common architecture. Such a traditional von Neumann model architecture, where memory resides on one end of the die and computational operators reside on the other end of the die, is impractical, if not impossible, for a large number of operators, implying a huge number of wires to transfer data from memory (or memory banks) to the operators. Instead, embodiments of the present invention take advantage of the memory locality property of NN computations to organize the architecture into a tiled configuration with memory distributed among tiles (as shown in Figures 2 and 8).
[0084] NNs are large, but not huge, so we can get close to effectively fitting all the parameters of one or a few NNs on one chip. NN accelerators are moving to self-contained architectures. With tiled architectures, memory can be partitioned within the chip. Instead of having one large SRAM in one corner of the chip, embodiments allocate an appropriate SRAM per tile, avoiding on-chip bandwidth issues as well. In a particular embodiment, a wide memory for each tile contains the parameters, and a wide ring (of roughly the same width as the wide memory) provides the wide memory with high bandwidth. The embodiments described in this specification consider architecture variations. Depending on the characteristics of the NN layer, embodiments can have at least two NN architectures:
[0085] In neural networks that are mostly composed of fully connected layers, there is little reuse of parameters between layers. A neuron in a fully connected layer has a full connection to all activations in the previous layer. Consider a fully connected neural network where parameters are not reused (in the absence of batch processing, for example, consider a real-time application). If all parameters are not contained in the wide memory, they must be fetched from an external memory via a ring bus. In that case, the performance of the entire design is limited by the external memory bandwidth. If all parameters reside in the wide memory, all operators can be supplied with parameters every cycle without the need to access the external memory, maximizing performance. Instead of utilizing the external memory only to fetch the parameters, the embodiment described in this specification keeps the parameters resident in the wide memory.
[0086] As an example, consider a model with 50M parameters in the fully connected layer (or, for simplicity's sake, 50MB). Consider an accelerator with 16384 MACs running at 1GHz. Consider the input to the model to be 16KB. All values are reasonable for the current application. Maximum performance corresponds to running the model in 50×10^6 / (16384×10^9)=3.05×10^-6 seconds. This corresponds to a memory bandwidth of (50×10^6+16,384) / (3.05×10^-6)=16.40TB / s. As a point of comparison, typical DRAM chips are around 10GB / s, while state-of-the-art high-bandwidth memory (HBM) is around 256GB / s.
[0087] Convolutional neural networks pass parameters from one tile to another. In neural networks that consist mostly of convolutional layers, where parameters are reused between neurons (also known as activations), the memory bandwidth requirements are not very high, but are generally higher than a typical external memory. The ring bandwidth may be enough to load the parameters into the tiles, but it is connected to a large on-die memory of the same width as the ring. In other words, for every inference, the tiles need to access / load all the parameters of the model. This is true for any model. The only difference with fully connected layers in neural networks is that each parameter is used only once during one inference. For convolutional layers, a parameter can be used multiple times within the layer. Used.
[0088] As an example, consider a model with 50M parameters in the convolutional layers. Some of these layers can be very small and others large, so there is variation in parameter reuse. A reasonable average across models is ~100 reuses per parameter. So, using the same theory as above, the bandwidth requirement goes down to 16.40TB / s / 100 ~ 164GB / s. However, the bandwidth requirement for cost-effective DRAM remains high. However, in the above architecture, a wide ring of 164x8=1312 bits connected to a large memory of the same width can supply the tiles at a reasonable speed.
[0089] Consider a first embodiment where the memory is sufficient to contain all the parameters of the layers of a fully connected model. If all tiles operate simultaneously on one layer, the parameters need to be distributed among the tiles. The embodiment splits the output neurons / activations of each layer among the tiles. At runtime, each tile processes a subset of the layers, computes the corresponding partial sums and passes them on to its neighbors. That is, the partial sums go around the ring and, after going around all of them, the tile produces the final sum.
[0090] A second embodiment involves preloading / caching the same subset of (currently used) parameters for all tiles, since the tiles utilize the same parameters simultaneously. At run-time, the parameters (subset) that are not partial activation sums rotate around the ring.
[0091] The number of tiles is a scale factor. Robust scaling can be achieved by improving latency and throughput without increasing memory requirements using the embodiments described herein. However, processing tiles together to increase computational power increases memory requirements and the number of activations required. Batched scaling is difficult without increasing memory bandwidth over traditional memory options. Batch processing is often associated with real-time applications and has latency and throughput requirements.
[0092] It is worth noting that having parameters in cache is different from having them in wide memory as part of the tiling. Arranging the tiles as described herein provides compiled-style locality of reference. For example, placing the fully connected model next to SRAM provides greater internal bandwidth compared to the cached model.
[0093] The embodiments described herein run faster than conventional neural network accelerators. Certain embodiments have many more operators, and the accelerator requires more internal bandwidth to supply the operators. To address this, the architecture must distribute memory and aggregate parameters on the accelerator.
[0094] Currently the largest chips are around 650 square millimeters. Therefore there is a limit to how much SRAM can be put on a chip. Embodiments include utilizing higher density memory, including utilizing 3D stacking, given space constraints.
[0095] The embodiments described in this specification apply to the inference (post-training) mode and the training mode.
[0096] An additional consideration is another level of hierarchy: the memory hierarchy, which usually consists of a register file. There is also a file hierarchy. Parameters are loaded into registers and the registers are reused, expanding memory bandwidth. There is memory bandwidth in the register file and in the memory. This means further cost reduction, i.e. less wiring from memory to the register file and from the register file to the computation. The embodiments described herein can reduce the wiring costs associated with the register file. Addressing is done by consuming parameters directly from SRAM, i.e. the memory feeds the ALU directly.
[0097] FIG. 7 illustrates a block diagram of another embodiment of a computational system 700 for accelerating tensor computations associated with deep neural networks (DNNs). The system 700 generally includes a controller / uncore (UNCORE) 702, a memory interface 708, and a number of tiles including a first tile set 112 and a second tile set 714. The controller 702 generally includes a data memory 704, an instruction memory 706, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. The instruction memory 706 can store one or more machine-readable instructions executable by the one or more processors of the controller 702. The data memory 704 can be any of a variety of data storage media for storing and subsequently accessing various data associated with computations occurring within the system 700.
[0098] The controller 702 is configured to execute one or more instructions related to tensor computations in the system 700, including instructions stored in an instruction memory 706. In some implementations, the data memory 704 and the instruction memory 706 are volatile memory unit(s). In some other implementations, the data memory 704 and the instruction memory 706 are non-volatile memory unit(s). The data memory 704 and the instruction memory 706 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In various implementations, the controller 702 may be referred to or called a core manager 702.
[0099] The memory interface 708 receives instructions and data parameters from the I / O link and provides the instructions and parameters to the controller 702. In general, instructions may be provided to one or more devices in the system 100 via an instruction bus (an instruction bus between the controller and the tiles is not shown) and parameters may be provided to one or more devices in the system 700 via a ring bus 728. In some implementations, instructions are initially received by the controller 702 from the host interface 708 and stored in the instruction memory 706 for later execution by the controller 702.
[0100] The ring bus 728 provides communications that originate from the controller 702 and couple via bus data paths that connect the tiles 712, 714 in a ring back to the controller 702. In various implementations, the ring bus 728 generally connects or couples all of the wide memory units in all of the tiles 712, 714. Thus, the payload width of the ring bus 728 corresponds to the width of the wide memory units located within each tile of the tile set 712, 714. As mentioned above, the ring bus 728 also includes a bitmap header that indicates which tiles should consume payload data, including commands or parameters, communicated over the ring bus 728.
[0101] Data (i.e., payload) received at a particular tile via the ring bus 728 With respect to, in response to receiving information, each tile zeroes (i.e., clears) the position data indicated in the bitmap header specific to the receiving tile before forwarding the data to another tile. Thus, when the header bitmap has no remaining bits set data indicating the particular tile that is to receive the payload, forwarding of the payload to another tile will stop. Payload data generally refers to activations and weights used by one or more tiles during tensor computations that are performed based on the execution of deeply nested loops.
[0102] In some implementations, the controller 702 may be described as being part of the ring bus 728. In one example, for a DMAOp instruction executed within a particular tile, the controller 702 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 702 may also commit the payload data to one or more wide memory units of the tile if required by an instruction in the bitmap header. The address of the one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.
[0103] In various implementations, each tile in the tile set 712, 714 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over the ring bus 728 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives data, writes it to one or more wide memory units in the tile, and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data over the ring bus 728, there is typically only one producer / master of data on the ring bus 728 at any given time. The DMAOp command execution order (e.g., a FIFO control scheme) in all tiles will ensure that there is only one producer / master of data on the ring bus 728 at a given time.
[0104] In some implementations, the controller 702 uses a sync flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 728 at a given time. In one example, each write by a tile to a ring output will trigger an increment of a corresponding sync flag count. The controller 702 can examine the payload data to determine the number of data chunks or segments that contain payload. The controller 702 then monitors the execution by a tile to ensure that the expected number of data segments are transferred and / or consumed by that tile before another tile executes in master mode.
[0105] An exception to ensuring that there is only one producer / master of data at a given time on the ring bus 728 occurs when there are local multicast groups connected via the ring bus 728 that do not have overlapping areas on the ring bus 728. A key requirement of this dual master multicast method is to ensure that different multicast groups cannot see each other's data packets, as this could result in packet duplication and cause one or more data computation errors.
[0106] In contrast to Figure 1, the tiles in Figure 7 form a rectangular pattern by themselves, while being connected in a zigzag or serpentine fashion by a ring bus. In the illustrated embodiment, the accelerators are made up of 8 and 16 tiles, respectively. In alternative embodiments, the accelerators may contain even more tiles.
[0107] FIG. 8 is a simplified diagram of the computational tile of FIG. 9 is an example flow chart of a process 900 for performing tensor computations using a neural network (NN) computational tile, such as computational tile 200 of FIG. 2. Process 900 begins at block 902 with loading NN weight parameters into a NN accelerator, e.g., prior to execution. Process 900 continues at block 904 with processing inputs to the accelerator, substantially without accessing the neural network weight parameters external to the accelerator. At block 906, the process generates at least one output activation based on processing the inputs to the accelerator.
[0108] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by or to control the operation of a data processing device. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, such as, for example, a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.
[0109] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be realized by, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).
[0110] A processor suitable for executing a computer program may be based, for example, on a general-purpose or special-purpose microprocessor or both, or on any type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions, and one or more memory devices for storing instructions and data. In general, a computer further includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from the one or more mass storage devices or to transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices.
[0111] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media and memory devices, including semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.
[0112] Although the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination, and may even be initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.
[0113] Similarly, although operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, to achieve desirable results, or that all of the shown operations need to be performed. In some situations, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system modules and components in the above-described embodiments does not require such separation in all embodiments, and that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.
[0114] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
Claims
1. An accelerator for accelerating tensor computations, comprising: a. a computation unit; b. a first memory bank for storing at least one of the input activations or the output activations; and c. a second memory bank for storing neural network parameters used in performing the calculations, the second memory bank configured to store a sufficient amount of the neural network parameters on the computation unit to enable, for a given NN model and architecture, a latency below a particular level and a throughput above a particular level, the accelerator further comprising: d. at least one cell including at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs a calculation; e. a first traversal unit in data communication with at least said first memory bank, said first traversal unit configured to provide control signals to said first memory bank to provide input activations to a data bus accessible by said MAC operator; f. an accelerator that performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between the input activations received from the data bus and a parameter received from the second memory bank.
2. The accelerator of claim 1 , wherein the second memory bank is configured to store greater than 100,000 parameters.
3. The accelerator of claim 1 , wherein the second memory bank is configured to store greater than 1,000,000 parameters.
4. 2. The accelerator of claim 1, wherein the second memory bank is configured to store greater than 100,000,000 parameters.
5. The accelerator of claim 1 , wherein the second memory bank comprises an SRAM.
6. The accelerator includes a plurality of computing units, each computing unit having: a. a first memory bank for storing at least one of input activations or output activations; and b. a second memory bank for storing neural network parameters used in performing the calculations, said second memory bank configured to store a sufficient amount of said neural network parameters on said computation units to enable, for a given NN model and architecture, a latency below a particular level and a throughput above a particular level, each computation unit further comprising: c. at least one cell including at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs a calculation; d. a first traversal unit in data communication with at least said first memory bank, said first traversal unit configured to provide control signals to said first memory bank to provide input activations to a data bus accessible by said MAC operator; e. the computation unit performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and based in part on the input activations received from the data bus and the second memory bank. The accelerator of claim 1 , further comprising a multiplication operation with a parameter received from
7. The accelerator of claim 1 , wherein the second memory bank includes 3D SRAM.
8. 1. A computer-implemented method for accelerating tensor computations, comprising the steps of: a. sending a first input activation by a first memory bank in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit, the method further comprising: receiving one or more parameters from a second memory bank for storing neural network parameters used in performing calculations by said at least one cell, said second memory bank storing a sufficient amount of said neural network parameters on said computation unit to enable, for a given NN model and architecture, a latency below a particular threshold with a throughput above a particular threshold, said at least one cell including at least one multiply-accumulate ("MAC") operator, said method further comprising: performing, by said MAC operator, one or more calculations relating to at least one element of a data array, said one or more calculations including, in part, a multiplication operation of at least said first input activation accessed from said data bus and at least one parameter received from said second memory bank.
9. The method of claim 8 , wherein the second memory bank is configured to store greater than 100,000 parameters.
10. The method of claim 8 , wherein the second memory bank is configured to store greater than 1,000,000 parameters.
11. The method of claim 8 , wherein the second memory bank comprises SRAM.
12. The method of claim 8 , wherein the second memory bank comprises 3D SRAM.
13. 9. The method of claim 8, further comprising the step of loading the neural network parameters into the second memory for use in performing calculations.
14. 1. A computer-implemented method for accelerating tensor computations, comprising the steps of: a. sending a first input activation by a first memory bank in response to said first memory bank receiving a control signal, said first input activation being sent over a data bus, said method further comprising: b. receiving one or more parameters from a second memory bank for storing neural network parameters used in performing calculations by at least one multiply-accumulate ("MAC") operator, said second memory bank storing a sufficient amount of said neural network parameters on said computation unit to enable, for a given NN model and architecture, a latency below a particular threshold and a throughput above a particular threshold, said method further comprising: performing, by said MAC operator, one or more calculations relating to at least one element of a data array, said one or more calculations including, in part, a multiplication operation of at least said first input activation accessed from said data bus and at least one parameter received from said second memory bank.
15. The method of claim 14 , wherein the second memory bank is configured to store greater than 100,000 parameters.
16. The method of claim 14 , wherein the second memory bank is configured to store greater than 1,000,000 parameters.
17. The method of claim 14 , wherein the second memory bank is configured to store greater than 100,000,000 parameters.
18. 15. The method of claim 14, wherein the second memory bank comprises SRAM.
19. The method of claim 14 , wherein the second memory bank comprises 3D SRAM.
20. 15. The method of claim 14, further comprising the step of loading the neural network parameters into the second memory for use in performing calculations.
Citation Information
Patent Citations
Neural network arithmetic device
JP2001117900A
Neural network compute tile
US9710265B1
