Neural Network Calculation Tile

Abstract

A computation unit and method for computing deep neural network layers that reduces instruction bandwidth and instruction memory is provided. [Solution] A neural network computational tile 200 is a self-contained computational unit that corresponds to any tile in a first tile set and a second tile set of a computing system and independently executes instructions on other corresponding tiles in the tile sets, and includes a TensorOp control 206 that includes a TensorOp tensor traversal unit (TTU) 226, and a DMAOp control 208 that includes a DMAOpTTU 228, where the TensorOp control manages writing to and reading from TensorOpTTU registers 232 and manages traverse operations for execution by the TensorOpTTU, and the DMAOp control manages writing to and reading from DMAOpTTU registers 234 and manages traverse operations for execution by the DMAOpTTU 228.

Description

[Background technology]

[0001] background This specification generally relates to neural net computation tiles for computing deep neural network ("DNN") layers that enable reduced instruction bandwidth and instruction memory. Summary of the Invention [Means for solving the problem]

[0002] overview In general, one innovative aspect of the subject matter described in this specification may be embodied in a computation unit for accelerating tensor computations. The computation unit includes a first memory bank having a first data width for storing at least one of input activations or output activations, and a second memory bank having a second data width greater than the first data width for storing one or more parameters used in performing the computation. The computation unit may further include at least one cell including at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs the computation. The computation unit further includes a first traversal unit in data communication with at least the first memory bank, the first traversal unit configured to provide control signals to the first memory bank to provide input activations to a data bus accessible by the MAC operator. The computation unit performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between input activations received from the data bus and parameters received from the second memory bank.

[0003] In general, another innovative aspect of the subject matter described herein may be embodied in a computer-implemented method for accelerating tensor computation. The computer-implemented method comprises: sending a first input activation by a first memory bank having a first data width in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit. The method further comprises receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell including at least one multiply-accumulate ("MAC") operator. The method further comprises performing, by the MAC operator, one or more calculations related to at least one element of a data array, the one or more calculations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank.

[0004] Another innovative aspect of the subject matter described herein may be embodied in a non-transitory computer-readable storage medium including instructions executable by one or more processors that, when executed, cause the one or more processors to perform an operation including sending a first input activation by a first memory bank having a first data width in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit. The performed operation may further include receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell receiving the one or more parameters from a second memory bank having a second data width greater than the first data width. A cell includes at least one multiply-accumulate ("MAC") operator. The operations performed may further include performing, by the MAC operator, one or more calculations related to at least one element of the data array, the one or more calculations including, in part, a multiplication operation between at least a first input activation accessed from the data bus and at least one parameter received from the second memory bank.

[0005] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages: Using registers to track memory address values ​​allows a program to iterate through deeply nested loops with a single instruction. Tensors accessible from narrow and wide memory units in a single computational tile are traversed based on memory address values ​​retrieved from registers. The memory address values ​​correspond to elements of the tensor. Tensor computations occur in individual computational tiles based on the execution of deep loop nests. Computations can be distributed across multiple tiles. Computational efficiency is improved and accelerated based on distributing tensor computations of a multi-layer neural network across several computational tiles. Tensors can be traversed and tensor computations can be performed with fewer instructions.

[0006] The subject matter described herein may also be implemented in particular embodiments to realize other advantages, such as achieving high utilization of MAC operators for DNN layers of vastly different dimensionality and maximizing computational locality by employing a memory hierarchy that combines narrow, low-bandwidth memories with wide, high-bandwidth memories, where addressing flexibility allows multidimensional arrays to be traversed in any order.

[0007] Other implementations of this and other aspects include corresponding systems, devices, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof that is installed on the system and that, when executed by a data processing device, causes the system to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the device to perform the actions.

[0008] The subject matter described herein also relates to an image recognition and / or classification method / system. The system can be implemented using the disclosed techniques and the described hardware computing system having hardware computing units or computing tiles. The computing units process tensor calculations for computing inferences using neural networks with multiple neural network layers.

[0009] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an example computing system. [Figure 2] 1 illustrates an exemplary neural network computational tile. [Figure 3] 1 illustrates an exemplary tensor traversal unit (TTU) structure. [Figure 4] 1 illustrates an example architecture that includes a narrow memory unit that provides input activations to one or more multiply-accumulate (MAC) operators. [Figure 5] 5 illustrates an exemplary architecture including an output bus that provides output activation to the narrow memory units of FIGS. 2 and 4. [Figure 6] 3 is an exemplary flowchart of a process for performing tensor computations using the neural network computational tile of FIG. 2. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference numbers and designations in the various drawings indicate like elements. Detailed Description The subject matter described herein relates to a hardware computing system including multiple computing units configured to accelerate machine learning inference workloads of neural network layers, where each computing unit of the hardware computing system is self-contained and capable of independently performing the computations required by a given layer of the multi-layer neural network.

[0012] A neural network with multiple layers can be used to compute an inference. For example, given an input, the neural network can compute an inference for that input. The neural network computes this inference by processing the input through each layer of the neural network. In particular, each layer of a neural network has its own set of weights. Each layer receives an input and processes the input according to the set of weights for that layer to generate an output.

[0013] Thus, to compute an inference from a received input, a neural network receives the input and processes it through each neural network layer to generate an inference, with the output from one neural network layer being given as the input to the next neural network layer. The data input to a neural network layer, e.g., the input to the neural network, or the output to a neural network layer of a layer below it in the sequence, can be referred to as the activation input to that layer.

[0014] In some implementations, the layers of a neural network are arranged in a sequence. In other implementations, the layers are arranged in a directed graph, meaning that any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be configured so that the output of one layer can be sent back as input to a previous layer.

[0015] The hardware computing system described herein can perform neural network layer computations by distributing tensor computations across multiple computation tiles. The computation process performed within the neural network layer can include multiplication of input tensors including input activations and parameter tensors including weights. The computation includes multiplying the input activations by the weights in one or more cycles and performing an accumulation of the products over many cycles.

[0016] Tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. Generally, software algorithms are executed by computational tiles to perform tensor computations by processing nested loops to traverse N-dimensional tensors. In one exemplary computation process, each loop may be responsible for traversing a particular dimension of an N-dimensional tensor. For a given tensor construct, a computational tile may need access to elements of a particular tensor to perform multiple dot product computations associated with that tensor. Computations occur when input activations provided by narrow memory structures are multiplied by parameters or weights provided by wide memory structures. Because tensors are stored in memory, a set of tensor indices may require conversion to a set of memory addresses. Generally, a tensor traversal unit in a computational tile performs control operations that provide the indices for each dimension associated with a tensor and the order in which the index elements are traversed to perform computations. The multiplication result The tensor computation is complete when the result is written to the output bus and stored in memory.

[0017] FIG. 1 illustrates a block diagram of an exemplary computing system 100 for accelerating tensor computations associated with deep neural networks (DNNs). System 100 generally includes a controller 102, a host interface 108, an input / output (I / O) link 110, a plurality of tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a data bus (shown for clarity but not included in system 100) identified in a bus map 118. Controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. Instruction memory 106 can store one or more machine-readable instructions executable by the one or more processors of controller 102. Data memory 104 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations occurring within system 100.

[0018] The controller 102 is configured to execute one or more instructions related to tensor computations in the system 100, including instructions stored in the instruction memory 106. In some implementations, the data memory 104 and the instruction memory 106 are volatile memory units or units. In other implementations, the data memory 104 and the instruction memory 106 are non-volatile memory units or units. The data memory 104 and the instruction memory 106 may also be another form of computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In various implementations, the controller 102 may be referred to as or be called a core manager 102.

[0019] As shown, host interface 108 is coupled to I / O link 110, controller 102, and classifier portion 116. Host interface 108 receives instructions and data parameters from I / O link 110 and provides instructions and parameters to controller 102. Generally, instructions may be provided to one or more devices in system 100 via instruction bus 124 (described below), and parameters may be provided to one or more devices in system 100 via ring bus 128 (described below). In some implementations, instructions are initially received by controller 102 from host interface 118 and stored in instruction memory 106 for later execution by controller 102.

[0020] The classifier portion 116 is similarly coupled to the controller 102 and tile 7 of the second tile set 114. In some implementations, the classifier portion 116 is implemented as a separate tile within the system 100. In alternative implementations, the classifier portion 116 is disposed or located within the controller 102 as a subcircuit or subdevice of the controller 102. The classifier portion 116 is generally configured to perform one or more functions on accumulated pre-activation values ​​received as the output of a fully connected layer. The fully connected layer may be divided across tiles in the tile sets 112 and 114. Thus, each tile is configured to generate a subset of pre-activation values ​​(i.e., linear outputs) that can be stored in the tile's memory unit. A classification result bus 120 provides a data path from the classifier portion 116 to the controller 102. Data including post-function values ​​(i.e., results) are provided from the classifier portion 116 to the controller 102 via the classification result bus 120.

[0021] The bus map 118 is a map of the first tile set 112 and the second tile set 114. 1 illustrates a data bus providing one or more interconnected data communication paths between tiles. Bus map 118 provides a description of the symbols used to identify classification result bus 120, CSR / master bus 122, instruction bus 124, mesh bus 126, and ring bus 128, as shown in FIG. 1 . Generally, tiles are core components within the accelerator architecture of system 100 and are the focal point for tensor computations occurring within the system. Each tile is an individual computational unit, and multiple tiles can interact with other tiles in the system to accelerate computations (e.g., tensor computations) across one or more layers of a multilayer neural network. For example, computations can be distributed across multiple tiles. Computational efficiency can be improved and accelerated based on distributing the tensor computations of a multilayer neural network across several computational tiles. While tiles within a tile set 112, 114 can share the execution of tensor computations associated with a given instruction, individual computational units are self-contained computational components configured to perform a subset of tensor computations independently of other corresponding tiles within the tile set 112, 114.

[0022] The control and status register (CSR) bus 122 is a single-master, multiple-slave bus that allows the controller 102 to send one or more instructions that set program configurations and read status registers associated with one or more tiles. The CSR bus 122 can be connected in a single daisy-chain configuration with one master bus segment and multiple slave bus segments. As shown in FIG. 1 , the CSR bus 122 provides communication coupling the tiles of the tile sets 112, 114 and the controller 102 via a bus data path that connects them in a ring to the host interface 110. In some implementations, the host interface 110 is the single master of the CSR bus ring, and the entire CSR bus address space is memory-mapped into memory space within the host interface 110.

[0023] The CSR bus 122 may be used by the host interface 110 to perform one or more operations, including, for example, programming memory buffer pointers within the controller 102 to enable the controller 102 to begin fetching instructions from the instruction memory 106, updating / programming various tile settings (e.g., coefficient tables for polynomial approximation calculations) that remain static for one or more calculations, and / or loading / reloading firmware for the classifier portion 116. In one example, a firmware reload may include a new function to be applied to the linear output (i.e., pre-activation values). Thus, every slave with access to the CSR bus 122 will have a distinct node identifier (node ​​ID) associated with and identifying that slave. The node ID is part of the instruction address and is used, inspected, or otherwise consulted by the CSR slave (i.e., the controller 102, tiles 112, 114, and classifier 116) to determine whether a CSR packet is addressed to the slave.

[0024] In some implementations, one or more instructions can be sent by the host interface 102 through the controller 102. An instruction may be, for example, 32 bits wide, with the first 7 bits containing header information indicating the instruction address / destination that is to receive and execute the instruction. The first 7 bits of the header may contain a data parameter that represents a specific node ID. Thus, a slave (e.g., each tile) on the CSR bus ring can inspect the header of the instruction to determine whether a request by the master (host interface 110) is addressed to the tile inspecting the header. If the node ID in the header does not indicate that the destination is the inspection tile, the inspection tile copies the incoming CSR instruction packet to the CSR bus input connected to the next tile for inspection by the next tile.

[0025] The instruction bus 124 provides communications that originate from the controller 102 and couple the tiles in the tile sets 112, 114 via bus data paths that, similar to the CSR bus 122, connect the tiles in a ring back to the controller 102. In one implementation, the controller 102 broadcasts one or more instructions via the instruction bus 124. The instructions broadcast by the controller 102 may differ from the instructions provided via the CSR bus 122. However, the manner in which tiles receive and / or consume or execute instructions received via the bus 124 may be similar to the process for executing instructions received via the CSR bus 122.

[0026] In one example, the header of an instruction (i.e., a bitmap) indicates to a receiving tile that it needs to consume a particular instruction based on the bitmap associated with that instruction. The bitmap may have a particular width defined in terms of bits. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of instruction bus 124 may be configured to be smaller than the size / width of the instruction. Thus, in such a configuration, the transmission of instructions occurs over several cycles, and the bus stops of instruction bus 124 have decoders to place the instructions received at that tile into the appropriate target instruction buffer associated with that tile.

[0027] As described further below, tiles within tile sets 112, 114 are generally configured to support two broad categories of instructions. The two broad categories are also referred to as instruction types. The instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, a DMAOp instruction has one or more specializations that are allowed to occur simultaneously. The one or more specializations may be referred to as DMAOp instruction subtypes or opcodes. In some cases, every unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a particular tile.

[0028] At a particular one of the tiles 112, 114, the bus stop associated with the instruction bus 124 examines the header bitmap to determine the instruction type / subtype. The instruction may be received by the tile and subsequently written to the tile's instruction buffer prior to execution of the instruction by the tile. The tile's instruction buffer to which the instruction is written may be determined by the instruction's type and subtype indicator / fields. The instruction buffer may include a first-in, first-out (FIFO) control scheme that prioritizes consumption of one or more associated instructions. Therefore, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which they arrive on the instruction bus.

[0029] The different instruction buffers within a tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As noted above, instruction types include TensorOp instructions and DMAOp instructions. With respect to the DMAOp instruction, instruction subtypes (indicating the "write to" buffer location) include: 1) mesh inbound instruction buffer; 2) mesh outbound instruction buffer; 3) narrow-wide DMA instruction buffer; 4) wide-narrow DMA instruction buffer; and 5) ring bus DMA instruction buffer. These buffer locations are described in more detail below with reference to FIG. 2. The wide and narrow designations are used throughout this specification and generally refer to the approximate width size (bits / bytes) of one or more memory units. As used herein, "narrow" may refer to one or more memory units each having a size or width less than 16 bits, and "wide" may refer to one or more memory units each having a size or width less than 64 bits.

[0030] The mesh bus 126 is connected to the CSR bus 122, the command bus 124, and the ring bus 128. (described below). As shown in FIG. 1, mesh bus 126 provides a communication path that couples or connects each tile to its corresponding neighboring tiles in both the X and Y dimensions. In various implementations, mesh bus 126 can be used to transport input activation amounts between one or more narrow memory units in adjacent tiles. As shown, mesh bus 126 does not allow input activation data to be directly transferred to non-adjacent tiles.

[0031] In various implementations, the mesh bus 126 and the various tiles connected via the mesh bus 126 may have the following configurations: The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge, non-corner tiles have four inbound ports and four outbound ports. Generally, in an example NxN tile layout, edge tiles are tiles with no more than three neighboring tiles, and corner tiles are tiles with two neighboring tiles. With regard to the dataflow methodology over the mesh bus 126, generally, all input activations arriving via the mesh bus 126 for a particular tile must be committed to one or more narrow memory units of that tile. Furthermore, for tile configurations with fewer than four inbound ports, a DMAOp instruction may write zero values ​​to locations in the tile's narrow memory instead of waiting for data on nonexistent input ports. Similarly, for tile configurations with fewer than four outbound ports, DMAOp instructions do not perform narrow memory reads and port writes associated with transfers to non-existent ports.

[0032] In some implementations, the location or address of the narrow memory unit to which a particular input activation will be written or read will be generated by a tensor traversal unit (hereinafter "TTU") based on inbound / outbound DMAOps provided via mesh bus 126. The inbound and outbound DMAOps may be executed simultaneously, with any necessary synchronization being managed by a synchronization flag control scheme managed by controller 102. The TTU is described in further detail below with reference to Figures 2 and 3.

[0033] The ring bus 128 provides communications that originate from the controller 102 and couple through bus data paths that connect the tiles 112, 114 in a ring-like fashion back to the controller 102, similar to the CSR bus 122 and the instruction bus 124. In various implementations, the ring bus 128 generally connects or couples all of the wide memory units (described in more detail below with reference to FIG. 2) within all of the tiles 112, 114. Thus, the payload width of the ring bus 128 corresponds to the width of the wide memory units located within each tile of the tile set 112, 114. As mentioned above, the ring bus 128 also includes a bitmap header that indicates which tiles should consume payload data, including instructions or parameters, communicated over the ring bus 128.

[0034] With respect to data (i.e., payload) received at a particular tile via ring bus 128, in response to receiving the information, each tile zeros (i.e., clears) the position data indicated in the bitmap header specific to the receiving tile before forwarding the data to another tile. Thus, when the header bitmap does not have any remaining bits set data indicating the particular tile that is to receive the payload, forwarding of the payload to another tile will stop. Payload data generally refers to the activations and weights used by one or more tiles during tensor computations performed based on the execution of deeply nested loops.

[0035] In some implementations, the controller 102 may be described as part of the ring bus 128. In one example, for a DMAOp instruction executed within a particular tile, the controller 102 may be used to pop data / payload from a ring bus stop and forward the payload to a ring bus stop in the next tile in the ring. The controller 102 may also commit the payload data to one or more wide memory units of the tile if required by an instruction in the bitmap header. The addresses of the one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.

[0036] In various implementations, each tile in tile set 112, 114 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over ring bus 128 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives data, writes it to one or more wide memory units within the tile, and forwards the payload data for consumption by one or more other tiles. With regard to the movement of payload data over ring bus 128, there is typically only one producer / master of data on ring bus 128 at any given time. The DMAOp instruction execution order (e.g., a FIFO control scheme) among all tiles will ensure that there is only one producer / master of data on ring bus 128 at a given time.

[0037] In some implementations, the controller 102 uses a synchronization flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 128 at a given time. In one example, each write by a tile to a ring output will trigger an increment of a corresponding synchronization flag count. The controller 102 can examine the payload data to determine the number of data chunks or segments that comprise the payload. The controller 102 then monitors execution by a tile to ensure that the expected number of data segments is transferred and / or consumed by that tile before another tile executes in master mode.

[0038] An exception to ensuring that there is only one producer / master of data on ring bus 128 at a given time occurs when there are local multicast groups connected via ring bus 128 that do not have overlapping regions on ring bus 128. For example, Tile 0 (master) may multicast (i.e., generate data) to a tile in the Tile 0-Tile 3 grouping, and Tile 4 (master) may do the same for a tile in the Tile 4-Tile 7 grouping. A key requirement of this dual-master multicast method is that different multicast groups cannot see each other's data packets, as this could result in packet duplication and lead to one or more data calculation errors.

[0039] 1, the controller 102 provides communication data paths that couple or connect the tiles in the tile sets 112, 114 to the I / O 110, and includes several core functions. The core functions of the controller 102 generally include providing one or more I / O input activations to the tiles in the tile sets 112, 114, providing one or more input activations and parameters received from the I / O 110 to the tiles, providing one or more instructions received from the I / O 110 to the tiles, sending I / O output activations to the host interface 108, and acting as a ring stop for the CSR bus 122 and the ring bus 128. As described in more detail below, the first tile set 112 and the second tile set 114 each have an inner loop and an outer loop. It contains multiple tiles used to perform one or more tensor computations that are performed based on a deep loop nest of loops.

[0040] The system 100 generally operates as follows: The host interface 108 provides the controller 102 with one or more instructions that define the direct memory access operations (DMAOps) to occur for a given computation. Descriptors associated with the instructions provided to the controller 102 will contain information needed by the controller to facilitate large-scale dot product computations involving multidimensional data arrays (tensors). Generally, the controller 102 receives input activations, tile instructions, and model parameters (i.e., weights) from the host interface 108 to perform tensor computations for a given layer of the neural network. The controller 102 can then multicast the instructions to the tiles 112, 114 in a dataflow manner defined by the instructions. As described above, a tile that consumes an instruction can then initiate broadcasting new / subsequent instructions to another tile based on the bitmap data in the instruction header.

[0041] In terms of data flow, input activations and parameters are sent to tiles in the tile set 112, 114 via the ring bus 128. Each of the tiles 112, 114 will store a subset of the input activations needed to compute the subset of output activations assigned to that particular tile. A DMAOp instruction to a tile moves the input activations from wide memory to narrow memory. A computation within a tile begins when the necessary input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) become available to the tile. A computation occurring within a tile ends when the MAC operator (described below) within the tile has completed all dot product operations defined by the instruction set and the pre-activation function has been applied to the results of the multiplication operations (i.e., the output activations).

[0042] The results of one or more tensor computations include writing the output activations of the computational layer to the narrow memory unit of the tile performing the computation. Some tensor computations result in the transfer of output edge activations to neighboring tiles via the mesh bus 126. Transferring output edge activations to neighboring tiles is required to calculate output activations for subsequent layers when the computation spans multiple layers. Once the computations for all layers are complete, a DMAOp moves the final activations to the classifier tile 116 via the ring bus 128. The controller 102 then reads the final activations from the classifier tile 116 and executes a DMAOp to move the final activations to the host interface 108. In some implementations, the classifier portion 116 performs the computations for the output layer (i.e., the final layer) of the NN. In other implementations, the output layer of the NN is a classifier layer, a recurrent layer, or one of another layer types commonly associated with neural networks.

[0043] FIG. 2 illustrates an exemplary neural network (NN) computational tile 200. In general, the exemplary tile 200 may correspond to any tile within the first tile set 112 and the second tile set 114 described above with reference to FIG. 1. In various implementations, the computational tile 200 may also be referred to as or referred to as a computational unit 200. Each computational tile 200 is a self-contained computational unit configured to execute instructions independently of other corresponding tiles within the tile sets 112, 114. As briefly described above, each computational tile 200 executes two types of instructions: TensorOp instructions and DMAOp instructions. Generally, each instruction type involves computational operations related to deep loop nests; therefore, each instruction type will generally be executed over multiple time epochs to ensure completion of all loop iterations.

[0044] As discussed in more detail below, different instruction types are managed within the computational tile 200. The instruction types are executed by independent control units within the computational tile 200 that synchronize on data via synchronization flag control provided by the computational tile 200. The synchronization flag control manages concurrency between the execution of different instruction types within the computational tile 200. Each computational operation associated with each instruction type is executed in strict issue order (i.e., first-in-first-out). For two instruction types, TensorOP and DMAOp, there are no ordering guarantees between these different instruction types, and each type is treated as a separate control thread by the computational tile 200.

[0045] With respect to the data flow configuration, the computational tile 200 generally includes a data path 202 and a data path 205, each of which provides a communication path for data flow into and out of the computational tile 200. As described above, the system 100 includes three different data bus structures laid out in a ring configuration: the CSR bus 122, the instruction bus 124, and the ring bus 128. With reference to FIG. 2 , the data path 205 corresponds to the instruction bus 124, and the data path 202 generally corresponds to one of the CSR bus 122 and the ring bus 128. As shown, the data path 202 includes a ring output 203 that provides an output path for data exiting the computational tile 200, and a ring input 204 that provides an input path for data entering the computational tile 200.

[0046] Compute tile 200 further includes TensorOp control 206, which includes TensorOp tensor traversal units (TTUs) 226, and DMAOp control 208, which includes DMAOpTTUs 228. TensorOp control 206 generally manages writes to and reads from TensorOpTTU registers 232, managing traverse operations for execution by TensorOpTTU 226. Similarly, DMAOp control 208 generally manages writes to and reads from DMAOpTTU registers 234, managing traverse operations for execution by DMAOpTTU 228. TTU registers 232 include an instruction buffer for storing one or more instructions, including operations to be performed by TensorOpTTU 226 upon execution of the instruction by TensorOp control 206. Similarly, the TTU registers 234 include an instruction buffer for storing one or more instructions containing operations to be performed by the TTU 228 upon execution of the instruction by the DMAOp control 208 .

[0047] As described further below, TTUs 226 and / or 228 are used by computation tiles 200 to traverse array elements of one or more tensors that typically reside in narrow memory 210 and wide memory 212. In some implementations, TTUs 226 are used by TensorOp control 206 to provide tensor operations for traversing dimensions of multidimensional tensors based on the execution of deep loop nests.

[0048] In some implementations, an instruction for execution by a computational tile 200 arrives at the tile via data path 205 (i.e., part of instruction bus 124). The computational tile 200 examines a header bitmap to determine the instruction type (TensorOp or DMAOp) and instruction subtype (read operation or write operation). Instructions received by the computational tile 200 are then written to a particular instruction buffer depending on the instruction type. Generally, instructions are received and stored (i.e., written to a buffer) prior to execution of the instructions by components of the computational tile 200. As shown in FIG. 2, the instruction buffers (i.e., TensorOpTTU register 232 and DMAOpTTU register 234) can each include a first-in, first-out (FIFO) control scheme to prioritize consumption (execution) of one or more associated instructions.

[0049] As briefly mentioned above, tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. Algorithms that include deeply nested loops may be executed by computational tiles 200 to generate a result that is the result of one or more nested loops. Tensor computations may be performed by iterating through loops to traverse an N-dimensional tensor. In one exemplary computation process, each loop in a loop nest may be responsible for traversing a particular dimension of the N-dimensional tensor. As described herein, TensorOp control 206 generally manages one or more tensor operations that drive the sequence of traversing and accessing dimensional elements of a particular tensor construct to complete the computation defined by the deeply nested loops.

[0050] Computational tile 200 further includes narrow memory 210 and wide memory 212. The narrow and wide designations generally refer to the width size (bits / bytes) of the memory units of narrow memory 210 and wide memory 212. In some implementations, narrow memory 210 includes memory units each having a size or width of less than 16 bits, and wide memory 212 includes memory units each having a size or width of less than 32 bits. Generally, computational tile 200 receives input activations via data path 205, and DMA control 208 performs operations to write the input activations to narrow memory 210. Similarly, computational tile 200 receives parameters (weights) via data path 202, and DMA control 208 performs operations to write the parameters to wide memory 212. In some implementations, narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine which control unit (e.g., TensorOp control 206 or DMAOp control 208) is allowed to access the shared memory unit of narrow memory 210 for each memory cycle.

[0051] The computational tile 200 further includes an input activation bus 216 and a MAC array 214 including multiple cells, each including a MAC operator 215 and a sum register 220. Generally, the MAC array 214 performs tensor computations, including arithmetic operations involving dot product calculations, using the MAC operators 215 and sum registers 220 across multiple cells. The input activation bus 216 provides a data path along which input activations are provided by the narrow memory 210, one for each access by each MAC operator 215 of the MAC array 214. Thus, based on the broadcast of the input activations one by one, a single MAC operator 215 of a particular cell will each receive the input activation. The arithmetic operations performed by the MAC operators of the MAC array 214 generally involve multiplying the input activations provided by the narrow memory 210 with parameters accessed from the wide memory 212 to generate a single output activation value.

[0052] During an arithmetic operation, partial sums may be accumulated and stored in corresponding, e.g., sum registers 220, or written to wide memory 212 and re-accessed by specific cells of MAC array 214 to complete a subsequent multiplication operation. A tensor computation can be described as having a first and a second part. The first part is completed when the multiplication operation produces an output activation, e.g., by completing the multiplication of the input activations with a parameter to produce the output activation. The second part involves the application of a nonlinear function to the output activation, and the second part is completed when the output activation is written to narrow memory 210 after the function is applied.

[0053] Computational tile 200 further includes an output activation bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 indicating core attributes of the components within computational tile 200. While reference map 230 is shown for clarity, it is not included in computational tile 200. Core attributes include whether a particular component is a unit, a storage device, an operator, a control device, or a data path. Generally, once the first portion of the tensor computation is complete, output activations are provided from MAC array 214 to NLU 222 via output activation bus 218. After arriving at NLU 222, data specifying an activation function, received via activation pipeline 224, is applied to the output activations, and the output activations are then written to narrow memory 210. In some implementations, the output activation bus 218 includes at least one pipelined shift register 236, and completing the second portion of the tensor computation includes shifting the output activations toward the narrow memory 210 using the shift register 236 of the activation bus 218.

[0054] For example, for a dot product calculation of two multidimensional data arrays for a single computational tile 200, the MAC array 214 provides robust single instruction, multiple data (SIMD) capabilities. SIMD generally means that all parallel units (multiple MAC operators 215) share the same instruction (based on deep loop nesting), but each MAC operator 215 executes the instruction on different data elements. In one basic example, adding arrays [1,2,3,4] and [5,6,7,8] element-by-element to obtain array [6,8,10,12] in one cycle typically requires four arithmetic units to perform the operation on each element. By using SIMD, the four units share the same instruction (e.g., "add") and can perform the calculation in parallel. Because the instruction is shared, the requirements for instruction bandwidth and instruction memory are reduced, thereby improving efficiency. Thus, the system 100 and computational tile 200 provide improved acceleration and parallelism in tensor computations over conventional methods.

[0055] In one example, and as described in more detail below, a single instruction may be provided by the controller 102 to multiple computational tiles 200 (see tile sets 112, 114 in FIG. 1 ) for consumption by multiple MAC arrays 214. In general, a neural network layer may include multiple output neurons, which may be partitioned such that tensor computations associated with a subset of the output neurons can be assigned to a particular tile of the tile set 112, 114. Each tile of the tile set 112, 114 may then perform the associated tensor computations on a different group of neurons for a given layer. Thus, the computational tiles 200 may provide at least two forms of parallel processing: 1) one form involving partitioning output activations (corresponding to a subset of output neurons) among multiple tiles of the tile set 112, 114; 2) another form involving simultaneous computation (with a single instruction) of multiple subsets of output neurons based on the partitioning among tiles of the tile set 112, 114.

[0056] FIG. 3 shows an exemplary tensor traversal unit (TTU) structure 300 containing four tensors to track, each with a depth of eight. The TTU 300 generally includes a counter tensor 302, a stride tensor 304, an initial tensor 306, and a limit tensor 308. The TTU 300 further includes an adder bank 310 and a tensor address index 312. As mentioned above, tensors are multidimensional geometric objects, and to access elements of a tensor, an index for each dimension must be given. Because tensors are stored in narrow memory 210 and wide memory 212, a set of tensor indices must be converted to a set of memory addresses. In some implementations, the conversion of an index to a memory address is performed by linearly combining the memory address with the indices and reflecting the address via the tensor address index 312.

[0057] There is a TTU for each thread of control, and a control thread for each instruction type (TensorOp and DMAOp) in the computation tile 200. Thus, as described above, there are two sets of TTUs in the computation tile 200: 1) TensorOpTTU 226; and 2) DMAOpTTU 228. In various implementations, the TensorOp control 206 causes the TTU 300 to load the TensorOpTTU counter 302, limit 308, and stride value 304 at the start of a particular tensor operation, and does not change the register values ​​before the instruction is retired. Each of the two TTUs accesses the following memory address ports in the computation tile 200: In this case, addresses will need to be generated for: 1) the wide memory 212 address port, and 2) the narrow memory 210, which has four independent, arbitrated banks presented as four address ports.

[0058] As mentioned above, in some implementations, narrow memory 210 may include a memory arbiter typically used in shared memory systems to determine, for each memory cycle, which control unit (e.g., TensorOp control 206 or DMAOp control 208) is permitted to access the shared memory resources of narrow memory 210. In one example, different instruction types (TensorOp and DMAOp) are independent control threads requesting memory access that must be arbitrated. When a particular control thread commits a tensor element to memory, that control thread increments a counter 302 of tensor references committed to memory.

[0059] In one example, when TensorOp control 206 executes an instruction to access a particular element of a tensor, TTU 300 can determine the address of the particular element of the tensor, and control 206 can access storage, such as narrow memory 210, to retrieve data representing the activation value of the particular element. In some implementations, a program can include nested loops, and control 206 can execute instructions to access elements of a two-dimensional array variable within the nested loop according to a current index variable value associated with the nested loop.

[0060] The TTU 300 may simultaneously maintain traversal state for up to X number of TTU rows for a given tensor. Each tensor simultaneously resident in the TTU 300 occupies a dedicated hardware tensor control descriptor. The hardware control descriptor can consist of X number of TTU counters 302 per row position, a stride 304, and a limit register 308 supporting tensors with up to X number of TTU counters per row dimension. In some implementations, the number of rows and the number of counters per row may vary.

[0061] For a given position register, the final memory address is calculated from an addition operation involving adding the position registers together. The base address is incorporated into counter 302. One or more adders are shared for tensor references that reside in the same memory. In one implementation, since there can only be a single load / store on any given port in a cycle, it is the function of the loop nesting control to ensure that multiple tensor references that reside in the same narrow or wide memory do not have their counters incremented in any given cycle. The use of registers to calculate memory access address values, including determining offset values, is described in patent application Ser. No. 15 / 014,265, entitled "Matrix Processing Apparatus," filed February 3, 2016. No. 6,399,433, the entire contents of which are expressly incorporated herein by reference.

[0062] For example, when a software algorithm processes an N-dimensional tensor, nested loops may be used, with each loop responsible for traversing a dimension of the N-dimensional tensor. The multidimensional tensor may be a matrix or a multidimensional matrix. Each dimension of the N-dimensional tensor may contain one or more elements, and each element may store a respective data value. For example, a tensor may be a variable in a program, and a variable may have three dimensions. The first dimension may be 300 elements long, the second dimension may be 1000 elements long, and the third dimension may be 20 elements long.

[0063] Traversing a tensor in a nested loop may require calculating the memory address values ​​of elements in order to load or store the corresponding data values ​​of the elements. For example, a for loop is a nested loop, and is tracked by three loop index variables: A three-dimensional tensor can be traversed using three nested loops. In some cases, the processor may need to perform loop boundary conditions, such as setting the loop bounds of an inner loop with an outer loop index variable. For example, when determining whether to exit the innermost loop of a set of nested loops, the program may compare the current value of the loop index variable of the innermost loop with the current value of the loop index variable of the outermost loop of the set of nested loops.

[0064] Generally, when a processing unit of a computational tile executes an instruction to access a particular element of a tensor, the tensor traversal unit determines the address of the particular element of the tensor, allowing the processing unit to access a storage medium (memory) and read data representing the value of the particular element. For example, a program may include nested loops, and the processing unit may execute an instruction to access an element of a two-dimensional array variable within the nested loop according to a current index variable value associated with the nested loop. Based on the current index variable value associated with the nested loop, the tensor traversal unit may determine an offset value representing an offset from a first element of the two-dimensional array variable. The processing unit then uses the offset value to access the particular element of the two-dimensional array variable from memory.

[0065] The following provides template parameters that may be used to instantiate a specialized TTU 300: 1) X number of TTU rows; 2) X number of TTU counters per row; 3) X number of TTU adder units; 4) for each TTU row, indicate the shared adder reference; and 5) for each counter, indicate X counter size [TTU][row][depth]. All TTU registers are architecturally visible. The address of the particular tensor element that needs to be accessed for a computation (i.e., tensor address 312) is the result of the counter addition. When an increment signal is issued from the controlling thread to a TTU row, TTU 300 performs a single cycle operation, incrementing the innermost dimension by that dimension's stride 304 and propagating the rollover through all depths.

[0066] Generally, the TTU 300 determines state associated with one or more tensors. The state may include loop bound values, a current loop index variable value, a dimension multiplier for calculating memory address values, and / or a program counter value for processing branch loop bounds. The TTU 300 may include one or more tensor state elements and an arithmetic logic unit. Each of the tensor state elements may be a storage element, such as a register or any other suitable storage circuit. In some implementations, the tensor state elements may be physically or logically organized into distinct groups, as described in further detail in patent application Ser. No. 15 / 014,265.

[0067] FIG. 4 illustrates an exemplary architecture including a narrow memory 210 that broadcasts activations 404 to one or more multiply-accumulate (MAC) operators via an input bus 216. A shift register 404 provides a shifting function whereby activations 404 are sent out one at a time onto the input bus 216 for receipt by one or more MAC operators 215 within the MAC cell 410. In general, a MAC cell 410 including a MAC operator 215 can be defined as a computational cell that computes partial sums and, in some implementations, is configured to write the partial sum data to an output bus 218. As shown, the cell 410 may consist of one or more MAC operators. In one implementation, the number of MAC operators 215 within the MAC cell 410 is referred to as the issue width of the cell. As an example, a dual-issue cell refers to a cell with two MAC operators that can compute the multiplication of two activation values ​​(from the narrow memory 210) by two parameters (from the wide memory 212) and perform an addition between the results of the two multipliers and the current partial sum.

[0068] As mentioned above, input bus 216 is a broadcast bus that provides input activations to MAC operators 215 of a linear unit (i.e., MAC array 214). In some implementations, the same input is shared among all MAC operators 215. The width of input bus 216 must be wide enough to supply the broadcast inputs to a corresponding number of cells for a given MAC array 214. Consider the following example to illustrate the structure of input bus 216. When the number of cells in the linear unit equals four and the activation width equals eight bits, input bus 216 can be configured to provide up to four input activations per cycle. In this example, all cells in MAC array 214 will access only one of the four broadcast activations.

[0069] Based on the TensorOp field settings of instructions received by the computational tile 200, cells of the MAC array 214 may need to perform a computation using the same input activations. This may be referred to as Zout partitioning within a cell of the MAC array 214. Similarly, Zin partitioning within a cell occurs when cells of the MAC array 214 require different activations to perform a computation. In the former case, a single input activation is replicated four times, and four activations read from the narrow memory 210 are broadcast over four cycles. In the latter case, a read of the narrow memory 210 is required every cycle. In the above example, the TensorOp control 206 orchestrates this broadcast method based on the execution of instructions received from the controller 102.

[0070] FIG. 5 shows an exemplary architecture including an output bus 218 for providing output activations to the narrow memory units 210 of FIGS. 2 and 4. Generally, each MAC cell 215 of the MAC array 214 in the computational tile 200 computes a different output activation. However, if the output feature depth for the output feature array is less than the number of MAC cells 215 in the computational tile 200, the cells may be grouped to form one or more cell groups. All MAC cells 215 in a cell group compute the same output (i.e., for the output feature map), but each cell only computes a subset of the outputs, corresponding to a subset of the Z dimensions. As a result, the outputs of the MAC cells 215 are now partial sums rather than a final linear output. In some implementations, the NLU 222 aggregates these partial sums into a final linear output based on control signals provided to the NLU 222 by the NLU control 238.

[0071] As mentioned above, the output bus 218 is a pipelined shift register. In various implementations, once the first portion of the tensor computation is finished and the TensorOp control 206 indicates (by executing an instruction) that the partial sums need to be written out, there will be parallel loads of partial sums provided to the output bus 218. The number of parallel loads will correspond to the number of MAC cells in the computation tile 200. The TensorOp control 206 then causes the partial sum amounts to be shifted out and sent through the nonlinear pipeline. In some implementations, there may be situations where not all MAC cells in a tile are actually utilized to perform the computation. In such situations, not all partial sums shifted onto the output bus will be valid. In this example, the TensorOp control 206 may provide control signals to the MAC array 214 to indicate the number of valid cells to be shifted out. The parallel load amount loaded onto the output bus 218 still corresponds to the number of MAC cells in the computation tile, but only valid values ​​will be shifted out and committed to the narrow memory 210.

[0072] 6 is an exemplary flowchart of a process 600 for performing tensor computations using the neural network computational tile 200 of FIG. 2. The process 600 begins at block 602, where the narrow memory 210 of the computational tile 200 sends (i.e., broadcasts) activations one by one onto the input activation data bus 216. The activation values ​​are stored in the narrow memory 210. 0. Narrow memory 210 may be a collection of static random access memory (SRAM) banks that permit addressing of specific memory locations to access input quantities. Activations read from narrow memory 210 are broadcast via input activation bus 216 to linear cells of MAC array 214 (i.e., a linear unit) that include multiple MAC operators 215 and sum registers 220. At block 604 of process 600, MAC operators 215 of computational tile 200 each receive two inputs: one input (activation) received from input activation bus 216; another input (parameter) received from wide memory 212. Thus, activations feed one of the inputs of each MAC operator 215, and each MAC operator 215 in a cell of MAC array 214 obtains its second multiplier input from wide memory 212.

[0073] In block 606 of process 600, the MAC array 214 of the computational tile 200 performs tensor computations, including dot product computations, based on elements of the data array structure accessed from memory. The wide memory 212 has a width in bits equal to the width of the linear unit (32 bits). Thus, the linear unit (LU) is a SIMD vector arithmetic logic unit (ALU) unit that receives data from the vector memory (i.e., the wide memory 212). In some implementations, the MAC operator 215 may also obtain an accumulator input (partial sum) from the wide memory 212. In some implementations, there is time sharing for the wide memory 212 ports for reading and / or writing for two different operands (parameters and partial sums). Generally, to optimize area, the wide memory 212 may have a limited number of ports. As a result, if an operand (e.g., a parameter) needs to be read from the wide memory 212 and an operand (e.g., a partial sum) needs to be written to the wide memory 212 at the same time, the pipeline associated with a particular operand may stall.

[0074] At block 608, the computational cell (having a MAC operator 215 and sum register 220) of the computational tile 200 generates at least one output activation based on the multiplication operation performed by the MAC / computational cell. The result of the MAC cell operation includes either a partial sum that is written back to wide memory (during a partial sum arithmetic operation) or an output activation that is sent to the output bus 218. At block 610, the NLU 222 of the computational tile 200 applies a nonlinear activation function to the output activations and then writes the activations to the narrow memory 210. In some implementations, the output bus 218 is a shift register that can accumulate parallel loads of results / output activations from the MAC operator 215 but shift them out one at a time for the application of the nonlinear function and write operation to the narrow memory 210 of the same tile.

[0075] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Alternatively, or in addition, the program instructions may be encoded on an artificially generated, propagated signal, such as, for example, a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0076] The processes and logic flows described herein operate on input data and generate output. The processes and logic flows may also be performed by, and apparatus may be realized by, special purpose logic circuitry such as an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0077] A processor suitable for executing a computer program may be based, by way of example, on a general-purpose or special-purpose microprocessor, or both, or on any type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices.

[0078] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0079] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of possible features specific to particular embodiments of particular inventions. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination, and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0080] Similarly, while operations are shown in a particular order in the figures, it should not be understood that such operations need to be performed in the particular order shown, or sequential order, or that all of the shown operations need to be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0081] Other implementations are summarized in the examples below. Example 1: A computation unit for accelerating tensor computations, the computation unit comprising: a first memory bank having a first data width for storing at least one of input activations or output activations; and a second memory bank having a second data width greater than the first data width and used in performing the computations. a second memory bank for storing one or more parameters to be calculated; at least one cell including at least one multiply-accumulate ("MAC") operator that receives the parameters from the second memory bank and performs a calculation; and a first traversal unit in data communication with at least the first memory bank, the first traversal unit configured to provide control signals to the first memory bank to provide input activations to a data bus accessible by the MAC operator, the calculation unit performing one or more calculations related to at least one element of the data array, the one or more calculations performed by the MAC operator and including, in part, a multiplication operation between the input activations received from the data bus and the parameters received from the second memory bank.

[0082] Example 2: The computation unit described in Example 1, wherein the computation unit performs one or more of the computations by executing a loop nest including multiple loops, the structure of the loop nest including respective loops used by the first traversal unit to traverse one or more dimensions of the data array.

[0083] Example 3: The computation unit of Example 2, wherein the one or more computations are performed based in part on tensor operations provided by the first traversal unit, the tensor operations including a loop nest structure for accessing one or more elements of a data array.

[0084] Example 4: The computing unit of any one of Examples 1 to 3, further comprising: a second traversal unit configured to access at least one memory location of the first memory bank and at least one memory location of the second memory bank based on instructions received from a source external to the computing unit.

[0085] Example 5: The computation unit of Example 4, wherein the first traversal unit is a tensor operation traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor including multiple elements.

[0086] Example 6: The computation unit of one of Examples 1 to 5, wherein the computation unit includes a nonlinear unit, a first portion of the computation includes generating one or more output activations based on a multiplication operation, and a second portion of the computation includes applying a nonlinear function to the one or more output activations by the nonlinear unit.

[0087] Example 7: The computation unit of Example 6, wherein the one or more computations performed by the computation unit include using a shift register to shift output activations to the first memory bank.

[0088] Example 8: The computing unit of any one of Examples 1 to 8, further comprising a portion of a ring bus extending outside the computing unit, the ring bus providing a data path between the first memory bank and a memory bank of another adjacent computing unit and between the second memory bank and a memory bank of another adjacent computing unit.

[0089] Example 9: The computational unit of any one of Examples 1 to 8, wherein the second memory bank is configured to store at least one of the partial sums or the one or more pooling layer inputs.

[0090] Example 10: A computer-implemented method for accelerating tensor computations, comprising: sending a first input activation by a first memory bank having a first data width in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation The data array is provided by a data bus accessible by at least one cell of the arithmetic unit, the method further comprising receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell including at least one multiply-accumulate ("MAC") operator, the method further comprising performing, by the MAC operator, one or more calculations related to at least one element of the data array, the one or more calculations including, in part, a multiplication operation between at least a first input activation accessed from the data bus and at least one parameter received from the second memory bank.

[0091] Example 11: The computer-implemented method of Example 10, wherein the one or more computations are performed based in part on the computation unit executing a loop nest including multiple loops, the structure of the loop nest including respective loops used by the first traversal unit to traverse one or more dimensions of the data array.

[0092] Example 12: The computer-implemented method of Example 11, further comprising providing, by the first traversal unit, a tensor operation including a loop nest structure for accessing one or more elements of the data array.

[0093] Example 13: The computer-implemented method of one of Examples 10 to 12, wherein the first traversal unit is a tensor arithmetic traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor including multiple elements.

[0094] Example 14: The computer-implemented method of one of Examples 10-13, further comprising performing a first portion of the one or more calculations by generating at least one output activation based on a multiplication operation.

[0095] Example 15: The computer-implemented method of Example 14, further comprising performing a second portion of the one or more calculations by applying a non-linear function to the one or more output activations.

[0096] Example 16: A non-transitory computer-readable storage medium including instructions executable by one or more processors, which instructions, when executed, cause the one or more processors to perform operations, the operations including sending a first input activation by a first memory bank having a first data width in response to the first memory bank receiving a control signal from a first traversal unit, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit, the operations further including receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell including at least one multiply-accumulate ("MAC") operator, the operations further including performing, by the MAC operator, one or more calculations related to at least one element of a data array, the one or more calculations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and the at least one parameter received from the second memory bank.

[0097] Example 17: The non-transitory computer-readable storage medium of Example 16, wherein the one or more calculations are performed based in part on the calculation unit executing a loop nest including multiple loops, the structure of the loop nest including respective loops used by the first traversal unit to traverse one or more dimensions of the data array.

[0098] Example 18: The non-transitory computer-readable storage medium of Example 17, further comprising providing, by the first traversal unit, a tensor operation including a loop nest structure to access one or more elements of the data array.

[0099] Example 19: The non-transitory computer-readable storage medium of one of Examples 16-18, further comprising performing a first portion of the one or more calculations by generating at least one output activation based on a multiplication operation.

[0100] Example 20: The non-transitory computer-readable storage medium of Example 19, further comprising performing a second portion of the one or more calculations by applying a non-linear function to the one or more output activations.

[0101] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computational unit for accelerating tensor computations, comprising: a first memory bank having a first data width for storing at least one of input activations or output activations; a second memory bank having a second data width greater than the first data width for storing one or more parameters used in performing the calculation; at least one cell including at least one multiply-accumulate ("MAC") operator that receives parameters from the second memory bank and performs a calculation; a first traversal unit in data communication with at least the first memory bank, the first traversal unit configured to provide control signals to the first memory bank to provide input activations to a data bus accessible by the MAC operator; The computation unit performs one or more computations related to at least one element of a data array, the one or more computations being performed by the MAC operator and including, in part, a multiplication operation between the input activation received from the data bus and a parameter received from the second memory bank.

2. 2. The computation unit of claim 1, wherein the computation unit performs one or more of the computations by executing a loop nest including a plurality of loops, the structure of the loop nest including respective loops used by the first traversal unit to traverse one or more dimensions of the data array.

3. 3. The computation unit of claim 2, wherein the one or more computations are performed based in part on tensor operations provided by the first traversal unit, the tensor operations including a loop nest structure for accessing one or more elements of the data array.

4. 2. The computing unit of claim 1, further comprising: a second traversal unit configured to access at least one memory location of the first memory bank and at least one memory location of the second memory bank based on instructions received from a source external to the computing unit.

5. 5. The computation unit of claim 4, wherein the first traversal unit is a tensor arithmetic traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor containing multiple elements.

6. 2. The computational unit of claim 1, wherein the computational unit includes a nonlinear unit, a first portion of the computation includes generating one or more output activations based on the multiplication operations, and a second portion of the computation includes applying a nonlinear function to the one or more output activations by the nonlinear unit.

7. The computation unit of claim 6 , wherein the one or more computations performed by the computation unit include using a shift register to shift the output activations into the first memory bank.

8. The memory device further includes a portion of a ring bus extending outside the computing unit, the ring bus providing a data path between the first memory bank and a memory bank of another adjacent computing unit, and between the second memory bank and a memory bank of another adjacent computing unit. The computing unit of claim 1 .

9. The computational unit of claim 1 , wherein the second memory bank is configured to store at least one of partial sums or one or more pooling layer inputs.

10. 1. A computer-implemented method for accelerating tensor computations, comprising: a first memory bank having a first data width, in response to receiving a control signal from a first traversal unit, sending a first input activation by the first memory bank, the first memory bank being disposed within a computation unit, the first input activation being provided by a data bus accessible by at least one cell of the computation unit, the method further comprising: receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell including at least one multiply-accumulate ("MAC") operator; 11. A computer-implemented method for accelerating tensor computations, comprising: performing, by the MAC operator, one or more computations related to at least one element of a data array, the one or more computations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank.

11. 11. The computer-implemented method of claim 10, wherein the one or more computations are performed in part based on the computation unit executing a loop nest including multiple loops, the structure of the loop nest indicating how the first traversal unit traverses one or more dimensions of the data array.

12. 12. The computer-implemented method of claim 11, further comprising providing, by the first traversal unit, a tensor operation including a loop nest structure for accessing one or more elements of the data array.

13. 11. The computer-implemented method of claim 10, wherein the first traversal unit is a tensor arithmetic traversal unit, the second traversal unit is a direct memory access traversal unit, and the data array corresponds to a tensor containing multiple elements.

14. 11. The computer-implemented method of claim 10, further comprising performing a first portion of the one or more calculations by generating at least one output activation based on the multiplication operation.

15. 15. The computer-implemented method of claim 14, further comprising performing a second portion of the one or more calculations by applying a non-linear function to the one or more output activations.

16. A non-transitory computer-readable storage medium containing instructions executable by one or more processors, the instructions, when executed, causing the one or more processors to perform operations, the operations including: a first memory bank having a first data width, in response to receiving a control signal from a first traversal unit, sending a first input activation by the first memory bank, the first memory bank being disposed within a computation unit, the first input activation being transmitted to a data bus accessible by at least one cell of the computation unit; and said operation is further given by: receiving, by the at least one cell, one or more parameters from a second memory bank having a second data width greater than the first data width, the at least one cell including at least one multiply-accumulate ("MAC") operator, the operations further comprising: performing, by the MAC operator, one or more calculations relating to at least one element of a data array, the one or more calculations including, in part, a multiplication operation between at least the first input activation accessed from the data bus and at least one parameter received from the second memory bank.

17. 17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more computations are performed based in part on the computation unit executing a loop nest including multiple loops, the structure of the loop nest indicating how the first traversal unit traverses one or more dimensions of the data array.

18. 20. The non-transitory computer-readable storage medium of claim 17, further comprising providing, by the first traversal unit, a tensor operation including a loop nest structure for accessing one or more elements of the data array.

19. 17. The non-transitory computer-readable storage medium of claim 16, further comprising performing a first portion of the one or more calculations by generating at least one output activation based on the multiplication operation.

20. 20. The non-transitory computer-readable storage medium of claim 19, further comprising performing a second portion of the one or more calculations by applying a non-linear function to the one or more output activations.

Images