Reallocating tensor elements among machine learning computing units

Abstract

This application relates to reallocating tensor elements among machine learning compute units. Methods, systems, and apparatuses including means for reallocating tensor elements among compute units are described. In one aspect, a method includes allocating tensor elements of an N-dimensional tensor among a plurality of compute units of a computing system. Each compute unit reallocates a subset of the tensor elements previously allocated to the compute unit to the compute unit. Each compute unit accesses reallocation partition data that specifies, for each compute unit, tensor elements to be stored by the compute unit after reallocating the tensor elements. For each tensor element previously allocated to a particular compute unit, the compute unit determines a global linearized index value for the tensor element based on a multi-dimensional index of the tensor element. The compute unit uses the reallocation partition data and the global linearized index value to determine a destination compute unit and sends the tensor element to the destination compute unit.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 202080058657.7, filed on October 7, 2020. Technical Field

[0003] This application relates to the reallocation of tensor elements among machine learning computational units. Background Technology

[0004] A neural network is a machine learning model that uses one or more layers to generate outputs—such as classification—for received inputs. The inputs to a neural network can include a multidimensional tensor containing tensor elements. In addition to the outer layers, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network—that is, the next hidden layer or output layer. Each layer generates its output from the received inputs based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] This specification generally relates to hardware neural network computing units and networks between computing units, which are configured to redistribute tensor elements among computing units.

[0006] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method comprising: distributing tensor elements of an N-dimensional tensor among multiple computational units of a computational system, wherein each computational unit performs computation using a subset of the tensor elements allocated to that computational unit; receiving an instruction to reallocate tensor elements of an N-dimensional tensor among the computational units; and, in response to receiving the instruction, each computational unit reallocating a subset of the tensor elements previously allocated to that computational unit to one or more computational units of the computational system, for each specific computational unit of the computational system, including: accessing the reallocated partitioned data, reallocating the partitioned data... The data segmentation process specifies the tensor elements that will be stored by the computational unit after the tensor elements are reallocated. For each tensor element previously allocated to a particular computational unit: a global linearized index value for the tensor element is determined based on the multidimensional index of the tensor element in the N-dimensional tensor, the multidimensional index of the tensor element including the index value corresponding to the position of the tensor element along that dimension of the N-dimensional tensor for each dimension; the destination computational unit of the computational system to which the tensor element is to be reallocated is determined using the reallocated segmented data and the global linearized index value of the tensor element; and the tensor element is sent to the destination computational unit.

[0007] These and other implementations may each optionally include one or more of the following features. In some aspects, tensor elements of an N-dimensional tensor are redistributed in response to reshaping of the N-dimensional tensor, the reshaping including adjusting the number of tensor elements in two or more dimensions of the N-dimensional tensor. Determining the destination computational unit of the computational system to which the tensor elements are to be redistributed using segmented data and global linearization index values ​​of the tensor elements may include: determining a second multidimensional index of the tensor elements in the reshaped N-dimensional tensor based on the global linearization index values ​​of the tensor elements and the number of tensor elements in each dimension of the reshaped N-dimensional tensor; and determining the destination computational unit to which the tensor elements are to be redistributed based on the multidimensional index of the tensor elements and the redistributed segmented data.

[0008] In some aspects, distributing tensor elements of an N-dimensional tensor among multiple computational units of a computational system includes: partitioning the N-dimensional tensor into multiple tensor slices based on one or more tiled dimensions of the N-dimensional tensor; and assigning one or more tensor slices of the N-dimensional tensor to each computational unit. The tensor elements of the N-dimensional tensor are reallocated in response to a change in one or more tiled dimensions on which the N-dimensional tensor was partitioned.

[0009] In some aspects, sending tensor elements to a destination computation unit may include: generating header information for the tensor elements that is specified for the destination computation unit; passing the header information and the tensor elements to a tile-to-tile network channel managed by a specific computation unit; and having the destination computation unit store the tensor elements in a queue of the specific computation unit, wherein each computation unit includes a corresponding queue for each computation unit of the computation system, and each corresponding queue stores tensor elements received from the corresponding computation unit corresponding to the corresponding queue.

[0010] Some aspects may include, for each computational unit of the computational system, traversing a second subset of tensor elements being reallocated to the computational unit based on reallocation partition data, and for each specific tensor element in the second subset, including: determining a global linearization index value for the specific tensor element; determining the originating computational unit from which the specific tensor element was received based on the global linearization index value of the specific tensor element and the allocation partition data, the allocation partition data specifying for each computational unit the tensor elements to be stored by the computational unit after the allocation of the tensor elements; obtaining the specific tensor element from the corresponding queue of the originating computational unit; and storing the specific tensor element in the local memory of the computational unit.

[0011] In some respects, determining the global linearization index value of a particular tensor element involves determining the global linearization index value based on the multidimensional index of the particular tensor element.

[0012] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. Using a global linearization index value for each tensor element allows computational units of a computational system to efficiently redistribute tensor elements among computational units in response to tensor reshaping or changes in the tiled dimension along which tensors are partitioned between computational units. For example, each computational unit can determine which computational unit owns the tensor element (e.g., stores it and / or uses it to perform computations) based on the global linearization index value of the tensor element. By having the computational unit receiving the tensor element compute the global linearization index value of the received tensor element, less data is transferred between computational units because the global linearization index value does not need to be transferred along with the tensor element. This allows the computational system to use narrower tile-to-tile networks and increases the addressing flexibility of tensor elements. For example, if a tensor comprises thousands of tensor elements, the unique identifier for each tensor element may require more data in the header than the actual payload of the tensor element, which would require a wider tile-to-tile network and more data to be transferred between computational units than using the techniques described in this document. Sending less data also results in faster data transmission, which in turn leads to faster machine learning computations.

[0013] Other embodiments of this and other aspects include corresponding systems, methods, and computer programs configured to perform actions of methods encoded on a computer storage device. A system of one or more computers may be configured in this way by software, firmware, hardware, or combinations thereof installed on the system and causing the system to operate in operation. One or more computer programs may be configured to have instructions that, when executed by a data processing device, cause the device to perform actions. Another embodiment includes a computer storage medium encoded with a computer program that includes instructions, when executed by one or more data processing devices, causing the data processing devices to perform operations including methods according to any aspect or embodiment described herein. The computer storage medium may be a non-transient computer storage medium, but this embodiment is not limited to non-transient computer storage media.

[0014] Details of one or more embodiments of the subject matter of this specification are given in the accompanying drawings and the following description. Other potential features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0015] Figure 1 This is a block diagram of an example computing system.

[0016] Figure 2 The illustration shows an example computational block.

[0017] Figure 3 The diagram illustrates an example tensor traversal unit (TTU) structure.

[0018] Figure 4 The diagram illustrates an example logical structure of a tile-to-tile network for redistributing tensor slices among computational tiles.

[0019] Figure 5 The illustration shows an example layout of a tile-to-tile network used to redistribute tensor slices among computation tiles.

[0020] Figure 6 This is a flowchart of an example process for passing tensor slices between computation tiles.

[0021] In the various figures, the same reference numerals and names denote the same elements. Detailed Implementation

[0022] Generally, the subject matter described in this specification relates to hardware computing systems comprising multiple computing units configured to accelerate machine learning workloads, such as neural network layers. Each computing unit of the hardware computing system is independent and can independently perform the computations required for a given layer of a multi-layer neural network. Although the system and techniques are described primarily in the context of neural networks, these systems and techniques can be used for other workloads that use tensors as input, such as other deep learning models.

[0023] Multi-layered neural networks can be used to compute inferences. For example, given an input, a neural network can compute an inference about that input. The neural network computes this inference by processing the input through each layer of the network. Specifically, each layer of the neural network has its own set of weights. Each layer receives an input and processes that input according to its set of weights to generate an output.

[0024] Therefore, in order to compute inferences from received input, the neural network receives the input and processes it through each neural network layer to generate inferences, where the output from one neural network layer is provided as the input to the next neural network layer. The data input to a neural network layer, such as the input of the neural network or the output of the layer below it in a sequence, can be referred to as the activation input of that layer.

[0025] In some implementations, the layers of a neural network are arranged sequentially. In other implementations, the layers are arranged as a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be arranged such that the output of a layer can be sent back as the input of the previous layer.

[0026] The hardware computing system described in this specification can perform computations of neural network layers by distributing tensor computations across multiple computational tiles. Each computational tile, also referred to for brevity as a "tile," is a separate computational unit that collaborates with other tiles in the system to accelerate computations across one or more layers of a multi-layer neural network. The computation performed within a neural network layer can include multiplying an input tensor, which includes input activations, with a parameter tensor, which includes weights. This computation can include multiplying the input activations with weights over one or more epochs and accumulating the products over many epochs.

[0027] Tensors are multidimensional geometric objects, and examples of multidimensional geometric objects include matrices and data arrays. Typically, software algorithms are executed by computational blocks to perform tensor computations by traversing an N-dimensional tensor through nested loops. In a typical computation, each loop may be responsible for traversing a specific dimension of the N-dimensional tensor. For a given tensor construction, the computational block may need to access elements of a specific tensor to perform dot product calculations associated with that tensor. Computation occurs when an input activation provided by a memory structure, such as a narrow memory structure, is multiplied by parameters or weights provided by a memory structure, such as a wide memory structure. Because tensors are stored in memory, the set of tensor indices may need to be translated to a set of memory addresses. Typically, the tensor traversal unit of the computational block performs a control operation that provides the index of each dimension associated with the tensor and the order in which the index elements are traversed to perform the computation. Tensor computation ends when the multiplication result is written to the output bus and stored in memory.

[0028] To distribute tensor computations across multiple computation tiles, a tensor can be partitioned into multiple tensor slices (which are also tensors, e.g., subtensors) spanning one or more dimensions of the tensor. For example, a tensor can have a shape of [5][4][4][8] for dimensions [w, y, x, z]. In this example, the y and x dimensions can be tiled dimensions along the partitioned tensor. If a tensor is being assigned to tiles arranged in a 4×2 pattern with a total of eight tiles, each tile can receive a tensor slice with a shape of [5][1][2][8]. Each tile can then perform tensor computations using the tensor elements of its tensor slices.

[0029] In some cases, tensors can be reshaped before performing additional computations, or the partitioning scheme (e.g., the tiled dimension of a tensor) can be changed based on a machine learning model. For example, the shape of the [5][4][4][8] tensor can be changed to that of the [5][8][2][8] tensor with the same total number of tensor elements (640 in this example). Based on the new shape or the different tiled dimension, the tensor elements may need to be redistributed among the tiles so that each tile has one or more tensor slices along the tiled dimension. In this example of reshaping, the tensor slices will now be [5][2][1][8]. Due to the new tensor slices, the tensor elements used for each tile (or for at least some tiles) can be different, thus requiring the redistribution of at least some of the tensor elements.

[0030] To reduce the number of hardware instructions required to manage reallocations, each tile can use nested loops to traverse the tensor elements it previously owned and send these tensor elements to other tiles. Similarly, each tile can use nested loops to traverse the tensor elements it receives during reallocations and store these tensor elements in the tile's local memory. Using this nested loop avoids the need for a large number of instructions to orchestrate the reallocation of tensor elements.

[0031] Tiles can use a global linearization index value to determine which tiles have tensor elements or changes in tiled dimensions before and after reshaping. Each tensor element can be associated with (e.g., assigned) a global linearization index value based on the multidimensional index of the tensor elements in the tensor before redistribution, such as the multidimensional index of the tensor elements in the original tensor before any redistribution operation. In some implementations, the global linearization index value of the tensor elements remains the same regardless of how many redistributions occur.

[0032] For each dimension of the tensor, the multidimensional index can include an index value corresponding to the position of the tensor element along the dimension of the tensor. For example, the multidimensional index of a tensor element at w = 2, y = 1, x = 3, and z = 4 could be 2134. As described in more detail below, tiles can convert these indices in the multidimensional index into global linearized index values ​​for the tensor elements, and use the global linearized index values ​​to determine which tile sends the tensor element and / or will (or has) received the tensor element from which tile. For example, even if a reallocation occurs in response to tensor reshaping, the global linearized index value of each element is the same before and after the reallocation.

[0033] Figure 1A block diagram of an example computing system 100 is shown. System 100 can accelerate tensor computations associated with deep neural networks (DNNs) or other deep learning models. System 100 typically includes a controller 102, a host interface 108, input / output (I / O) links 110, multiple tiles including a first tile set 112 and a second tile set 114, a classifier portion 116, and a data bus identified in a bus map 118 (shown for clarity but not included in system 100). Controller 102 typically includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in a computer-readable storage medium. Instruction memory 106 may store one or more machine-readable instructions executable by one or more processors of controller 102. Data memory 104 may be any of various data storage media for storing and subsequently accessing various data related to computations occurring within system 100.

[0034] Controller 102 is configured to execute one or more instructions related to tensor computations within system 100, including instructions stored in instruction memory 106. In some embodiments, data memory 104 and instruction memory 106 are one or more volatile memory cells. In other embodiments, data memory 104 and instruction memory 106 are one or more non-volatile memory cells. Data memory 104 and instruction memory 106 may also be another form of computer-readable medium, such as a floppy disk device, hard disk device, optical disk device, or magnetic tape device, flash memory or other similar solid-state storage device, or a device array including devices in a storage area network or other configuration. In various embodiments, controller 102 may also be referenced to or referred to as kernel manager 102.

[0035] As depicted, host interface 108 is coupled to I / O link 110, controller 102, and classifier section 116. Host interface 108 receives instructions and data parameters from I / O link 110 and provides instructions and parameters to controller 102. Typically, instructions can be provided to one or more devices in system 100 via instruction bus 124 (described below), and parameters can be provided to one or more devices in system 100 via ring bus 128 (described below). In some embodiments, instructions are received by controller 102 from host interface 118 at an initial time and stored in instruction memory 106 for execution by controller 102 at a later time.

[0036] The classifier section 116 is also coupled to block 7 in the controller 102 and the second block set 114. In some embodiments, the classifier section 116 is implemented as a separate block within the system 100. In alternative embodiments, the classifier section 116 is arranged or located within the controller 102 as a sub-circuit or sub-device of the controller 102. The classifier section 116 is typically configured to perform one or more functions on accumulated pre-activation values ​​received as the output of the fully connected layer. The fully connected layer can be segmented on the blocks in the block sets 112 and 114. Thus, each block is configured to produce a subset of pre-activation values ​​(i.e., linear outputs) that can be stored in the memory cells of the block. The classification result bus 120 provides a data path from the classifier section 116 to the controller 102. Data including the post-function values ​​(i.e., results) is provided from the classifier section 116 to the controller 102 via the classification result bus 120.

[0037] Bus mapping 118 illustrates a data bus that provides one or more interconnected data communication paths between tiles in the first tile set 112 and tiles in the second tile set 114. Bus mapping 118 provides illustrations for identifying the classification result bus 120, CSR / master bus 122, instruction bus 124, mesh bus 126, ring bus 128, and tile-to-tile network 129, such as... Figure 1 As shown. Typically, a tile is a core component within the accelerator architecture of system 100 and is the focus of tensor computations occurring within the system. Each tile is a separate computational unit that collaborates with other tiles in the system to accelerate computations across one or more layers of a multi-layer neural network. Although tiles in tile sets 112, 114 may share the execution of tensor computations associated with a given instruction, each separate computational unit is an independent computational component configured to perform a subset of tensor computations independently relative to other corresponding tiles within tile sets 112, 114.

[0038] CSR bus 122 is a single master-slave bus that enables controller 102 to transmit setup program configurations and read one or more instructions associated with status registers of one or more blocks. CSR bus 122 can be connected in a single daisy-chain configuration with one master bus segment and multiple slave bus segments. For example... Figure 1 As shown, CSR bus 122 provides communication coupling via a bus data path that connects the blocks in the block sets 112 and 114 of the ring and the controller 102 to the host interface 110. In some embodiments, the host interface 110 is the single master device of the CSR bus ring, and the entire CSR bus address space is a memory mapped to the memory space in the host interface 110.

[0039] The CSR bus 122 can be used by the host interface 110 to perform one or more operations, including, for example, programming a memory buffer pointer in the controller 102 to enable the controller 102 to begin fetching instructions from the instruction memory 106, updating / programming various tile settings that remain static during one or more computations (e.g., coefficient tables for polynomial approximation computations), and / or loading / reloading firmware into the classification section 116. In one example, firmware reloading may include a new function to be applied to the linear output (i.e., pre-activated value). Therefore, each slave device with access to the CSR bus 122 will have a distinct node identifier (node ​​ID) bound to and identifying that slave device. The node ID will be part of the instruction address and will be used, verified, or otherwise checked by the CSR slave devices (i.e., the controller 102, tiles 112, 114, and classifier 116) to determine whether a CSR packet addresses a slave device.

[0040] In some implementations, one or more instructions may be transmitted via controller 102 from host interface 110. Instructions may be, for example, 32 bits wide, with the first 7 bits including header information indicating the instruction address / destination to be received and executed. The first 7 bits of the header may contain data parameters representing a specific node ID. Slave devices on the CSR bus ring (e.g., each block) can thus examine the instruction header to determine if the master device (host interface 110) requests an address to the block whose header is being examined. If the node ID in the header does not indicate that the destination is a verification block, the verification block copies the input CSR instruction packet to the CSR bus input connected to the next block for verification by the next block.

[0041] The instruction bus 124 originates from the controller 102 and, similar to the CSR bus 122, also provides communication coupling via a bus data path that connects the blocks in the block sets 112, 114 in the ring back to the controller 102. In one embodiment, the controller 102 broadcasts one or more instructions via the instruction bus 124. The instructions broadcast by the controller 102 may differ from those provided via the CSR bus 122. However, the manner in which blocks receive and / or consume or execute instructions received via the bus 124 may be similar to the process used to execute instructions received via the CSR bus 122.

[0042] In one example, the instruction header (i.e., the bitmap) indicates to the receiving block, based on the bitmap associated with the instruction, that the receiving block needs to consume a specific instruction. The bitmap may have a specific width defined according to the bits. Instructions are typically forwarded from one block to the next based on the parameters of the instruction. In one implementation, the width of the instruction bus 124 can be configured to be smaller than the size / width of the instruction. Therefore, in such a configuration, instruction transmission will take several cycles, and the bus station of the instruction bus 124 will have a decoder to place the instruction received at that block into the appropriate target instruction buffer associated with that block.

[0043] As further described below, the blocks in block sets 112, 114 are generally configured to support two broad categories of instructions. These two broad categories may also be referred to as instruction types. Instruction types include TensorOp instructions and Direct Memory Access (DMAOp) instructions. In some implementations, DMAOp instructions have one or more specialized constraints that are allowed to be concurrent. The one or more specialized constraints may be referred to as DMAOp instruction subtypes or opcodes. In some cases, each unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a particular block.

[0044] At specific blocks in blocks 112 and 114, the bus station associated with instruction bus 124 examines the header bitmap to determine the instruction type / subtype. Instructions can be received by the block and subsequently written to the block's instruction buffer before being executed by the block. The instruction buffer of the block to which instructions are written can be determined by the instruction type and subtype indicator / field. The instruction buffer may include a first-in, first-out (FIFO) control scheme, which prioritizes the consumption of one or more related instructions. Therefore, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order they arrive on the instruction bus.

[0045] The different instruction buffers within the diagram are the TensorOp instruction buffer and the DMAOp instruction buffer. As mentioned above, instruction types include TensorOp instructions and DMAOp instructions. Regarding DMAOp instructions, the instruction subtypes (indicating the "write" buffer location) include the following: 1) Mesh inbound instruction buffer; 2) Mesh outbound instruction buffer; 3) Narrow-wide DMA instruction buffer; 4) Wide-narrow DMA instruction buffer; and 5) Ring bus DMA instruction buffer. These buffer locations will be referenced below. Figure 2A more detailed description follows. The terms "wide" and "narrow" are used throughout the specification and generally refer to the approximate size (bits / byte) of the width of one or more memory cells. As used herein, "narrow" can refer to one or more memory cells each having a size or width of less than 16 bits, and "wide" can refer to one or more memory cells each having a size or width of greater than 16 bits, but in some embodiments less than 64 bits.

[0046] Mesh bus 126 provides a data communication path different from CSR bus 122, instruction bus 124, and ring bus 128 (described below). For example... Figure 1 As shown, mesh bus 126 provides a communication path that couples or connects each tile to its corresponding neighboring tiles in the X and Y dimensions. In various embodiments, mesh bus 126 can be used to transport input activation values ​​between one or more narrow memory cells in neighboring tiles. As shown, mesh bus 126 does not allow input activation data to be directly forwarded to non-neighboring tiles.

[0047] In various implementations, the mesh bus 126 and the individual tiles connected via the mesh bus 126 can have the following configurations: The four corner tiles of the mesh have two outgoing ports and two incoming ports. The four edge tiles of the mesh have three incoming ports and three outgoing ports. All non-edge, non-corner tiles have four incoming ports and four outgoing ports. Typically, given an example N×N tile layout, an edge tile is a tile with only three adjacent tiles, while a corner tile is a tile with two adjacent tiles. Regarding the data flow method via the mesh bus 126, generally, each input activation arriving for a particular tile via the mesh bus 126 must be submitted to one or more narrow memory cells of that tile. Furthermore, for tile configurations with fewer than four incoming ports, the DMAOp instruction can write a zero value to a location in the tile's narrow memory instead of waiting for data on a non-existent input port. Similarly, for tile configurations with fewer than four outgoing ports, the DMAOp instruction will not perform narrow memory reads and port writes associated with any non-existent port transfers.

[0048] In some implementations, the location or address of the narrow memory cell to be written to or read from is generated by the tensor traversal unit (hereinafter referred to as "TTU") based on the inbound / outbound DMAOp provided via the mesh bus 126. The inbound and outbound DMAOp can be performed simultaneously, and any required synchronization is managed by a synchronization flag control scheme managed by the controller 102. See below for further details. Figure 2 and Figure 3 TTU will be described in further detail.

[0049] The ring bus 128 originates from the controller 102 and, similar to the CSR bus 122 and instruction bus 124, also provides communication coupling via a bus data path that connects blocks 112, 114 in the ring back to the controller 102. In various embodiments, the ring bus 128 typically connects or couples all wide memory cells in all blocks 112, 114 (see below). Figure 2 (To be described in more detail). Therefore, the payload width of the ring bus 128 corresponds to the width of the wide memory cell within each block arranged in the block sets 112, 114. As described above, the ring bus 128 also includes a bit header that indicates the blocks that need to consume payload data including instructions or parameters communicated via the ring bus 128.

[0050] Regarding data (i.e., payload) received at a specific block via ring bus 128, in response to the received information, each block will zero out (i.e., clear) the location data in the bitmap header that is unique to the receiving block before forwarding the data to another block. Therefore, when the header bitmap has no remaining set of bits indicating the specific block to receive the payload, forwarding of the payload to another block will cease. The payload data typically refers to the activations and weights used by one or more blocks during tensor computations performed based on deeply nested loops.

[0051] In some implementations, controller 102 may be described as part of ring bus 128. In one example, for a DMAOp instruction executed within a specific block, controller 102 may be used to fetch data / payload from a ring bus station and forward the payload to a ring bus station in the next block in the ring. If the instruction in the bitmap header requires this action, controller 102 may also cause payload data to be committed to one or more wide memory cells within the block. The addresses of the one or more wide memory cells to which the data needs to be written may be generated by the DMAOp instruction within the specific block.

[0052] In various implementations, each block in block sets 112, 114 can be either a producer or a consumer of payload data. When a block is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data via ring bus 128 for consumption by one or more other blocks. When a block is a consumer of payload data, it receives data and writes the data to one or more wide memory units within the block and forwards the payload data for consumption by one or more other blocks. Regarding the movement of payload data via ring bus 128, typically only one data producer / master will exist on ring bus 128 at any given time. The execution order of DMAOp instructions in all blocks (e.g., a FIFO control scheme) will ensure that only one data producer / master exists on ring bus 128 at any given time.

[0053] In some implementations, controller 102 uses a synchronization flag control architecture to ensure that only one payload data generator / master exists on the ring bus 128 at a given time. In one example, each write from a block to the ring output triggers an increment of the corresponding synchronization flag count. Controller 102 may examine the payload data to determine the number of data blocks or segments that include the payload. Controller 102 then monitors the execution of the block to ensure that the expected number of data segments are forwarded and / or consumed by the block before another block executes in master mode.

[0054] When local multicast groups connected via ring bus 128 that do not have overlapping areas on the ring bus occur, an exception occurs to ensure that only one data producer / master exists on ring bus 128 at a given time. For example, tile 0 (master) can multicast (i.e., produce data) to tiles in the tile 0-tile 3 group, while tile 4 (master) can do the same to tiles in the tile 4-tile 7 group. A crucial requirement of this dual-master multicast method is that different multicast groups must not be allowed to see each other's data packets, as packet overlap can occur and lead to one or more data calculation errors.

[0055] The tile-to-tile network 129 provides a data communication path distinct from the CSR bus 122, instruction bus 124, mesh bus 126, and ring bus 128. The tile-to-tile network 129 provides a communication path that couples or connects each tile to each other. For example, when the shape of a tensor changes or the tiled dimensions of a tensor change, the tile-to-tile network 129 is used to transfer tensor elements between tiles.

[0056] The logical structure of the tile-to-tile network 129 can be described as having N channels, where N is the number of tiles in system 100. In this example, the tile-to-tile network 129 will have eight channels because there are eight tiles. In some implementations, each channel has a tile as the master tile and each tile has exactly one channel. The master tile writes data to its channel, while each tile including the master tile reads the channel, copies, and buffers the data destined for it. Traffic on channels that are not destined for a tile is not stored by that tile.

[0057] Each tile may include bus stations that forward traffic not destined for that tile and terminate traffic addressed to that tile. Data transmitted from one tile to another on the tile-to-tile network 129 may include a payload and a header specifying the destination tile. The payload may be data of tensor elements passed to the tile based on changes in tensor shape or partitioning scheme.

[0058] Each tile includes a queue for each tile that includes itself. For example, tile 0 has eight queues, each queue for each tile that includes tile 0. The queues for a particular tile store tensor elements received from that particular tile in the order they are received from that particular tile. As described in more detail below, each tile accesses a queue and stores the tensor elements received from the tile in local memory.

[0059] In some implementations, the width of the tile-to-tile network 129 is NumberOfTiles. CanonicalWidth. The width of the tile-to-tile network 129 can be selected to maintain addressing flexibility, and the addressing granularity can match the width of the smallest atomic element supported by system 100. An example tile-to-tile network topology is shown in... Figure 4 and 5 As shown in the diagram and described below, the parameter NumberOfTiles is the number of tiles in system 100, which is 8 in this example. The parameter CanonicalWidth is the width of the smallest atomic element to be transmitted on the tile-to-tile network 129. For example, CanonicalWidth can be eight bits, sixteen bits, or another number of bits corresponding to the smallest atomic element.

[0060] like Figure 1As shown, controller 102 provides communication data paths for coupling or connecting tiles in tile sets 112, 114 to I / O 110, and includes several core functions. The core functions of controller 102 typically include feeding one or more I / O input activations to tiles in tile sets 112, 114; feeding one or more input activations and parameters received from I / O 110 to tiles; feeding one or more instructions received from I / O 110 to tiles; sending I / O output activations to host interface 108; and serving as a ring station for CSR bus 122 and ring bus 128. As described in more detail below, each of the first tile set 112 and the second tile set 114 includes multiple tiles for performing one or more tensor calculations based on deep loop nesting including inner and outer loops.

[0061] System 100 typically operates as follows. Host interface 108 provides controller 102 with one or more instructions defining direct memory access operations (DMAOs) occurring for a given computation. The descriptor associated with the instructions fed to controller 102 includes the information required by the controller to facilitate large-scale dot product computations associated with multidimensional data arrays (tensors). Typically, controller 102 receives input activations, tile instructions, and model parameters (i.e., weights) from host interface 108 for performing tensor computations for a given layer of a neural network. Controller 102 can then multicast the instructions in a data stream manner defined by the instructions (multiple instructions) to tiles 112, 114. As discussed above, the tile consuming the instructions can then initiate the broadcast of new / subsequent instructions to another tile based on the bitmap data in the instruction header.

[0062] Regarding the data flow, input activations and parameters are transferred to the tiles in the tile sets 112 and 114 via the ring bus 128. Each tile 112 and 114 stores a subset of input activations required to compute a subset of the output activations assigned to that particular tile. DMAOp instructions for the tile cause the input activations to move from wide memory to narrow memory. Computation within the tile begins when the required input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) are available in the tile. Computation within the tile ends when the MAC operator (described below) within the tile completes all dot product operations defined by the instruction set and the pre-activation function is applied to the result of the multiplication operation (i.e., the output activation).

[0063] The result of one or more tensor computations includes writing the output activations of the computation layer into narrow memory cells of the tile on which the computation was performed. For some tensor computations, there will be a pass-through of the output edge activations to adjacent tiles via mesh bus 126. When the computation spans multiple layers, the pass-through of output edge activations to adjacent tiles is required to compute the output activations of subsequent layers. When the computation of all layers is complete, DMAOp moves the final activation to classifier tile 116 via ring bus 128. Controller 102 then reads the final activation from classifier tile 116 and executes DMAOp to move the final activation to host interface 108. In some embodiments, classifier portion 116 performs the computation of the output layer (i.e., the last layer) of the NN. In other embodiments, the output layer of the NN is one of the following: a classifier layer, a regression layer, or another layer type typically associated with neural networks.

[0064] Controller 102 can generate and maintain segmentation data, for example, in the form of a table specifying the tensor elements possessed by each tile. As described above, tensor elements of a tensor can be distributed across tiles by segmenting the tensor across one or more tiled dimensions. The segmentation data can specify the range of each tile dimension possessed by the tile for each tile. For example, as described above, a tensor with a shape of [5][4][4][8] for dimensions [w, y, x, z] can be tiled along the [y] and [x] dimensions, such that each tile receives a tensor slice with a shape of [5][1][2][8]. In this example, the segmentation data can specify the range of tensor elements along the x and y dimensions possessed by each tile, as shown in Table 1 below. While Table 1 includes the ranges for dimensions w and z, the segmentation data may not include the ranges for non-tiled dimensions, as they are the same for each tile.

[0065]

[0066] Table 1

[0067] In some implementations, the segmentation data specifies a global linearization index value for each tensor element possessed by that tile. For example, the segmentation data may specify a global linearization index value for each tensor element in a tensor slice assigned to the tile. In addition to the dimensional data shown in Table 1, the segmentation data may also specify global linearization index values. In this way, the segmentation data maps global linearization index values ​​to the corresponding tile space, for example, to tiles that possess (or will receive) tensor elements corresponding to the global linearization index values.

[0068] Controller 102 also instructs tiles to reallocate tensor elements, for example, in response to a change in tensor shape or a change in tiled dimensions. For example, controller 102 may receive instructions from I / O link 110. The instructions may specify a new shape for the tensor and / or a new segmentation scheme for the tensor between tiles. The segmentation scheme may include the tiled dimensions and / or shape of the tensor slices to be received by each tile.

[0069] Controller 102 can generate (or receive from I / O link 110) new segmentation data based on a new shape and / or a new segmentation scheme for the tensors. For example, controller 102 can assign tensor slices to each tile based on the new shape and / or the new segmentation scheme and generate segmentation data (e.g., a segmentation table) that specifies the tensor slices that will be owned by each tile after the tensor elements are reallocated. Segmentation data specifying tiles that owned tensor elements before reallocation is called allocated segmentation data (or allocated segmentation table), and segmentation data specifying tiles that owned tensor elements after reallocation is called reallocated segmentation data (or reallocated segmentation table). Both types of segmentation data can specify, for each tile, a globally linearized index value for the tensor elements owned by the tile before or after reallocation.

[0070] Controller 102 provides each tile with allocation and redistribution of segmentation data. This allows each tile to determine which tile owns each tensor element of the tensor before and after redistribution. Each tile can use a table and a global linearized index value for the tensor elements to determine which tile to send the tensor elements to during redistribution and from which tile to receive the tensor elements during redistribution, as described in more detail below.

[0071] Figure 2 The illustration shows example calculation tile 200. Typically, example tile 200 can correspond to the reference above. Figure 1 The computational block 200 discussed within the first tile set 112 and the second tile set 114 may also be referred to as computational unit 200 or, in various embodiments, as computational block 200. Each computational block 200 is an independent computational unit configured to execute instructions independently relative to other corresponding tiles within tile sets 112, 114. As briefly discussed above, each computational block 200 executes two types of instructions: TensorOp instructions and DMAOp instructions. Typically, each instruction type will include computational operations associated with deep loop nesting, and therefore each instruction type will typically be executed over multiple time cycles to ensure the completion of all loop iterations.

[0072] As discussed in more detail below, different instruction types are executed by independent control units within computational block 200, which synchronize data through synchronization flag control managed within computational block 200. Synchronization flag control manages the concurrency between the execution of different instruction types within computational block 200. Each computational operation associated with each instruction type is executed in a strict release order (i.e., first-in, first-out). Regarding the two instruction types, TensorOP and DMAOp, there is no order guarantee between these different instruction types, and each type is treated as a separate control thread by computational block 200.

[0073] Regarding data flow construction, computational block 200 typically includes data paths 202 and 205, each providing a communication path for data to flow into and out of computational block 200. As described above, system 100 includes four different data bus structures that can be arranged, for example, in a ring configuration: CSR bus 122, instruction bus 124, ring bus 128, and block-to-block network 129. (Refer to...) Figure 2 Data path 205 corresponds to instruction bus 124, while data path 202 may correspond to one of CSR bus 122, ring bus 128, or block-to-block network 129. As shown, data path 202 corresponds to block-to-block network 202. Data path 202 includes block-to-block output 203, which provides an output path for data leaving computational block 200—e.g., tensor elements—and block-to-block input 204, which provides an input path for data entering computational block 200 (e.g., tensor elements).

[0074] The computational tile 200 also includes a TensorOp control 206 containing TensorOp traversal units (TTUs) 226 and a reshaping control 208 containing a plurality of reshaping control TTUs 228. Although not shown, the computational tile 200 may also include a DMAOp control comprising DMAOp TTUs arranged similarly to the reshaping control 228. An example computational tile including DMAop TTUs is described in U.S. Patent Application No. 15 / 335,769, filed October 27, 2016, entitled “Neural Network Compute Tile,” which is incorporated herein by reference.

[0075] TensorOp control 206 typically manages writes to and reads from TensorOp TTU register 232, and controls the traversal operations performed by TensorOp TTU 226. Similarly, DMAOp control typically manages writes to and reads from DMAOp TTU registers, and controls the traversal operations performed by DMAOp TTU. Likewise, remodeling control typically manages writes to and reads from remodeling TTU registers, and controls the traversal operations performed by remodeling control TTU 228.

[0076] TTU register 232 includes an instruction buffer for storing one or more instructions, including operations to be performed by TensorOp TTU 226 when instructions are executed by TensorOp control 206. Similarly, the DMAOp TTU register may include an instruction buffer for storing one or more instructions, including operations to be performed by DMAOp TTU when instructions are executed by DMAOp control. Likewise, the reshaping control TTU register 234 may include an instruction buffer for storing one or more instructions, including operations to be performed by reshaping control TTU when instructions are executed via reshaping control.

[0077] As further described below, the TTU is used by tile 200 to traverse the array elements of one or more tensors that typically reside in narrow memory 210 and wide memory 212. The TTU is also used to compute the global linearization index value of each tensor element in the global tensor and to use the global linearization index value of the tensor elements to determine the multidimensional index of each tensor element (e.g., the index value of each dimension of the global tensor).

[0078] In some implementations, certain instructions for execution by computation block 200 arrive at the block via data path 205 (i.e., a portion of instruction bus 124). Computation block 200 examines the header bitmap to determine the instruction type (TensorOp or DMAOp) and instruction subtype (read operation or write operation). Instructions received by computation block 200 are then written to a specific instruction buffer, depending on the instruction type. Typically, instructions are received and stored (i.e., written to a buffer) before the components of computation block 200 execute the instructions. Figure 2 As shown, the instruction buffers (i.e., TensorOp TTU register 232 and DMAOp TTU register) can both include a first-in-first-out (FIFO) control scheme, which prioritizes the consumption (execution) of one or more related instructions.

[0079] As briefly discussed above, a tensor is a multidimensional geometric object, and example multidimensional geometric objects include matrices and data arrays. Algorithms involving deeply nested loops can be executed by computational block 200 to perform tensor computations by iterating through one or more nested loops over an N-dimensional tensor. In a sample computation, each nested loop can be responsible for traversing a specific dimension of the N-dimensional tensor. As described herein, the TensorOp control 206 typically manages one or more tensor operations that drive the dimensional elements of a specific tensor construction to be traversed and accessed to complete a sequence of computations defined by deeply nested loops.

[0080] Computation block 200 also includes narrow memory 210 and wide memory 212. Narrow and wide specifications typically refer to the width (bits / byte) of the memory cells in narrow memory 210 and wide memory 212, respectively. In some embodiments, narrow memory 210 includes memory cells each having a size or width of less than 16 bits, and wide memory 212 includes memory cells each having a size or width of greater than 16 bits and less than 64 bits. Typically, computation block 200 receives input activations via data path 205, and the DMA control performs the operation of writing the input activations into narrow memory 210. Similarly, computation block 200 receives parameters (weights) via data path 202, and the DMA control performs the operation of writing the parameters into wide memory 212. In some embodiments, narrow memory 210 may include a memory arbitrator, typically used in shared memory systems, to determine, for each memory cycle, which control device (e.g., TensorOp control 206, DMAOp control, or reshaping control) will be allowed to access the shared memory cells of narrow memory 210.

[0081] The computational block 200 also includes an input activation bus 216 and a MAC array 214 comprising multiple single zones, each including a MAC arithmetic unit 215 and a summation register 220. Typically, the MAC array 214 uses the MAC arithmetic unit 215 and the summation register 220 to perform tensor computations across the multiple single zones, which include arithmetic operations related to dot product calculations. The input activation bus 216 provides a data path where input activations are provided sequentially by narrow memory 210 for corresponding access by each MAC arithmetic unit 215 of the MAC array 214. Thus, based on the sequential broadcast of input activations, a single MAC operator 215 of a particular single zone will each receive input activations. The arithmetic operations performed by the MAC arithmetic units of the MAC array 214 typically involve multiplying the input activations provided by narrow memory 210 with parameters accessed from wide memory 212 to produce a single output activation value.

[0082] During arithmetic operations, partial summations can be accumulated and stored in a corresponding summation register, such as 220, or written to wide memory 212 and re-accessed by a specific single area of ​​MAC array 214 to complete subsequent multiplication operations. Tensor computation can be described as having a first part and a second part. The first part is completed when a multiplication operation produces an output activation, for example by performing a multiplication of the input activation with parameters to produce an output activation. The second part involves applying a nonlinear function to the output activation, and is completed after the application of this function, when the output activation is written to narrow memory 210.

[0083] The computational block 200 also includes an output activation bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 indicating core attributes of components in the computational block 200. The reference map 230 is shown for clarity but is not included in the computational block 200. Core attributes include whether a particular component is a cell, storage device, arithmetic unit, control device, or data path. Typically, upon completion of the first part of the tensor computation, an output activation is provided from the MAC array 214 to the NLU 222 via the output activation bus 218. After reaching the NLU 222, data for a specified activation function received via the activation pipeline 224 is applied to the output activation, which is then written to the narrow memory 210. In some embodiments, the output activation bus 218 includes at least one pipeline shift register 236, and completing the second part of the tensor computation includes using the shift register 236 of the activation bus 218 to shift the output activation to the narrow memory 210.

[0084] For example, in calculating the dot product of two multidimensional data arrays, the MAC array 214 provides robust Single Instruction Multiple Data (SIMD) functionality for a single computational tile 200. SIMD generally means that all parallel units (multiple MAC arithmetic units 215) share the same instruction (based on deep loop nesting), but each MAC arithmetic unit 215 executes the instruction for different data elements. In a basic example, adding arrays [1, 2, 3, 4] and [5, 6, 7, 8] element-wise in one cycle to obtain array [6, 8, 10, 12] would typically require four arithmetic units to perform the operation on each element. By using SIMD, the four units can share the same instruction (e.g., "add") and perform computations in parallel. Therefore, system 100 and computational tile 200 provide enhanced acceleration and parallelism in tensor computations that are superior to existing methods.

[0085] In one example, and as described in more detail below, controller 102 can direct input to multiple computational blocks 200 (see...). Figure 1The tile sets 112 and 114 provide a single instruction for consumption by multiple MAC arrays 214. Typically, a neural network layer may include multiple output neurons, and these output neurons can be partitioned such that tensor computations associated with subsets of output neurons can be assigned to specific tiles in the tile sets 112 and 114. Each tile in the tile sets 112 and 114 can then perform relevant tensor computations on different groups of neurons in a given layer. Therefore, computation tile 200 can provide at least two forms of parallelism: 1) one form involves partitioning output activations (corresponding to subsets of output neurons) among multiple tiles in the tile sets 112 and 114; and 2) another form involves simultaneously computationating multiple subsets of output neurons (using a single instruction) based on partitioning between tiles in the tile sets 112 and 114.

[0086] During the reallocation of tensor elements among tiles, the reshaping control 208 sends tensor elements from tiles to tiles on the tile network 129, and from (by...) Figure 2 Data path 202 (represented in the diagram) indicates that tiles on the tile network 129 receive tensor elements. Before reallocation, tile 200 possesses tensor slices of tensors previously assigned to it (e.g., storing them and / or using them to perform computations). Each other tile in the computation system 100 also possesses tensor slices of tensors. The tensor elements of the tensor slices owned by a tile are stored in that tile's local memory (e.g., in narrow memory 210 and wide memory 212).

[0087] The reshaping control 208 can send tensor elements previously assigned to block 200 to blocks (e.g., other blocks and block 200). For example, instead of determining which tensor elements will remain at block 200 and which tensor elements will need to be transferred to different blocks based on a new shape or a new segmentation scheme, each block of the computing system 100 can send each tensor element it possesses to the block that will possess that tensor element (for one or more tensor elements, this could be the block that currently possesses that tensor element) based on the new shape or a new segmentation scheme.

[0088] The reshaping control 208 can use TTUs to traverse tensor elements when sending and receiving tensor elements. For example, when sending tensor elements to a tile, the reshaping control 208 can use outbound TTUs 228B and 228D to traverse the tensor elements currently owned by the tile 200 and send these tensor elements to the tile. When receiving tensor elements from a tile, the reshaping control 208 can use inbound TTUs 228A and 228C to traverse the tensor elements.

[0089] Outgoing TTUs 228B and 228D can traverse the tensor elements owned by tile 200 in the same order based on the global linearization index values ​​of the tensor elements, for example, from the lowest global linearization index value to the highest global linearization index value. Outgoing TTUs 228B and 228D can use segmented data to determine the tensor elements owned by tile 200 and their order.

[0090] Both TTUs 228B and 228D can iterate over the same nested loops of tensor elements, but perform different computations for each iteration of the inner loop of the nested loops. One outgoing TTU 228B can compute the global linearization index value of the tensor element, and the other outgoing TTU 228D can compute the local memory address of the tensor element in the local memory of tile 200. By computing the global linearization index value, tile 200 can use the reallocated segmentation data to determine which tile sends each tensor element.

[0091] Nested loops can be configured based on the size of the tensor slices owned by tile 200 and the tensor segmentation scheme before reallocation. Nested loops can include loops for each dimension. A loop for a dimension iterates through that dimension. The stride of each loop is also based on the size of the tensor slices owned by tile 200 and the tensor segmentation scheme before reallocation.

[0092] As an example, consider a 3D tensor of size [Y, X, Z], partitioned across the x and y dimensions, and with a tiled strategy applied on the z dimension such that the Z dimension is divided into Z and ZZ. The size of the local tensor slice owned by tile 200 is [Y', X', Z]. The nested loop used to determine the global linearized index value for each tensor element could be: For zz in the range (ZZ), stride = Z: For y in the range (Y'), step size = X ZZ Z: For x in the range (X'), the stride = ZZ Z: For z in the range (Z), step size = 1: Global linear index value = zz + y + x + z + base value For the calculation of the global linearized index value, parameter zz= is the cyclic index for zz, parameter y is the cyclic index for y, parameter x is the cyclic index for x, and parameter z is the cyclic index for z. The cyclic index is also equal to the index of the tensor element in that dimension for calculating the global linearized index value. The base value is the base global linearized index value of the tensor elements owned by tile 200. In some implementations, the base value is equal to (start_minor_dimension) local_partition.tiled_minor_elements) + (start_major_dimension The `local_partition.tiled_major_elements` parameter sets the starting index of the minimum tiled dimension. The `local_partition.tiled_minor_elements` parameter is the cardinality of the minimum tiled dimension of the current tensor partition, and can be equal to the product of the cardinality of all dimensions smaller than this dimension. The `start_major_dimension` parameter sets the starting index of the major tiled dimension. The `local_partition.tiled_major_elements` parameter is the cardinality of the major tiled dimension of the current tensor partition, and can be equal to the product of the cardinality of all dimensions smaller than this dimension. These are collectively based on the cyclic tiled formula of the current tiled tensor.

[0093] Similarly, nested loops used to determine the local memory addresses of tensor elements owned by a tile could be: For zz in the range (ZZ), stride = Z: For y in the range (Y'), step size = X ZZ Z: For x in the range (X'), the stride = ZZ Z: For z in the range (Z), step size = 1: Local address = zz + y + x + z + base address As described above, parameter zz= is the cycle index for zz, parameter y is the cycle index for y, parameter x is the cycle index for x, and parameter z is the cycle index for z. The base address is the starting memory address of the tensor slice in the local memory of tile 200.

[0094] As shown above, the nested loops are identical for both the global linearized index value and the local memory address. The TTU 228B and 228D can iterate the loops in parallel with each other and output the global linearized index value and local memory address for each tensor element to the reshaping control 208.

[0095] The reshaping control 208 can use local memory addresses to obtain tensor elements from local memory. The reshaping control 208 can use the global linearization index value for the tensor element and the reallocated segmentation data to determine the destination map tile to which the tensor element should be passed. For example, the reallocated segmentation data can specify, for each computed tile, the global linearization index value of the tensor element that will be owned by the computed tile after reallocation.

[0096] In some implementations, the reshaping control 208 can determine the index of the tensor element in each tiled dimension using the global linearized index value of the tensor element (which is the same before and after redistribution) and the segmentation scheme. If the tensor element is redistributed based on a change in the tensor's segmentation scheme, that segmentation scheme can be the new segmentation scheme. If the tensor is redistributed in response to a change in its shape and the tensor's segmentation scheme remains the same, the segmentation scheme used to determine the index of the tensor element in each tiled dimension can be the same as the segmentation scheme used to assign the tensor element before redistribution.

[0097] Then, the reshaping control 208 can compare the index of the tensor element in each tiled dimension with the redistribution segmentation data to determine which tile the tensor element should be sent to. For example, as described above, the redistribution segmentation data can specify a range of index values ​​for each tile. For each tiled dimension, the reshaping control 208 can compare the index value of the tensor element in that dimension with the redistribution segmentation data. If the index value is within the range specified for the tiles in each tiled dimension, the reshaping control 208 can determine that the tensor element should be passed to that tile.

[0098] For each tensor element, the reshaping control 208 can send data on its channel in the tile-to-tile network 129. This data may include the tensor element obtained from local memory as a payload. The data may also include a header specifying the destination tile for the tensor element determined by the reshaping control 208.

[0099] The reshaping control 208 can send tensor elements on its channels in the tile-to-tile network 129 so that TTUs 228B and 228D traverse the tensor slices owned by tile 200. Each other tile in the computing system 100 can similarly traverse its respective tensor slices and send tensor elements to other tiles on their channels in the tile-to-tile network 129. In some embodiments, the reshaping control 208 can send tensor elements in order based on global linearization index values, for example, from lowest to highest global linearization index values.

[0100] As described above, each tile may include a queue for computing each tile in system 100. Each queue may store tensor elements received from the corresponding slice in the order in which the tensor elements are received. For example, if tile 0 receives three tensor elements from tile 1, then the queue for tile 1 on tile 0 will store the three tensor elements in the order they are received.

[0101] Then, the reshaping control 208 for each tile can rearrange the tensor elements in the queue stored in local memory to create a tensor slice that the tile has after the reallocation. Based on the index values ​​of the tensor elements in each dimension of the tensor, the tensor elements of the tensor slice are stored in local memory. For example, as described below, the TTU can translate tensor indices into memory addresses for storing and retrieving tensor elements.

[0102] The reshaping control 208 can use the outputs of the two inbound TTUs 228A and 228C to sample tensor elements in the queue and store them in the appropriate memory storage location. Since tensor elements are not passed along with their global linearized index values, the order in which tensor elements are sampled is important to ensure that the appropriate tensor elements are stored in the appropriate memory location.

[0103] The reshaping control 208 can use inbound TTUs 228A and 228C to traverse tensor slices of tensors that tile 200 will possess after redistribution. TTUs 228A and 228C can traverse the tensor elements of the tensor slices in the same order. For example, TTUs 228A and 228C can traverse the tensor elements in order based on the global linearization index values, such as from lowest to highest, or in the same order as outbound TTUs 228B and 228D.

[0104] In some implementations, TTUs 228A and 228C traverse tensor elements in order of the dimensions of the tensor slices that tile 200 will have after reallocation. For example, the two TTUs can iterate over the same nested loops of tensor elements, but perform different computations for each iteration of the inner loop of the nested loops. One inbound TTU 228A can compute the global linearized index value of the tensor element, and the other inbound TTU 228C can compute the local memory address where the tensor element will be stored in the local memory of tile 200.

[0105] Nested loops can be configured based on the dimensions of the tensor slices owned by tile 200 and the tensor segmentation scheme before reallocation. Nested loops can include loops for each dimension. The loops for each dimension iterate through that dimension. The stride of each loop is also based on the dimensions of the tensor slices owned by tile 200 and the tensor segmentation scheme before reallocation.

[0106] To ensure that tensor elements are sent and received in the correct order, the cycles used to calculate the inbound TTU and the memory addresses and global linearization index values ​​can both be ordered in ascending order based on the global linearization index values. In this way, the queue for each tile will include tensor elements received from the tile corresponding to the queue in order of lowest to highest global linearization index values.

[0107] The reshaping control 208 can use the global linearization index value for each tensor element to determine which tile sends the tensor element to tile 200. For example, the reshaping control 208 can use reallocated segmentation data to determine the global linearization index value of the tensor element reallocated to computation tile 200. The reshaping control 208 can also use segmentation data to determine the computation tile that previously owned the tensor element. This informs the reshaping control which queue to sample for each tensor element.

[0108] In some implementations, the reshaping control 208 can use a global linearization index value and a partitioning scheme used to assign tensor elements to tiles before reallocation to determine the multidimensional addresses of tensor elements in the original tensor (the tensor before reshaping or modification in the partitioning scheme). The reshaping control 208 can then compare the index value of the tensor element for each tiled dimension with the allocated partitioning data to determine which tile possessed the tensor element before reallocation. The reshaping control 208 can then retrieve the tensor element from the queue of that tile and store it in local memory at the memory address determined by the TTU 228C.

[0109] The reshaping control 208 can perform the same technique on each tensor element traversed by the TTUs 228A and 228C. For example, in each iteration of the nested inner loop, the TTUs 228A and 228C can output a global linearization index value and a local memory address to the reshaping control 208, respectively. The reshaping control 208 can receive this data for each iteration of the inner loop, obtain the tensor element from the queue corresponding to the tile from which it receives the tensor element, and store the tensor element in local memory using the local memory address output by the TTU 228C.

[0110] Each other tile can traverse its tensor slices in a similar manner. Once each tile has stored its tensor elements in local memory, the reallocation is complete. If, for example, the tensor elements are reallocated again based on another reshaping or change in the segmentation scheme, a similar technique can be used to reallocate the tensor elements. In the next reallocation, the previously reallocated data used to reallocate the tensor elements will be the allocation data, and the new reallocation data will be generated based on the new shape or new segmentation scheme. The DMA control 208 of each tile can reconfigure its TTUs 228A-228D based on the dimensions of the tensor slices passed to / from the tile, such that the tensor slices are traversed in the appropriate order and with the appropriate stride. For example, the controller 102 can generate (or receive) data specifying the configuration of each TTU for each tile and provide that data to the reshaping control of each tile.

[0111] Figure 3 An example Tensor Traversal Unit (TTU) structure 300 is shown, which includes four tensors to track each at a depth of eight. TTU 300 typically includes a counter tensor 302, a stride tensor 304, an initial tensor 306, and a constraint tensor 308. TTU 300 also includes an adder library 310 and a tensor address index 312. As mentioned above, tensors are multidimensional geometric objects, and in order to access the elements of a tensor, an index for each dimension must be provided. Because tensors are stored in narrow memory 210 and wide memory 212, the set of tensor indices must be converted to a set of memory addresses. In some implementations, the index-to-memory address conversion is accomplished by making the memory addresses a linear combination of indices and reflecting the addresses via tensor address index 312.

[0112] In computation block 200, each control thread has a TTU and each instruction type (TensorOP and DMAOP) has a control thread. As described above, there are three sets of TTUs in computation block 200: 1) TensorOp TTU 226; 2) DMAOp TTU; and a reshaping control TTU 228. In various implementations, TensorOp control 206 will cause TTU 300 to load TensorOp TTU counter 302, limit 308, and step value 304 at the beginning of a specific tensor operation, and will not change the register values ​​until the instruction is retired. Each of the two TTUs will need to generate addresses in computation block 200 for the following memory address ports: 1) wide memory address port 212, and 2) narrow memory address port 210, which has four independent arbitration libraries presented as four address ports.

[0113] As described above, in some embodiments, narrow memory 210 may include a memory arbitrator typically used in shared memory systems to determine, for each memory cycle, which control device (e.g., TensorOp control 206 or DMAOp control 208) will be allowed to access the shared memory resources of narrow memory 210. In one example, the different instruction types (TensorOp and DMAOp) are independent control threads requesting memory access that requires arbitration. When a particular control thread submits a tensor element to memory, the control thread increments a counter 302 of the tensor reference submitted to memory.

[0114] In one example, when TensorOp control 206 executes instructions for accessing a specific element of a tensor, TTU 300 can determine the address of that specific element, allowing control 206 to access storage, such as narrow memory 210, to read data representing the activation value of that specific element. In some implementations, the program may include nested loops, and control 206 may execute instructions to access elements of a two-dimensional array variable within the nested loop based on the current index variable value associated with the nested loop.

[0115] TTU 300 can simultaneously maintain the traversal state of up to X TTU rows of a given tensor. Each tensor residing in TTU 300 occupies a dedicated hardware tensor control descriptor. The hardware control descriptor can consist of X TTU counters 302 per row position, a stride 304, and a limit register 308, supporting tensors with up to X TTU counters per row dimension. In some implementations, the number of rows and the number of counters per row can be different.

[0116] For a given location register, the final memory address is calculated from an addition operation that adds the location registers together. The base address is incorporated into counter 302. One or more adders are shared for tensor references residing in the same memory. In one implementation, because only a single load / store can exist on any given port within a cycle, it will be a function of loop nesting control to ensure that multiple tensor references residing in the same narrow or wide memory do not cause their counters to increment within any given cycle. The use of registers for calculating memory access address values, including determining offset values, is described in more detail in patent application No. 15 / 014,265 entitled “Matrix Processing Apparatus,” filed February 3, 2016, the entire disclosure of which is expressly incorporated herein by reference in its entirety.

[0117] The following template parameters can be used to instantiate a dedicated TTU 300: 1) X TTU rows; 2) X TTU counters per row; 3) X TTU adder units; 4) a shared adder reference for each TTU row; and 5) X counter size [TTU][Row][depth] for each counter. All TTU registers are architecturally visible. The address of a specific tensor element that needs to be accessed for computation (i.e., tensor address 312) is the result of the summation of the counters. When an increment signal is issued from the control thread to a row of TTUs, the TTU 300 performs a single-cycle operation, incrementing the innermost dimension by a step size 304 and propagating the flip through all depths.

[0118] Typically, the TTU 300 defines a state associated with one or more tensors. The state may include loop boundary values, current loop index variable values, dimension multipliers used to compute memory address values, and / or program counter values ​​used to process branch loop boundaries. The TTU 300 may include one or more tensor state elements and arithmetic logic units. Each of the tensor state elements may be a storage element, such as a register or any other suitable storage circuit. In some embodiments, the tensor state elements may be physically or logically arranged into different groups, as described in more detail in patent application serial number 15 / 014,265, which is incorporated herein by reference in its entirety.

[0119] Figure 4 An example logic structure 400 for a tile-to-tile network 401 used to redistribute tensor slices between computation tiles 410A-410H is shown. Figure 4 As shown, for each tile, the tile-to-tile network 401 includes data communication paths from that tile to each tile that includes it. For example, tile 410A has a data communication path 402 from itself to itself and a data communication path 403 to each of the other tiles 410B-410H.

[0120] Figure 5 An example layout 500 of a tile-to-tile network 502 for reallocating tensor slices among computation tiles 510A-510H is shown. The tile-to-tile network 500 includes a ring-shaped bus data path 502 between tiles 510A-510H. The tile-to-tile network 500 includes a bus path 502 for each tile 510A-510H. Each tile is the master tile for its data bus path 502. As described above, each tile 510A-510H may include a bus station that forwards traffic not destined for that tile and terminates traffic addressed to that tile.

[0121] Figure 6This is a flowchart of an example procedure 600 for transferring tensor slices between computational tiles. Example procedure 600 can be executed by a computing system with multiple computational units. For example, the example procedure can be performed by... Figure 1 The computing system 100 executes.

[0122] The system distributes tensor elements (610) of an N-dimensional tensor among computational units—e.g., computational tiles. For example, the system may include a set of one or more computational units arranged in a two-dimensional configuration. The system can segment the N-dimensional tensor into tensor slices across one or more dimensions of the tensor—e.g., two or more dimensions. Each computational unit may receive one or more tensor slices, and each tensor slice includes one or more tensor elements.

[0123] The system receives an instruction (620) to reallocate tensor elements. For example, the system may receive an instruction to reallocate tensor slices from another computing system based on a machine learning model executed by the computing system. In a particular example, the machine learning model may need to reshape the tensor at a point in the model before performing additional computations. In another example, the dimensions in which the tensor is segmented may change, for example, based on a new segmentation scheme for the tensor.

[0124] Each computational unit reallocates its tensor elements (630). As described above, each computational unit can send its tensor elements to other computational units in the computational system. That is, each computational unit can send tensor elements to itself and other computational units based on a new shape or a new partitioning scheme of the tensor. Each computational unit can use constructive operations 632-638 to reallocate its tensor elements.

[0125] The computation unit accesses the redistributed segmentation data (632). As described above, the redistributed segmentation data can specify tensor elements for each computation unit, which will be stored by the computation unit after the tensor elements are redistributed. That is, the redistributed segmentation data can specify tensor slices for each computation unit, which will be stored by the computation unit as a result of the tensor's new tensor shape or a new segmentation scheme. As described above with reference to Table 1, the redistributed segmentation data can specify the range of each tiled dimension possessed by the tile for each computation unit, without specifying the values ​​of the non-tiled dimensions.

[0126] The computation unit determines a global linearization index value (634) for each tensor element. The global linearization index value of a tensor element is based on the multidimensional address of the tensor element in the tensor (e.g., the tensor before receiving an instruction to reallocate tensor elements). That is, the global linearization index value of a tensor element can be based on the tensor index of the tensor element in the original tensor.

[0127] In some implementations, the global linearization index value for each tensor element is along a linear range, such as from zero to N-1, where N is the number of tensor elements in the tensor. The equation used to determine such a global linearization index value can be based on the dimension of the N-dimensional tensor and the number of tensor elements along each dimension. For example, the equation for a three-dimensional [y, x, z] tensor can be shown as Equation 1 below.

[0128] Global linearized index value = (1)

[0129] In Equation 1, parameter ix is ​​the index of the tensor element in the x-dimensional direction, iy is the index of the tensor element in the y-dimensional direction, iz is the index of the tensor element in the z-dimensional direction, and x... length It is the number of tensor elements along the x-axis, and z length This is the number of tensor elements along the z-axis. For other dimensions, the global linearization index can be calculated in a similar way using the product of the index of the penultimate dimension and the stride of the last dimension. For example, if the fourth dimension is the depth "d", the global linearization index value can be calculated using Equation 2 below.

[0130] Global linearized index value = (2)

[0131] The computation unit determines the destination computation unit (634) for the tensor element. The computation unit can determine that the destination computation unit sends the tensor element using the global linearized index value of the tensor element and reallocated segmentation data (as mentioned, for one or more tensor elements, the destination computation unit can be the computation unit currently possessing the tensor element). As described above, the computation unit can use the global linearized index value to determine the tensor index of the tensor element along each tiled dimension of the reshaped or re-tiled tensor. The computation unit can then compare the index value with a range of index values ​​assigned to each computation unit for each tiled dimension to determine the destination computation unit for the tensor element. For example, if a tensor element has an index of 3 for the x-dimensional, and a particular computation unit is being reallocated a tensor slice that includes tensor elements with indices between 3 and 4 on the x-dimensional, the computation unit can determine that the particular computation unit is the destination computation unit for the tensor element.

[0132] As described above, a computation unit (TTU) can iterate through tensor elements previously assigned to it and use nested loops to determine the global linearization index value of the tensor elements. For example, a TTU can use nested loops to iterate through tensor elements and compute the global linearization index value of the tensor elements. The same nested loops can be used by another TTU to determine the local memory address where the tensor elements are stored. In this way, the computation unit can use the output of the TTU to determine which tensor element to access and where to send each tensor element in a specific order.

[0133] The computation unit sends each tensor element to its corresponding destination computation unit (638). As described above, each computation unit can be a host of a tile-to-tile network channel. Each computation unit can send its tensor elements to the computation unit via its tile-to-tile network channel. The computation unit can include a header specifying the destination computation unit for each tensor element.

[0134] Each computation unit receives and stores its tensor elements (640). As described above, each computation unit may include a queue for each computation unit. The queue for a particular computation unit stores the tensor elements received from that particular computation unit. The computation unit can access the tensor elements from the queue and store the tensor elements in the appropriate local memory location using construct operations 642-650.

[0135] The computation unit accesses the allocated segmentation data (642). The allocated segmentation data can specify tensor elements that are pre-stored by the computation unit before the tensor elements are reallocated. That is, the allocated segmentation data can specify tensor slices that are stored by the computation unit before instructions for a new tensor shape or a new segmentation scheme for the tensor are received. As described above with reference to Table 1, the allocated segmentation data can specify the range of each tiled dimension owned by the tile for each computation unit, without specifying the values ​​of non-tiled dimensions.

[0136] The computation unit determines a global linearization index value (644) for each tensor element. This global linearization index value for the tensor element is the same as the global linearization index value determined in operation 634 above. However, this time the value is determined by the computation unit receiving the tensor element, rather than the computation unit sending the tensor element.

[0137] As described above, each computation unit can use nested loops to iterate over the tensor elements reassigned to it. Nested loops can sequentially traverse the tensor elements of a tensor slice along each dimension, allowing computation units to access tensor elements from each queue in the order they are received by the queues.

[0138] For each tensor element, the computation unit can use the determined global linearization index value to determine the source computation unit (646) for that tensor element. The source computation unit is the computation unit that sent the tensor element as part of the redistribution to the computation unit. For example, the computation unit can, for instance, use the inversion of Equation 1 for a three-dimensional tensor to convert the global linearization index value of the tensor element into its multidimensional address. This is, for example, the multidimensional address of the tensor element in the original tensor before reshaping. The computation unit can then compare the index of the tensor element in each tiled dimension with the allocated segmentation data to determine which computation unit previously owned the tensor element before the redistribution.

[0139] The computation unit also determines the local memory address of the tensor element (648). The local memory address is the address in local memory where the tensor element will be stored. As described above, the computation unit can use the same nested loops used to determine the global linearized index value of the tensor element to traverse the tensor element to which it has been reassigned. The computation unit stores the tensor element in local memory using the determined local memory address (650).

[0140] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., modules of one or more computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof.

[0141] The processes and logic described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic can also be performed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), or GPGPU (General-Purpose Graphics Processing Unit).

[0142] As an example, a computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, or operatively coupled to receive data from or transfer data to, or both, such as disks, magneto-optical disks, or optical disks. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0143] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated into dedicated logic circuitry.

[0144] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or claimable content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while the features described above may be described as functioning in certain combinations or even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0145] Similarly, although operations are shown in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems can typically be integrated together in a single software product or packaged into multiple software products.

[0146] Specific embodiments of this subject matter have been described. Other embodiments fall within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes shown in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A system comprising: Piece-to-tile network, wherein the tile-to-tile network is used to convey tensor elements of tensors; Multiple computing units, which are communicatively coupled to each other via the tile-to-tile network, each computing unit comprising: For each of the plurality of computing units, one or more queues of each other computing unit, wherein each of the one or more queues of each other computing unit is configured to store tensor elements sent to the other computing unit or to store tensor elements received from the other computing unit; A reshaping control, configured to send and receive tensor elements through the tile-to-tile network using one or more queues for each computational unit; and One or more tensor traversal units are configured to compute a globally linearized index value for each received tensor element and each transmitted tensor element based on the multidimensional index of the tensor elements in the tensor, wherein the multidimensional index of each tensor element includes an index value for each dimension of the tensor corresponding to the position of the tensor element along that dimension; and A controller configured to manage the redistribution of the tensor elements of the tensor among the plurality of computing units.

2. The system according to claim 1, wherein, Each computation unit is configured to perform machine learning computations using tensor elements sent to the computation unit.

3. The system according to claim 1, wherein, Each computation unit includes a memory for storing tensor elements sent to the computation unit.

4. The system according to claim 1, wherein, The global linearization index value of each tensor element uniquely identifies the tensor element.

5. The system according to claim 1, wherein, The one or more queues of each other computing unit include a receive queue for storing tensor elements received from the other computing unit and a send queue for storing tensor elements sent to the other computing unit.

6. The system according to claim 5, wherein, The one or more tensor traversal units include: An inbound tensor traversal unit is configured to traverse tensor elements stored in the receive queue and compute the global linearized index value for each tensor element stored in the receive queue; and An outbound tensor traversal unit is configured to traverse tensor elements stored in the sending queue and calculate the global linearization index value for each tensor element stored in the sending queue.

7. The system according to claim 6, wherein, The reshaping control for each computation unit is configured to send tensor elements in order based on the global linearized index value of each tensor element sent from the computation unit.

8. The system according to claim 7, wherein, Each inbound tensor traversal unit is configured to compute the global linearized index value based on segmented data, which indicates the multidimensional index of each tensor element in the tensor and the computation unit from which each tensor element is received.

9. The system according to claim 8, wherein, Each computation unit includes an additional inbound tensor traversal unit configured to determine a local memory address for storing each tensor element received by the receive queue based on the multidimensional index of the tensor element.

10. The system according to claim 8, wherein, Each computation unit includes an additional outbound tensor traversal unit configured to determine, based on the multidimensional index of the tensor elements, the local memory address at the computation unit where each tensor element sent by the computation unit is stored.

11. The system according to claim 1, wherein, Each tensor traversal unit is configured to traverse tensor elements using nested loops, the nested loops including a loop through each dimension of the tensor.

12. The system according to claim 1, wherein, The tile-to-tile network includes a channel for each computation unit, and the computation unit is configured to send data, including tensor elements, to another computation unit among the plurality of computation units on the channel.

13. The system according to claim 12, wherein, The data, including the tensor elements, also includes a header identifying the computational unit to which the data was sent.

14. The system according to claim 13, wherein, The data including the tensor elements does not include the global linearization index value of the tensor elements.

15. The system according to claim 1, wherein, The controller is configured to: Receive instructions to reallocate tensor elements among the computing units; The segmentation data is determined, wherein for each tensor element of the tensor, the segmentation data indicates (i) the computational unit that will possess the tensor element after reallocation, and (ii) the multidimensional index of the tensor element in the original version of the tensor received by the system, wherein the multidimensional index of each tensor element includes an index value for each dimension of the tensor corresponding to the position of the tensor element along that dimension of the tensor; and The segmented data is sent to each computing unit.

16. The system according to claim 15, wherein, The segmented data indicates the global linearization index value of the tensor element for each tensor element of the tensor.

17. The system according to claim 1, wherein, The controller communicates with the plurality of computing units via a bus different from the tile-to-tile network.

18. The system according to claim 1, wherein, Each computing unit includes a bus station configured to forward traffic to the computing unit that is not destined for the path along the tile-to-tile network.

19. A system comprising: Multiple computing units, each computing unit including: For each of the plurality of computing units, one or more queues of each other computing unit, wherein each of the one or more queues of each other computing unit is configured to store tensor elements sent to the other computing unit or to store tensor elements received from the other computing unit; A reshaping control, configured to send and receive tensor elements via a tile-to-tile network using one or more queues for each computational unit; and One or more tensor traversal units, configured to compute a global linearized index value for each received tensor element and each transmitted tensor element based on the index values ​​of the tensor elements; and The tile-to-tile network connects each computational unit to other computational units. The network includes channels for each computational unit, along which the computational unit sends data, including tensor elements, to the other computational units. A controller configured to manage the redistribution of tensor elements of the tensor along the tile-to-tile network among the plurality of computational units.

20. The system according to claim 19, wherein, Each computation unit is configured to perform machine learning computations using tensor elements sent to the computation unit.

Images