Tensor Mapping Unit for Hardware Integrated Circuits

Abstract

A method and system, including a computer-readable medium, are described for a tensor mapping unit of a hardware integrated circuit configured to accelerate tensor computations of a machine learning model, wherein the tensor mapping unit is configured to: i) generate tensor element positions of a tensor based on instructions received from a processing engine of the integrated circuit; ii) determine that a first subset of the tensor element positions is assigned to a first computational tile of the integrated circuit; and iii) generate control signals to retrieve a group of inputs from a tile memory of the first computational tile based on the first subset.

Description

[Background technology]

[0001] This specification relates generally to mapping computational operations onto hardware integrated circuit devices.

[0002] A neural network is a machine learning model that utilizes one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. Some neural networks may be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Different types of neural network architectures can be used to perform a variety of tasks related to classification or pattern recognition, prediction involving data modeling, and information clustering.

[0003] A neural network layer can have a corresponding set of parameters or weights. The weights are used to process inputs (e.g., batches of inputs) through the neural network layer and generate corresponding outputs of the layer for computing neural network inferences. The batches of inputs and sets of kernels can be represented as tensors, i.e., multidimensional arrays, of inputs and weights. A hardware accelerator is a dedicated integrated circuit for executing neural networks. The circuit includes a memory with locations corresponding to elements of the tensors that can be traversed or accessed using the circuit's control logic. Summary of the Invention

[0004] This document describes techniques for implementing a tensor mapping unit on a dedicated hardware integrated circuit, such as a hardware accelerator, tensor processing unit, or neural network processor, that runs neural networks and other machine learning algorithms. The tensor mapping unit can be implemented in hardware, software, or both. The hardware architecture of the tensor mapping unit includes a first tensor module and multiple second tensor modules. The first tensor module is external to multiple computational tiles of the dedicated integrated circuit, while each second tensor module is included in each of multiple computational tiles of the integrated circuit.

[0005] The tensor mapping unit uses the first tensor module and one or more of the second tensor modules to speed up mapping from a logical space of tensor indices to a space after a transform is applied to the input samples (e.g., images) before the input samples are processed by an integrated circuit using the tensor mapping unit. The input samples may be images or tensors, and the transform is an affine transform. Generally, the input samples are some kind of machine learning input (e.g., audio samples), and the transform is an operation performed on the input samples as a preprocessing step to compute a model output from the input samples.

[0006]

[0006] One aspect of the subject matter described herein may be embodied in an integrated circuit configured to accelerate tensor computations of a machine learning model executed on the integrated circuit. The integrated circuit includes a processing engine and a tensor mapping unit. The tensor mapping unit is configured to: i) generate a plurality of tensor element positions of a tensor based on instructions received from the processing engine; ii) determine that a first subset of the plurality of tensor element positions is assigned to a first computational tile of the integrated circuit; and iii) generate control signals to retrieve a group of inputs from a tile memory of the first computational tile based on the first subset.

[0007] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the tensor mapping unit is configured to generate the plurality of tensor element positions by performing an affine transformation on the input image based on instructions received from the processing engine. In some implementations, the integrated circuit includes a plurality of computational tiles, and the tensor mapping unit is further configured to determine an assignment of different subsets of the plurality of tensor element positions to one or more of the plurality of computational tiles.

[0008] The tensor mapping unit includes a first tensor module that i) generates a plurality of tensor element positions and ii) generates the plurality of tensor element positions for each of the plurality of tensors based on one or more instructions from the processing engine. In some implementations, the tensor mapping unit includes a plurality of second tensor modules that generate control signals to obtain data corresponding to the plurality of tensor element positions. The plurality of tensors are processed in the integrated circuit using each second tensor module and each of the plurality of computational tiles.

[0009] The tensor mapping unit is configured to generate a tensor mapping table, the tensor mapping table including mapping information specific to: i) a plurality of tensor element locations; ii) a plurality of subsets of the plurality of tensor element locations; and iii) an assignment of the first subset to a first computational tile. In some implementations, the tensor mapping unit is configured to determine a mapping from a first set of addresses in a virtual address space to a second, different set of addresses in a physical address space. The first tensor module is coupled to the processing engine and is external to each of the plurality of computational tiles.

[0010] In some implementations, a second tensor module of the plurality of second tensor modules is located in each computational tile of a plurality of computational tiles included in the integrated circuit, and each second tensor module is configured to access the tile memory of the computational tile that includes that second tensor module, and the second tensor module accesses the tile memory based on a particular subset of the plurality of tensor element positions assigned to that computational tile.

[0011] Each input in the group of inputs can correspond to a respective tensor element location of the first subset, where each element of the tensor corresponds to an input and an address location in the tile memory where the input is stored. The tensor element locations can correspond to different dimensions of two or more multidimensional tensors, where at least one of the multidimensional tensors is an input tensor or a parameter tensor that is processed in the integrated circuit using the processor engine, the tensor mapping unit, and one or more computational tiles.

[0012] Another aspect of the subject matter described herein may be embodied in a computer-implemented method for accelerating tensor computations for a machine learning (ML) model executed on an integrated circuit including a processing engine and a tensor mapping unit. The method includes generating, by the processing engine, instructions used to compute outputs of the ML model, the instructions executable on the integrated circuit to cause performance of operations including: generating, by the tensor mapping unit, a plurality of tensor element positions of a tensor based on the instructions; determining, by the tensor mapping unit, that a first subset of the plurality of tensor element positions is assigned to a first computational tile of the integrated circuit; and generating, by the tensor mapping unit, control signals to retrieve a group of inputs from a tile memory of the first computational tile based on the first subset.

[0013] These and other implementations may each optionally include one or more of the following features. For example, in some implementations, the tensor mapping unit includes a plurality of second tensor modules, and the method further includes generating, by the plurality of second tensor modules, control signals to obtain data corresponding to the plurality of tensor element positions, and processing the plurality of tensors in the integrated circuit using each second tensor module and each of the plurality of computational tiles.

[0014] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs encoded on computer storage devices and configured to perform the actions of the methods. One or more computer systems may be so configured by software, firmware, hardware, or a combination thereof installed on the systems that, when operated, causes the systems to perform the actions. One or more computer programs may be so configured by having instructions that, when executed by a data processing device, cause the systems to perform the actions.

[0015] The subject matter described herein can be implemented in particular embodiments to achieve one or more of the following advantages: The disclosed technology efficiently exposes the cores of a processor engine to a large scratchpad memory located in and distributed across multiple computational tiles of an integrated circuit. In particular, the disclosed tensor mapping unit exposes the tile memory of each computational tile as a scratchpad accessible to the processor engine. Thus, the tile memory is not limited to only specific arithmetic operations that are specifically structured and reduced for execution in the tile.

[0016] Rather, this disclosure extends and improves the utility of tiled memory to support multiple processor engine operations that may be performed on an integrated circuit. Relatedly, the tensor mapping unit enables cross-complementary support, such that resources of a computational tile can complement processor engine operations and vice versa. Processor engine cores can be tightly coupled to the tiled memory, such that complementary operations occur without expensive data transfer overhead.

[0017] In an exemplary system-on-chip (SoC), transformations (e.g., affine transformations) may be performed to preprocess images for machine learning workloads. For example, preprocessing may include resizing or cropping an image before the image undergoes inference calculations to recognize one or more objects in the image. Conventional systems that include dedicated hardware accelerators and general-purpose central processing units (CPUs) offload such preprocessing to the CPU. This offloading increases the overall computation cycles of the machine learning task and unnecessarily consumes CPU and memory resources (e.g., DRAM).

[0018] The disclosed tensor mapping unit and associated programming logic enable improved distribution of tensor data across multiple computational tiles. More specifically, this improved distribution is made possible by efficient and dynamic mapping of the virtual address space of the new processing core(s) to the physical address space of the computational tiles. Thus, rather than offloading preprocessing to the CPU and DRAM, the tensor mapping unit and processing core(s) can utilize this address mapping to efficiently preprocess images locally on a set of computational tiles.

[0019] The processor cores and tensor mapping units can cooperate to instantly remap resized (or preprocessed) image data for inference computation. For example, after preprocessing, the SoC can utilize the processor cores and tensor mapping units to automatically reallocate the resized image locally in computation tiles and then perform intensive neural network computations to generate image recognition outputs. Thus, by avoiding the expensive data transfer and resource penalties of offloading and reloading input data, the SoC can more efficiently accomplish its data transformation and inference computations.

[0020] The disclosed technology enables uniform distribution of tensor data across multiple compute tiles, allowing efficient control and status register (CSR) programming for accessing tensors evenly distributed across two or more tiles. The programming logic includes an instruction set including byte-address mode instructions that enable the processor engine to perform byte-addressable accesses to tile memory. This byte-addressability allows localized access to byte-level data of various lengths, providing the processor engine with fine-grained tensor access capabilities.

[0021] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram of an exemplary computing system that maps portions of tensors to computational tiles of an integrated circuit. [Figure 2] FIG. 1 is a block diagram of an exemplary computing system with one or more computational tiles for executing machine learning models. [Figure 3] 1 illustrates an example architecture of a portion of a tensor mapping unit within a compute tile. [Figure 4] FIG. 1 is a block diagram illustrating aspects of a memory mapping process. [Figure 5] 1 shows examples of virtual tensor layouts and physical tensor layouts. [Figure 6] 1 illustrates an exemplary tensor mapping table. [Figure 7] 1 shows example memory regions and the distribution of tensor data within those memory regions. [Figure 8] 1 is an exemplary process of a tensor mapping operation performed using an exemplary tensor mapping unit. [Figure 9] Examples of input tensors, parameter tensors, and output tensors are shown below. DETAILED DESCRIPTION OF THE INVENTION

[0023] Like reference numbers and designations in the various drawings refer to like elements.

[0024] System 100 includes a processor engine 102, a tensor mapping unit 104, and one or more computational tiles 106. In some implementations, processor engine 102 is a general-purpose central processing unit (CPU) (e.g., a multi-core CPU). Processor engine 102 can be configured as an instruction and vector data processing engine that processes data retrieved from system memory, such as dynamic random access memory (DRAM), of a system-on-chip (SoC). In some implementations, system 100 is an SoC of a consumer electronics / mobile device, such as a smartphone, tablet, or laptop.

[0025] Processor engine 102 may include multiple (e.g., more than two) processor cores. In the example of Figure 1, processor engine 102 includes a first processor engine / core 102-0 and a second processor engine / core 102-1. In some implementations, first processor engine 102-0 and second processor engine 102-1 are separate processors, and in some other implementations, first processor engine 102-0 and second processor engine 102-1 are separate cores of a single processor.

[0026] The tensor mapping unit 104 handles address remapping between logical / virtual or physical address spaces of the system 100, as described below with reference to the example of Figure 4. The tensor mapping unit 104 can be implemented in hardware, software, or both. The hardware architecture of the tensor mapping unit 104 can include a first portion that is external to the computational tiles 106 and a second portion that is local to each of the computational tiles 106.

[0027] The first portion may alternatively be referred to as a first tensor module, and the second portion may alternatively be referred to as a plurality of second tensor modules 104-0, 104-1, 104-2, ..., 104-n. The first tensor module is coupled to the processor engine 102 and exchanges data communications, such as instructions and requests, with the processor engine 102. In some implementations, the tensor mapping unit 104 includes a plurality of tensor modules 104-n, where n is an integer equal to or greater than 0.

[0028] Each of the one or more computational tiles 106 includes a respective tile memory 108 that stores data such as operands for arithmetic and other computational operations performed to execute the machine learning workload. The computational tiles 106 are described in further detail below with reference to the example of Figure 2. The system 100 includes a processor engine 102, a tensor mapping unit 104, and a bus 109 and interface block 110 for routing data communications between one or more of the computational tiles 106.

[0029] The bus 109 can be configured as an instruction and vector data bus that, in cooperation with an interface block 110 (described below), routes instructions and data to each of the multiple computation tiles 106. In some implementations, data traffic is routed and / or managed on the bus 109 based on an on-chip communication bus protocol, such as the Advanced Extensible Interface (AXI) protocol or other related bus protocol. The data traffic routed over the bus 109 includes requests involving tensor data stored in one or more tile memories 108. For example, the requests can be read / load (LD) requests to retrieve tensor data from the tile memories 108 and write / store (ST) requests to provide tensor data to the tile memories 108.

[0030] In some implementations, the computational tiles 106 are arranged in a ring configuration, and the bus 109 is configured as a ring bus, such that a request / response path 112 traverses each of the computational tiles 106. In this implementation, the system 100 can send requests to some (or all) of the tensor modules 104-n in each computational tile 106 via the ring bus 109, and receive responses to the corresponding requests via the ring bus 109. In some implementations, the request / response path 112 is non-blocking, meaning that a request can continue to traverse the ring bus 109 from a first computational tile 106 to a next / second computational tile 106 without waiting for a response from the previous tile 106.

[0031] In general, the processor engine 102 uses or generates LD / ST instructions to access the tile memory 108 in one or more computational tiles 106. A request may be or issue from a LD / ST instruction. The instruction may target an address range of the tile memory 108 of a particular computational tile 106, where the address range corresponds to a portion, slice, or dimension of a tensor. For example, each computational tile 106 may be assigned one slice of a tensor, where a "slice" may be represented as a range of indices in each dimension of a multidimensional tensor. The rank of the slice may be the same as the rank of the multidimensional tensor.

[0032] In some implementations, when a request corresponding to a LD / ST instruction is issued, the request is broken down or segmented into one or more 32-byte (32B) requests. Thus, in some cases, a single request may result in multiple responses or response packets. In some examples, a single LD / ST instruction may be generated based on an instruction set architecture that supports various byte granularities. For example, a request associated with a LD / ST instruction may be a 1-byte request, an 8-byte request (scalar register), or a request of up to 512 bytes (vector LD / ST). As described above, for requests larger than 32 bytes, the processor engine 102 may be configured to break them down into multiple requests, each transaction of 32 bytes or less.

[0033] Bus 109 can route requests and responses to requests through an exemplary tiled memory fabric. In some implementations, the tiled memory fabric is a fabric interface with one or more bus lines (e.g., data path 112) used to route data for requests and for responses to requests. Request and response traffic traversing the tiled memory fabric can be controlled, ordered, and / or managed via control logic in interface block 110. For example, interface block 110 can include a reorder buffer to ensure that responses sent back to processor engine 102 are in the same order as the requests corresponding to those responses.

[0034] In some implementations, interface block 110 is configured to detect when a response to a particular request has arrived, read (or scan) the mask field of the response, and perform any necessary shifting or positioning operations. For example, interface block 110 can scan the response, detect the mask field, shift the response data to the appropriate position, and merge the response data with corresponding data from one or more previously received responses. Interface block 110 can support scanning or detecting the mask field and shifting the response data to perform any necessary parsing operations. Mask fields are described below with reference to the example of FIG. 3.

[0035] Additionally, this gather / merge operation may include at least two parameters: i) num_bits_set(mask), and ii) request_size. If num_bits_set(mask)==request_size, this indicates that the merging of the response data is complete and a response can be generated and sent back to the processor engine 102.

[0036] 2 is a block diagram of an example computing system 200 for executing neural network models on a hardware integrated circuit, such as a machine learning hardware accelerator. Generally, computing system 200 is included in computing system 100, e.g., as a subsystem of computing system 100. Computing system 200 includes one or more computational tiles 106, a host interface 220, and a higher-level controller 225 (“controller 225”). As described in more detail below, host interface 220 and controller 225 cooperate to provide data sets and instructions to one or more computational tiles 106 of system 200.

[0037] In some implementations, the host interface 220 and the controller 225 are separate devices, while in other implementations, the host interface 220 and the controller 225 are the same device. The host interface 220 and the controller 225 can perform separate functions or can be integrated into a single device package. For example, the host 220 and the controller 225 can form a central processing unit (CPU) that interacts or cooperates with a hardware accelerator including multiple computational tiles 106. In some implementations, the host interface 220, the controller 225, and the multiple computational tiles 106 are included or formed as different sections on a single integrated circuit die. For example, the host 220, the controller 225, and the multiple computational tiles 106 can form a dedicated system-on-chip (SoC) optimized to perform processing of machine learning workloads, including neural network models for image processing applications.

[0038] Each computational tile 106 generally includes a controller 203 that provides one or more control signals 205 that cause inputs or activations of a vector of inputs 202 (“input vector 202”) to be stored in or accessed from memory locations in a first memory 208 (“memory 208”). Similarly, the controller 203 may also provide one or more control signals 205 that cause weights (or parameters) of a weight matrix structure 204 to be stored in or accessed from memory locations in a second memory 210 (“memory 210”). In some implementations, the vector of inputs 202 is obtained from an input tensor, while the weight matrix structure is obtained from a parameter tensor. Each of the input tensor and the parameter tensor may be a multidimensional data structure, such as a multidimensional matrix or tensor. This is described in more detail below with reference to FIG. 9.

[0039] Memory 208 and memory 210 are part of tiled memory 108. Each memory location in memories 208, 210 may be identified by a corresponding memory address, such as a logical address that has a corresponding mapping to a physical row of a physical memory bank of memory. Thus, similar to processor engine 102, computational tile 106 may also derive a set of contiguous addresses, such as virtual / logical addresses or physical addresses, from a group of requests.

[0040] Each of the memories 208, 210 may be implemented as a series of physical banks, units, or any other related storage medium or device. Each of the memories 208, 210 may include one or more registers, buffers, or both. In some implementations, each bank of the tile memory 108 of a computing tile 106 includes an arbiter that arbitrates access to that bank. For example, access to a bank may be arbitrated according to a bank generation function configured to alleviate some (or all) of the requests routed to the same physical memory bank.

[0041] In some implementations, memory 208 is an input / activation memory, while memory 210 is a parameter memory. In some other implementations, inputs or activations are stored in memory 208, memory 210, or both, and weights are stored in memory 210, memory 208, or both. For example, inputs and weights may be transferred between memory 208 and memory 210 to facilitate certain neural network calculations. In some implementations, each of memory 208 and memory 210 may be referred to as a tile memory 108.

[0042] Each computational tile 106 also includes an input activation bus 211, an output activation bus 215, and a computational unit 212, which has one or more hardware multiply-accumulate circuits (MACs) within each cell 214 a / b / c. The controller 203 can generate control signals 205 to retrieve operands stored in the memory of the computational tile 106. For example, the controller 203 can generate control signals 205 to retrieve i) an example input vector 202 stored in memory 208 and ii) weights 204 stored in memory 210. Each input retrieved from memory 208 is provided to the input activation bus 211 for routing (e.g., direct routing) to a computational cell 214 a / b / c within the computational unit 212. Similarly, each weight retrieved from memory 210 is routed to a cell 214 a / b / c of ​​the computational unit 212.

[0043] As described below, each cell 214a / b / c performs a calculation to obtain a partial sum or accumulation value to generate the output of a given neural network layer. An activation function may be applied to the set of outputs to generate a set of output activations for the neural network layer. In some implementations, the outputs or output activations are routed for storage and / or transmission via an output activation bus 215. For example, a set of output activations may be transmitted from a first computational tile 106 to a second, different computational tile 106 for processing as input activations for a different layer of the neural network in the second computational tile 106.

[0044] In general, each computational tile 106 and system 200 may include additional hardware structures to perform computations associated with multi-dimensional data structures, such as tensors, matrices, and / or data arrays. In some implementations, inputs for input vectors (or tensors) 202 and parameter tensor weights 204 may be preloaded into memories 208, 210 of a computational tile 106. The inputs and weights are received as a set of data values ​​that arrive at a particular computational tile 106 from a host 220 (e.g., an external host) via a host interface or from a higher-level control, such as a controller 225.

[0045] Each of the computational tiles 106 and the controller 203 may include one or more processors, processing devices, and various types of memory. In some implementations, the processors of the computational tiles 106 and the controller 203 include one or more devices, such as a microprocessor or central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a combination of different processors. Each of the computational tiles 106 and the controller 203 may also include other computational and storage resources, such as buffers, registers, control circuits, etc. These resources cooperate to provide additional processing options for performing one or more of the decisions and calculations described herein.

[0046] In some implementations, the processing unit(s) of the controller 203 execute instructions stored and programmed in memory to cause the controller 203 and the computational tiles 106 to perform one or more functions described herein. The memory of the controller 203 may include one or more non-transitory machine-readable storage media. The non-transitory machine-readable storage media may include solid-state memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information or instructions.

[0047] The system 200 receives instructions that define specific computational operations that the computational tiles 106 are to perform. The system 200 also receives data, such as inputs, activations, and weights (or parameters), that are operands for the computational operations. The system 200 may include one or more buses 209 used to route instructions and data. For example, the system 200 may include a first bus 209-1 that provides instructions, including associated commands, opcodes, and operational parameters (but not weights), to each of the one or more computational tiles 106 of the system 200. The system 200 may also include a second bus 209-2 that provides data or operands to each of the one or more computational tiles 106 of the system 200.

[0048] The bus 209 can be configured to provide one or more interconnected data communication paths between two or more computational tiles of the system 200. For example, the first bus 209-1 can be a ring bus that traverses each computational tile to convey a data set and an instruction (or instructions) to one or more computational tiles to execute an ML workload, while the second bus 209-2 can be a mesh bus that interconnects two or more tiles to provide data or operand sets between two or more computational tiles 106.

[0049] In some implementations, bus 209-1 and bus 209-2 are separate data buses. In some other implementations, bus 209-1 and bus 209-2 are the same data bus. In some cases, to process the exemplary workload, a first computational tile 106 arbitrates and executes access requests (e.g., read / write requests) for memory 208, where the requests are based on external data communications originating from outside the first computational tile 106, such as from a second, different computational tile 106. Such external communications may be received at the first tile 106 via bus 209-2 (e.g., a mesh bus).

[0050] Each computational tile 106 is an individual computing unit that cooperates with other tiles 106 in the system 200 to accelerate computations across one or more layers of a multi-layer neural network or across one or more sections of other machine learning structures. Each computational tile 106 can function as an individual computing unit. For example, each computational tile 106 is a self-contained computing component configured to perform a subset of tensor or ML computations independently of other computational tiles. In some implementations, computational tiles 106 can also share the execution of tensor computations associated with a given instruction.

[0051] In some implementations, the host can generate a set of parameters (i.e., weights) and corresponding inputs for processing in the neural network layers. The host can send the parameters to the computational tiles 106 via the host interface 220 for further processing in the tile. In some implementations, the host is the processor engine 102 or each core or processor of the processor engine 102. The controller 203 can execute programmed instructions to analyze the data stream associated with the received weights and inputs.

[0052] The controller 203 causes the inputs and weights of the data stream to be stored in the computational tiles 106. For example, the controller 203 may store the inputs and weights / parameters in the local tile memory 108 of the computational tiles 106. The controller 203 may also analyze the input data stream to detect operation codes (“opcodes”). The system 200 may support various types of opcodes, such as opcode types that indicate vector-matrix multiplication operations and / or element-wise vector operations.

[0053] Based on the one or more opcodes, the controller 203 may activate or execute a bank generation function to arbitrate requests for access to the tiled memory 108 of the computing tile 106. For example, the controller 203 may utilize the bank generation function to arbitrate two or more requests such that each of the two or more requests is processed to a different physical bank of the tiled memory. The controller 203 may employ a predetermined bank selection scheme that is programmed or encoded in the controller 203 before an inference decision is made in the computing tile 106.

[0054] In some implementations, a given computational operation involves multiple requestors, each requesting access to resources in memory 208. For example, a computational workload executing on a computational tile 106 may trigger memory access requests through tensor traversal operations that require read and write access to individual address locations in memory 208. System 200 may process these requests to access one or more bytes of data stored at address locations in memory 208, 210 using byte-level addressing functions of an exemplary instruction set architecture (ISA). As described below, these address locations may correspond to elements of input (or parameter / weight) tensors that are processed to execute the machine learning workload.

[0055] In addition to tensor read / write operations, processing a workload may also include processing read or write access requests for i) moving data (e.g., parameters) from memory 208 (narrow memory) to memory 210 (wide memory), and ii) moving data from memory 210 (wide memory) to memory 208 (narrow memory). Tensor operations (TensorOps) may be indicated by opcodes in instructions (e.g., single instructions) received at the computational tiles 106. For example, a “ByteAddressingMode” instruction may include one or more opcodes for tensor operations to be performed at the computational tiles 106 to traverse each element of an input tensor. In general, tensor operations can be performed to i) consume (or read) tensors from memories 208, 210, and ii) produce (or write) tensors to memories 208, 210.

[0056] As used in this document, "narrow" may refer to one or more memory units that each operate on and store data having a size (or width) of 16 bits or less, and "wide" may refer to one or more memory units that each operate on and store data having a size (or width) of 64 bits or less. For example, the size or width of narrow memory 208 may be between 8 bits and 16 bits, while the size or width of wide memory 210 may be between 32 bits and 64 bits. Of course, other sizes and data widths beyond these exemplary ranges are also contemplated for memories 208, 210.

[0057] 3 illustrates an example architecture 300 of a portion of a tensor mapping unit 104 within a compute tile 106. Specifically, architecture 300 is an example architecture for a second tensor module 104-n located in each of the compute tiles 106. Architecture 300 includes an input buffer 302 for storing one or more requests routed to a particular tensor module 104-n, and optional disassembly logic 304 for processing and / or parsing the component parts of the example request. The tensor mapping unit 104 is configured to receive requests that may include up to 32 bytes. Requests may also include more than 32 bytes based on design preference.

[0058] In some implementations, the respective data (e.g., 32B bytes) for each of these requests is mapped to one or more tile memory instances. A tile memory instance may refer to the narrow memory 208 of a computational tile 106. Aspects of how tensor data is mapped to a particular tile memory instance are established or defined based on a slicing function. An exemplary slicing function and its application to tensors is described below with reference to FIG. 5. In some implementations, the disassembly logic 304 includes a slicing function used to disassemble or parse the bytes of the 32B requests according to one or more tile memory instances. For example, in each computational tile 106, each second tensor module 104-n of the tensor mapping unit 104 may disassemble a request received on input (e.g., a request of 32 bytes or less) into several smaller sub-requests based on the disassembly logic 304.

[0059] The architecture 300 also includes access logic 306 for performing one or more lookup or access operations on memory locations in the tiled memory 108. In some implementations, the access logic 306 is configured to perform two or more sequential lookups on memory locations in the tiled memory 108. The sequential lookups may be based on nested loops or loop nests. The memory locations correspond to elements of a tensor, and the access logic 306 may perform multiple access iterations to access the memory locations. Each iteration is used to access one or more tensor elements, and each element (or tensor element) corresponds to a position element (x, y) along one or more dimensions of the tensor. In general, the access logic 306 may be based on a tensor mapping algorithm, which is described below with reference to FIG. 6.

[0060] The architecture 300 includes mask generation logic 308 for generating mask fields to be applied to data along the response path of a tensor module 104-n or a compute tile 106 that includes the tensor module 104-n. For example, when a tensor module 104-n receives a request, logic and compute elements of the architecture 300 are used to generate data representing a response to the request. This data is communicated using an exemplary data structure, such as a 32B data structure. The data structure may contain more than 32 bytes (e.g., 48B or 64B) or less than 32 bytes (e.g., 16B or 8B), depending on design preference.

[0061] Each tensor module 104-n is configured to receive and process at least read and write requests. Using the architecture 300, for a read request, a tensor module 104-n causes a 32-bit mask field to be generated from its respective computational tile 106. In the example of a 32B response to a request, this mask field indicates which bytes of the 32 bytes of data are valid in the response.

[0062] In some implementations, interface block 110 is configured to detect when a response to a particular request has arrived, read (or scan) the mask field of the response, and perform any necessary shifting or positioning operations. For example, interface block 110 can scan the response, detect the mask field, shift the response data to the appropriate position, and merge the response data with corresponding data from one or more previously received responses. Interface block 110 can support scanning or detecting the mask field and shifting the response data to perform any necessary parsing operations.

[0063] The tensor mapping unit 104 is configured to issue a write to the tile memory 108 when the corresponding strobe of the request bit is set. For example, each write request has a strobe field that indicates the byte offset to write. The tensor module 104-n is configured to generate a write response for that computational tile 106 to indicate the number of bytes written to the corresponding tile memory 108 of that computational tile 106. Furthermore, this write operation may include at least two parameters: i) num_bytes_written and ii) request_size. If num_bytes_written==request_size, this indicates that the write operation is complete and a write response can be generated and sent back to the processor engine 102.

[0064] As noted above, the size of an input request provided to a second tensor module 104-n of a tensor mapping unit 104 may be limited by some maximum number of bytes, such as 32 bytes, 16 bytes, or 64 bytes. If the size of an input request to a tensor mapping unit 104 is limited by the parameter max_input_bytes, then the number of smaller sub-requests that can be generated by the disassembly logic 304 is limited by the formula max_input_bytes / bytes_per_element, where "bytes_per_element" defines the size data (in bytes) of a given element of a tensor.

[0065] The tensor mapping unit 104 includes merge logic 310, which may optionally incorporate an output buffer that stores the response to each sub-request. In some cases, the merge logic 310, and any associated buffers, are implemented as a state machine configured to track each of the sub-requests. The merge logic 310 (or state machine) may track each sub-response for the corresponding sub-request and store each sub-response in an output buffer. For example, by utilizing the merge logic 310, given a 32B request, the tensor mapping unit 104 may send a response back to the ring bus 109 (e.g., fabric interface) only after the sub-responses from all sub-requests have been received and merged.

[0066] 4 is a block diagram illustrating aspects of a memory mapping process 400 (“process 400”). Process 400 represents a memory mapping operation for converting a set of logical (e.g., virtual) addresses maintained by the processor engine 102 to a set of physical addresses maintained by one or more computational tiles 106.

[0067] The system 100 executes the process 400 using a tensor mapping unit 104. More specifically, the tensor mapping unit 104 is configured to determine a mapping from a first set of addresses in a virtual address space 402 to a second, different set of addresses in a physical address space 404. The virtual address space 402 may be a contiguous virtual address view of a given tensor, and the physical address space 404 is a physical mapping of addresses distributed across two or more compute tiles 106.

[0068] The processor engine 102 may include a system port memory region 406. In some implementations, the processor engine 102 accesses the tile memory 108 by sending requests through the system port and routing those requests based on the AXI protocol. The tensor mapping unit 104 can receive and process these requests (e.g., AXI requests) at the system port boundary without having visibility into the source of the requests (e.g., LD / ST instructions).

[0069] The system port memory region 406 is used to expose programs executing on the processor engine 102 to the physical address space 404 of the computational tile 106. The virtual address space 402 may include a derived set of contiguous virtual addresses for logical memory accessed by programs on the processor engine 102. In general, the logical memory and virtual addresses may correspond to the respective physical aspects of each tile memory 108 of a particular computational tile 106.

[0070] The logical memory, its ports, and any associated arbiter may be implemented in software, hardware, or both. In some implementations, the logical memory and its ports, and the arbiter are controlled based on control signals generated by a controller of the processor engine 102. The logical memory may have multiple logical ports, and each port may be connected to or associated with a different requestor that requires access to the physical resources of the tiled memory 108. In some implementations, the requestor is the processor engine 102, a processor or core of the processor engine 102, or a program running on the processor engine 102.

[0071] To process a given access request (e.g., per port), the tensor mapping unit 104 determines the computational tile to which the request is routed based at least on one or more addresses in the request. For example, based on one or more instructions, the tensor mapping unit 104 generates a plurality of tensor element positions for a given tensor and then determines an assignment or mapping of a subset of the plurality of tensor element positions to the computational tile 106. The tensor mapping unit 104 can then analyze the address of the request for the tensor element position mapping or assignment information. Based on the analysis, the tensor mapping unit 104 can determine the tiled memory 108 to which the request is routed.

[0072] 5 shows examples of virtual and physical tensor layouts. In some implementations, programs executing on the processor engine 102 can access a virtual address space 402. The virtual address space 402 can include a group of linearized addresses that represent a tensor layout 502.

[0073] The tensor mapping unit 104 can map a virtual tensor layout 502 associated with a virtual address space to physical addresses of memory banks within the tile memory 108 of a given computational tile 106. More specifically, the tensor mapping unit 104 is configured to perform a mapping between linearized addresses of the tensor layout 502 in the program's virtual address space and physical addresses of the tile memory 108. The tensor layout 502 can include multiple tensor element positions 504 for an example tensor, such as tensor (A) described below.

[0074] The system 100 can use its tensor mapping unit 104 to determine one or more subsets 504-1, 504-2 for a given set of tensor element positions. In some implementations, the system 100 determines one or more subsets 504-1, 504-2 based on bypass mode computation logic, as also described below. Each subset 504-1, 504-2 can include one or more tensor element positions 504, where each tensor element position corresponds to a logical offset 506 per dimension. Each logical offset 506 can be generated as the output of a tensor mapping algorithm executed by the tensor mapping unit 104.

[0075] The tensor mapping unit 104 allows a program instance / context executing on the processor engine 102 to view the tensor(s) as if the tensors were located within a contiguous region of memory. This advantageous expanded view of tensors reduces overall program complexity in the system 100. For example, from a program's perspective, tensors (or tensor data) stored in the tiled memory 108 are exposed to the program in a logical layout such that the program instantiation on the processor engine 102 can access the tensors as if they were stored in main memory.

[0076] An example of this is shown in Figure 4 using virtual address space 402 and in Figure 5 using tensor layout 502. In other words, a program executing on processor engine 102 can view tensors as if they were stored in a contiguous memory region. However, in reality, tensors may be divided into multiple slices across different dimensions and stored in and across multiple tile locations. For example, an integrated circuit may include a grid of four computational tiles 106 identified as tile(0,0), tile(0,1), tile(1,0), and tile(1,1).

[0077] The exemplary tensor 508, tensor(A), may be divided (or sliced) based on a slicing function. The slicing function specifies how to slice the tensor across tiled memory instances. More specifically, the slicing function specifies the range of tensor data to be stored in each tiled memory instance. For multidimensional tensors, when a tensor has multiple dimensions, the slicing function can specify the range of tensor data in each dimension. The range can be specified based on a start (or beginning) element position [x,y] and an end element position [x,y], where each element position represents the position of a tensor element along a given dimension of the tensor.

[0078] As an example, tensor 508, tensor(A), is a two-dimensional 4x4 tensor (T[4][4]), with each dimension having four elements. Tensor(A) can be sliced ​​into a set of 2x2 computational tiles 106 using the following slicing function: slice_begin=stride_y*tile_id_y+stride_x*tile_id_x and slice_end=slice_begin+end (where stride_y=2, stride_x=2, end=1, and tile_id_y and tile_id_x are the coordinates of the computational tile). The following table shows the dimensions and extent of the tensor data that is written to the 2x2 set of computational tiles 106 according to the slicing function: [Table 1]

[0079] 5, tensor (A) is divided (or sliced) such that a first partition 508-1 is stored in tile (0,0), a second partition 508-2 is stored in tile (0,1), a third partition 508-3 is stored in tile (1,0), and a fourth partition 508-4 is stored in tile (1,1). As noted above, this ability to divide / slice, distribute, and store tensor dimensions across computational tiles 106 and tile memory 108 locations is enabled, in part, by dedicated portions of computational logic and / or hardware circuitry for the tensor mapping unit 104 located within each computational tile 106.

[0080] 6 shows an example tensor mapping table 600. The tensor mapping table 600 includes a number of data items 602 (e.g., parameters) corresponding to an example tensor mapping algorithm. For example, the tensor mapping table 600, including one or more of the values ​​of each of the table's parameters, may be generated based on the example tensor mapping algorithm. The tensor mapping algorithm is executed or implemented in the system 100 using the tensor mapping unit 104.

[0081] A tensor mapping algorithm may be executed to perform address remapping from the linearized set of virtual / logical addresses of the processor engine 102 to physical addresses of the respective tile memories 108 across one or more computational tiles 106. The tensor mapping unit 104 is configured to store tensor mapping information in a data structure representing a tensor mapping table 600. In some implementations, the tensor mapping unit 104 stores multiple instances of the tensor mapping table 600, one or more of the multiple instances being stored across each of the second tensor modules 104-n.

[0082] For example, a first tensor mapping table 600 may be stored in the tensor module 104-0 of a first computational tile 106-0, and a second, different tensor mapping table 600 may be stored in the tensor module 104-1 of a second, different computational tile 106-1. In some implementations, the tensor mapping table 600 is stored in the tensor module 104-n using a set of CSRs 230. In some other implementations, the tensor mapping table 600, including the various data / parameter values ​​associated with the table, is stored in the tensor module 104-n using one or more exemplary memory devices, such as registers 230, buffers, program counters, etc. The tensor mapping table 600 may also be stored using a combination of such devices.

[0083] Generally, the system 100 uses tensor modules 104-n to process one or more tensors based on requests or instructions from a program instance. For each incoming request, the tensor mapping unit 104 is configured to traverse one or more appropriate instances of its tensor mapping table 600. For example, when processing a given tensor, the tensor mapping unit 104 can generate control signals that trigger traversal operations on respective tensor mapping tables 600 stored in some (or all) of the computational tiles 106. More specifically, based on instructions or incoming requests, control signals are generated to cause a particular tensor module 104-n to traverse various portions of the data / parameters 602 of the tensor mapping table 600 stored in the tensor module 104-n.

[0084] A tensor module 104-n traverses its tensor mapping table 600 to identify and / or determine whether its table 600 contains mapping information relevant to the incoming request. In some implementations, some of the information in the tensor mapping table 600 is replicated across computational tiles 106 such that each tensor mapping table 600 stored in each computational tile 106 contains the same portion of information. For example, values ​​for a substantial portion of the information or data / parameters 602 may be replicated across computational tiles 106 such that each computational tile 106 has the same CSR value.

[0085] This replication translates into programming efficiencies. For example, rather than programming values ​​for each computational tile 106 (e.g., programming kNumTiles times), the system 100 can program these register values ​​once and broadcast the values ​​to the computational tiles 106. In some implementations, the interface fabric includes a register / CSR broadcast bus. The system 100 uses this CSR broadcast bus to broadcast tensor mapping table values ​​that are replicated across the computational tiles 106. One or more data items and / or parameters of the tensor mapping table 600 can be programmed by the control logic (e.g., a system controller) of the system 100, the processor engine 102, or both. For example, “TPU context ID” information can be programmed by the control logic, while other information or parameters can be programmed by the processor engine 102.

[0086] As described above, the tensor mapping table 600, including one or more of its respective parameter values, can be generated based on a tensor mapping algorithm. The tensor mapping algorithm can include one or more steps. A request for tensor data includes an address that is converted to a tiled memory physical address based on one or more steps of the tensor mapping algorithm. In some implementations, the tensor mapping algorithm includes: i) a first step in which a linearized set of virtual / logical addresses is converted from the virtual address space of the processor engine 102 to a logical offset per dimension; and ii) a second step in which the logical offset per dimension is converted to a tiled memory physical address.

[0087] Exemplary pseudocode for the first transformation step of the tensor mapping algorithm is shown below: / / Convert the linearized addresses from DIVE to logical offsets per dimension. for each TPUContextID: if (IncomingTpuContextId != TPUContextID) continue; for each TensorMappingTable: if (IncomingAddress >= TensorMappingTable.base_address && IncomingAddress < TensorMappingTable.base_address + TensorMappingTable.tensor_size) { offset = IncomingAddress - TensorMappingTable.base_address; dim0_offset = offset % dive_tensor_shape.dim0 dim1_offset = offset / dive_tensor_shape.dim0 % dive_tensor_shape.dim1 dim2_offset = offset / (dive_tensor_shape.dim0 * dive_tensor_shape.dim1) % dive_tensor_shape.dim2 dim3_offset = … } } }

[0088] Based on the results of the first conversion step, the tensor mapping unit 104 can determine whether the corresponding tile memory 108 contains the data of the request. This determination can include a scan or probe performed by a particular tensor module 104-n of the tensor mapping unit 104. This scan or probe can be performed to detect whether there is an overlap between the access range of the incoming request and the slice range in the corresponding tile memory 108 of the computational tile 106 that includes the particular tensor module 104-n.

[0089] If a particular tile memory 108 does not contain data associated with a request, the system 100 may generate an early response indicating, for example, that there is no data for the request. In some implementations, the system 100 generates an early response and terminates the tensor mapping algorithm, thereby avoiding operations for the second transformation step. The early response and corresponding termination translates into computational savings and efficiency by avoiding unnecessary processor cycles and resource overhead for the second transformation step.

[0090] To simplify hardware implementation, the tensor mapping unit 104 may perform rounding on the access range of a request so that the access range is a hyper-rectangle. A hyper-rectangle is a generalized rectangle of d-dimension (higher dimension). For example, an access range of five elements may be rounded to eight elements to form a rectangle. This rounding may be performed before performing an overlap check between the access range and the slice range.

[0091] In some implementations, the slice range (slice_range) is set (e.g., set consistently) as a hyper-rectangle. Thus, in these implementations, the overlap check involves determining whether two hyper-rectangles overlap. This check can be performed using a set of comparators configured based on any known algorithm to find the overlap length along each corresponding dimension of the hyper-rectangles. In some implementations, the check can be performed to evaluate the correctness of the hardware's functionality.

[0092] If the tensor mapping unit 104 determines that the tile memory 108 of the computational tile 106 should accommodate the request, the tensor mapping unit 104 causes the tensor mapping algorithm to proceed to the second transformation step. Exemplary pseudocode for the second transformation step of the tensor mapping algorithm is shown below: #The dimension ordering between the logical (PE view) and physical (tile view) is the same, #Ensures that nested loops can be terminated after a size iteration. #The size is provided through the ARSIZE field of the AR channel. count = 0 slice_range = … ​​ / / Derive slice range using affine coefficients slice_shape = … ​​ / / Derive the slice shape using affine coefficients for dim2_offset = i; dim2_offset < dim2_dim; i++: for dim1_offset = j if first_iteration else 0; dim1_offset < dim1_dim; j++: for dim0_offset = k if first_iteration else 0; dim0_offset < dim0_dim; k++: if (dim2_offset, dim1_offset, dim0_offset) in slice_range: tile_mem_addr = flatten_addr((dim2_offset, dim1_offset, dim0_offset), slice_shape) mask |= (1 << count) count++; if (count == size): return

[0093] The above pseudocode for the second transformation step uses reads from a rank-3 tensor as an example. In some implementations, to reduce hardware costs, the system 100 may implement one or more assumptions about aspects of tensor slicing in the tile memory 108. Under this assumption, the address translation operations associated with the second transformation step are performed using nested loops. This assumption may include configuring the exemplary slicing function to be thread-oblivious and ensuring that the ordering of dimensions is the same between the logical address space (the PE's view) and the physical address space (the compute tile's view). Configuring the slicing function to be thread-oblivious reduces hardware costs by having a single base_address CSR per compute tile 106 in the tensor mapping table 600.

[0094] In some implementations, the slicing function is based on a transformation operation performed on the input samples, e.g., as a preprocessing step prior to the inference computation. For example, the slicing function may be based on an affine transformation used to determine a mapping of tensors to the computational tiles 106. The tensor mapping unit 104 may determine the assignment of different subsets of tensor element positions to one or more of the computational tiles 106 according to a slicing function that is based on the affine transformation.

[0095] A respective tensor mapping algorithm may be executed in each of the computational tiles 106. In some implementations, the tensor mapping algorithm is executed in each computational tile simultaneously. In each computational tile 106, the tensor mapping algorithm is executed to read that tile's tile memory 108 and generate (and / or populate) a mask that indicates which bytes are to be read from the particular slice of tensor elements that are allocated or mapped to that computational tile 106.

[0096] 7 shows example memory regions and the distribution of tensor data within those memory regions. In some implementations, system 100 includes a bypass mode in which processor engine 102 can be configured to access tile memory 108 without going through (or directly using) tensor mapping unit 104.

[0097] Example scenarios for using bypass mode may include, but are not limited to, i) the number of tensors shared between the processor engine 102 and the computational tile 106 is greater than the number of hardware-provisioned tensor mapping tables, and ii) the processing and resource overhead for programming the CSR 230 in the tensor module 104-n exceeds the associated software translation overhead (which may occur when the data being accessed is very small (e.g., less than 64 data elements or between 64 and 128 data elements)).

[0098] The bypass mode functions and capabilities are based in part on a dual domain mapping configuration in system 100. Dual domain mapping enables bypass mode functions such that one or more tensors can be accessed while bypassing tensor mapping unit 104, while one or more other tensors can be accessed through (e.g., using) tensor mapping unit 104. With dual domain mapping, system 100 can include (or determine) a mapping of tile memory 108 in the address space of processor engine 102. For example, system 100 can map tile memory 108 across two regions within the address space of processor engine 102.

[0099] As shown in the example of FIG. 7 , the two regions include a continuous region 702 (“C region 702”) and a distributed region 704 (“D region 704”). Each of the regions 702, 704 may have a size (in bytes) that is kNarrowMemoryCapacity*kNumTiles bytes. In some implementations, for a given number of tiles (kNumTiles), the size may also be based on or include kWideMemoryCapacity. Referring again to dual region mapping, the system 100 may map the tiled memory 108 across the C region 702 and the D region 704 within the address space of the processor engine 102. This address space of the processor engine 102 may include or correspond to the virtual address space 402 described above.

[0100] The C region 702 can be configured to provide a contiguous virtual address view of a given tensor, such that the processor engine 102 sees the tensor (or tensor data) stored in the C region as laid out contiguously in memory. Therefore, addresses in the C region 702 cannot be used to directly access the tile memory 108. Once a tensor in the C region 702 is mapped using the tensor mapping unit 104, the processor engine 102 can generate a request and communicate the address of this C region 702 directly via the system port 406.

[0101] In some implementations, the fabric interface detects such a request and triggers a conversion operation in the tensor mapping unit 104. For example, the interface block 110 and the first and second tensor modules of the tensor mapping unit 104 cooperate to detect such a request and trigger a conversion operation using respective portions of the mapping information stored in the tensor mapping table of the tensor module 104-n.

[0102] The D region 704 is configured to directly map the physical address space of the tiled memories 108 to the memory / address space of the processor engine 102. For example, an integrated circuit may include one or more clusters of computational tiles 106, with each cluster including at least four tiles, Tile 0 through Tile 3. The D region 704 may provide a direct mapping of the physical address space for each tiled memory 108 in each of the computational tiles 106-0, 106-1, 106-2, and 106-3.

[0103] In some implementations, addresses contained within the D region 704 do not pass through the tensor mapping unit 104. Rather, the processor engine 102 may include an exemplary bus interface that couples the processor engine 102 to the bus 109 and / or a fabric interface. The processor engine 102 may generate requests that are routed to the bus interface for transmission through the tiled memory fabric. The requests may be incoming requests that include one or more addresses. The system 100 may generate a command packet for the incoming request, and the bus interface may populate a bypass field in the command packet based on the region (e.g., D region 704) in which the incoming request falls.

[0104] Tensors for a particular program instance (e.g., an inference workload) may be mapped to C domain 702. In some cases, a request for data related to that tensor may imply mapping information that is not present in tensor mapping unit 104. If the fabric interface receives a request that falls into C domain 702, but tensor mapping unit 104 does not include any associated mapping information specific to that request, system 100 may be configured to generate an error (e.g., an error flag).

[0105] Specifically, when the mapping for that tensor does not pass through the tensor mapping unit 104, the program instance of the processor engine 102 can trigger or perform a software-based translation of the addressing information associated with that tensor. The software-based translation translates addresses in the C region 702 to addresses in the D region 704 before sending the addressing information to the system port 406. As described above, addresses in the D region 704 represent physical addresses in the tiled memory 108.

[0106] The system 100 can use two / dual region mapping to map the same tile memory 108 to avoid address aliasing. For example, two tensors (both uint8) can be stored in a four-tile configuration including computation tiles 106-0, 106-1, 106-2, and 106-3. The four computation tiles 106 can include a total of 4 MB of memory space. A first tensor 702-A, "tensor A," can include one million ("1M") elements, while a second, different tensor 702-B, "tensor B," can include two million ("2M") elements.

[0107] Data for 1M elements of tensor 702-A may be distributed and stored in the first 256 KB of memory 708 within each tile memory 108 across four tiles. Relatedly, data for 2M elements of tensor 702-B may be distributed and stored in the last 512 KB of memory 710 within each tile memory 108 across four tiles. The example of FIG. 7 shows the respective memory layouts of each memory region 702, 704. As this example implies, system 100 cannot represent both tensor A and tensor B using one memory region without aliasing. Therefore, system 100 is operable to use two / dual regions to map the same tile memory 108 into a virtual address space to avoid address aliasing.

[0108] 8 is an example process 800 of a tensor mapping operation performed using the tensor mapping unit 104 described above. In general, process 800 can be performed or implemented using the systems 100, 200 described above. Accordingly, the description of process 800 may refer to the aforementioned computing resources of systems 100, 200. In some examples, the steps or actions of process 800 are enabled by programmed firmware instructions, software instructions, or both. Each type of instruction can be stored in a non-transitory, machine-readable storage device and can be executed by one or more of the processors or other resources described herein, such as a hardware accelerator or computational tiles of a neural network processor.

[0109] In some implementations, the steps of process 800 are performed in a hardware integrated circuit to generate machine learning (ML) output, including the output of a neural network layer of a neural network that executes an ML model. For example, the output may be part of a computation for an ML task or inference workload to generate an image processing or image recognition output. As indicated above, the integrated circuit may be a dedicated neural network processor or a hardware ML accelerator configured to accelerate computations to generate different types of data processing outputs.

[0110] Referring again to process 800, system 100 generates executable instructions for computing the output of an ML model (802). As described above, the ML model executes on an integrated circuit configured to accelerate tensor computations for an ML task or inference workload. The instructions may be generated by processor engine 102 and routed to tensor mapping unit 104. For example, the instructions may be routed to a first tensor module of tensor mapping unit 104.

[0111] The system 100 generates 804 a plurality of tensor element positions for the tensor based on the instructions. The first tensor module processes and / or executes the instructions to generate 804 a plurality of tensor element positions corresponding to a tensor, such as an input tensor processed by the ML model to generate a model output. The input tensor may correspond to an input image processed by the ML model. In some implementations, the tensor mapping unit 104 generates the tensor element positions by performing an affine transformation on the input image based on the instructions received from the processing engine 102.

[0112] The system 100 determines 806 that a first subset of tensor element locations are to be assigned to a first computational tile of the integrated circuit. For example, the tensor mapping unit 104 generates a tensor mapping table 600 that includes mapping information specific to: i) tensor element locations; ii) a subset of the tensor element locations; and iii) an assignment of the subset to a computational tile 106. In some implementations, the tensor mapping unit 104 can determine 806 that the first subset is to be assigned to the first computational tile based on the tensor slicing and / or tensor element assignments of the tensor mapping table 600.

[0113] For example, the tensor mapping unit 104 may determine the allocation of different subsets of tensor element positions to one or more of the computational tiles 106 according to information in the tensor mapping table 600. In some implementations, the exemplary tensor allocation and slicing is based on a slicing function, which in turn is based on a particular transformation operation applied to the input samples. In some examples, the transformation operation is an affine transformation performed on the input image (e.g., the input samples).

[0114] The system 100 generates control signals to retrieve groups of inputs from the tile memory of the first computational tile based on the first subset (808). More specifically, the tensor mapping unit 104 includes a plurality of second tensor modules 104-n that generate control signals to be used / processed in the computational tiles 106 and / or the tile memory 108 to retrieve data corresponding to the first subset of tensor element positions. For example, each tensor module 104-n of the tensor mapping unit 104 can include a set of tensor mapping registers (e.g., CSRs). Each tensor module 104-n can be configured to generate control signals for programming and address translation operations in the computational tile 106 that includes the tensor module 104-n.

[0115] In some implementations, each tensor module 104-n in a computational tile 106 is configured to process a predefined number of multidimensional tensors, and the tensor module 104-n may include a set of CSRs for each tensor. Control signals may be generated to program the values ​​of the CSRs and to convert tensor indices in the tensor mapping table 600 of the tensor module 104-n to corresponding memory address locations. For example, the tensor indices may be represented by tensor element positions, and each element position may correspond to an input (e.g., a pixel of an image) in a group of inputs.

[0116] In some implementations, the affine transformation is performed to preprocess an image for an image recognition task of a machine learning workload. For example, the preprocessing task may perform image resizing or cropping before processing an inference task on the image to recognize one or more objects in the image. Thus, tensor offsets or tensor element positions may be generated based on a slicing function that defines how tensor elements and / or dimensions are sliced ​​for allocation and mapping to tile memory 108 that performs the affine transformation.

[0117] Figure 9 shows an example of a tensor or multidimensional matrix 900 that includes an input tensor 904, variations of a parameter tensor 906, and an output tensor 908. In the example of Figure 9, each of the tensors 900 includes individual elements, where each element can correspond to an individual data value (or operand) for a computation performed at a given layer of a neural network.

[0118] For example, each input of the input tensor 904 may correspond to a respective element along a given dimension of the input tensor 904, each weight of the parameter tensor 906 may correspond to a respective element along a given dimension of the parameter tensor 906, and each output value or activation in the set of outputs may correspond to a respective element along a given dimension of the output tensor 908. Relatedly, each element may correspond to an individual memory location or address in the memory of the computational tile 106 that is assigned to operate on one or more dimensions of the given tensors 904, 906, 908.

[0119] The computations performed at a given neural network layer may include multiplying an input / activation tensor 904 and a parameter / weight tensor 906 over one or more processor clock cycles to obtain a layer output, which may include an output activation. Multiplying an activation tensor 904 and a weight tensor 906 includes multiplying an activation from an element of tensor 904 with a weight from an element of tensor 906 to obtain one or more partial sums. The example tensors 906 in FIG. 9 may be unmodified parameter tensors, modified parameter tensors, or a combination thereof. In some implementations, each parameter tensor 906 corresponds to a modified parameter tensor that includes non-zero compressed sparse parameter values ​​derived based on a sparsity exploitation technique that exploits sparsity (e.g., zero values) in the parameter tensor.

[0120] The processor cores of the system 100 can operate on i) scalars corresponding to discrete elements in a multidimensional tensor 904, 906, ii) vectors of values ​​(e.g., input vector 102) that include multiple discrete elements 907 along the same or different dimensions of a multidimensional tensor 904, 906, or iii) combinations thereof. In a multidimensional tensor, a discrete element 907, or each of multiple discrete elements 907, may be represented using X, Y coordinates (2D) or X, Y, Z coordinates (3D), depending on the dimensionality of the tensor.

[0121] The system 100 can calculate multiple partial sums corresponding to products generated from multiplying the batch inputs by corresponding weight values. As described above, the system 100 can perform the accumulation of products (e.g., partial sums) over many clock cycles. For example, the product accumulation can be performed in random access memory, shared memory, or scratchpad memory of one or more computational tiles based on the techniques described herein. In some implementations, the input weight multiplication can be written as a sum of products of discrete inputs of the input vector 102 multiplied by each weight element, such as a row or slice of the input tensor 904. The row or slice can represent a given dimension, such as a first dimension 910 of the input tensor 904 or a second, different dimension 915 of the input tensor 904.

[0122] In some implementations, a set of exemplary calculations can be used to calculate the output of a convolutional neural network layer. The calculation of a CNN layer can include performing a 2D spatial convolution between a 3D input tensor 904 and at least one 3D filter (weight tensor 906). For example, one 3D filter 906 can be convolved onto the 3D input tensor 904 to obtain a 2D spatial plane 920 or 925. The calculation can include calculating a sum of dot products for a particular dimension of an input volume that includes the input vector 102.

[0123] For example, spatial plane 920 may contain output values ​​of sum-of-products computed from inputs along dimension 910, while spatial plane 925 may contain output values ​​of sum-of-products computed from inputs along dimension 915. The computations to generate the sum-of-products of the output values ​​in each of spatial planes 920 and 925 may be performed i) in computation cells 214a / b / c, ii) directly in memory 210 using arithmetic units coupled to a shared bank of memory 210, or iii) both. In some implementations, various techniques for reducing accumulated values ​​may be used to streamline the reduction operation and perform it directly in memory cells (or locations) of memory 210.

[0124] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and structural equivalents thereof, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus.

[0125] Alternatively or additionally, the program instructions may be encoded in an artificially generated propagated signal, such as a mechanically generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof.

[0126] The term "computing system" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, computer, or multiple processors or computers. An apparatus can include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can also include code that establishes an execution environment for a subject computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0127] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted, or declarative or procedural languages, and can be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0128] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

[0129] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array), or an ASIC (application specific integrated circuit), or a GPGPU (general purpose graphics processing unit).

[0130] A computer suitable for executing a computer program can include, for example, be based on, a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory, a random-access memory, or both. Some elements of a computer are the central processing unit for implementing and executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated into other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0131] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0132] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented in a computer having a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices can also be used to interact with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0133] Embodiments of the subject matter described herein can be implemented in a computing system including a back-end component, e.g., a data server, or a computing system including a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface or web browser to allow a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0134] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0135] While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0136] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0137] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. An integrated circuit configured to accelerate tensor computations of a machine learning model executed on the integrated circuit, the integrated circuit comprising: a processing engine; a tensor mapping unit; the tensor mapping unit comprises: generating a plurality of tensor element positions of a tensor based on instructions received from the processing engine; determining that a first subset of the plurality of tensor element locations is assigned to a first computational tile of the integrated circuit; generating control signals to obtain a group of inputs from a tile memory of the first computational tile based on the first subset; configured to perform Integrated circuit.

2. The integrated circuit of claim 1 , wherein the tensor mapping unit is configured to generate the plurality of tensor element positions by performing an affine transformation on an input image based on the instructions received from the processing engine.

3. The integrated circuit includes a plurality of computation tiles, and the tensor mapping unit further comprises: The integrated circuit of claim 1 , configured to determine an allocation of different subsets of the plurality of tensor element positions to one or more of the plurality of computational tiles.

4. The tensor mapping unit includes a first tensor module, the first tensor module comprising: generating the plurality of tensor element positions; The integrated circuit of claim 3 , further configured to generate a plurality of tensor element positions for each of a plurality of tensors based on one or more instructions from the processing engine.

5. the tensor mapping unit includes a plurality of second tensor modules that generate control signals to obtain data corresponding to the plurality of tensor element positions; The integrated circuit of claim 4 , wherein the plurality of tensors are processed in the integrated circuit using a respective second tensor module and a respective one of the plurality of computation tiles.

6. The tensor mapping unit is configured to generate a tensor mapping table, the tensor mapping table comprising: i) the plurality of tensor element positions; ii) a plurality of subsets of the plurality of tensor element positions; iii) assigning the first subset to the first computational tile; and 6. The integrated circuit of claim 5, further comprising mapping information that is specific to a

7. 6. The integrated circuit of claim 5, wherein the tensor mapping unit is configured to determine a mapping from a first set of addresses in a virtual address space to a second, different set of addresses in a physical address space.

8. The integrated circuit of claim 5 , wherein the first tensor module is coupled to the processing engine and is external to each of the plurality of computational tiles.

9. The integrated circuit of claim 8 , wherein a second tensor module of the plurality of second tensor modules is located in each computational tile of the plurality of computational tiles included in the integrated circuit.

10. each second tensor module configured to access the tile memory of the computational tile that includes that second tensor module; The integrated circuit of claim 9 , wherein the second tensor module accesses the tile memory based on a particular subset of the plurality of tensor element positions assigned to that computational tile.

11. each input of the group of inputs corresponds to a respective tensor element position of the first subset; The integrated circuit of claim 1 , wherein each element of the tensor corresponds to an input and an address location in a tiled memory where the input is stored.

12. the plurality of tensor element positions correspond to different dimensions of two or more multidimensional tensors; 2. The integrated circuit of claim 1, wherein at least one of the multidimensional tensors is an input tensor or a parameter tensor that is processed in the integrated circuit using the processing engine, the tensor mapping unit, and one or more computational tiles.

13. 1. A method for accelerating tensor computations for a machine learning (ML) model executed on an integrated circuit including a processing engine and a tensor mapping unit, the method comprising: generating, by the processing engine, instructions used to calculate outputs of the ML model; The instructions are executable on the integrated circuit to cause performance of operations, the operations being: generating, by the tensor mapping unit, a plurality of tensor element positions of a tensor based on the instructions; determining, by the tensor mapping unit, that a first subset of the plurality of tensor element locations is assigned to a first computational tile of the integrated circuit; generating, by the tensor mapping unit, control signals to retrieve a group of inputs from a tile memory of the first computational tile based on the first subset; Including, method.

14. Generating the plurality of tensor element positions includes:

14. The method of claim 13, comprising generating the plurality of tensor element positions by performing an affine transformation on an input image based on the instructions generated by the processing engine.

15. The integrated circuit includes a plurality of computational tiles, and the method comprises: The method of claim 13 , comprising determining, by the tensor mapping unit, assignments of different subsets of the plurality of tensor element positions to one or more of the plurality of computational tiles.

16. The tensor mapping unit includes a first tensor module, the first tensor module comprising: generating the plurality of tensor element positions; The method of claim 15 , further comprising generating a plurality of tensor element positions for each of a plurality of tensors based on one or more instructions from the processing engine.

17. The tensor mapping unit includes a plurality of second tensor modules, and the method includes: generating, by the plurality of second tensor modules, control signals for acquiring data corresponding to the plurality of tensor element positions; processing the plurality of tensors on the integrated circuit using each second tensor module and each of the plurality of computation tiles; 17. The method of claim 16, comprising:

18. The method further includes generating a tensor mapping table by the tensor mapping unit, the tensor mapping table comprising: i) the plurality of tensor element positions; ii) a plurality of subsets of the plurality of tensor element positions; iii) assigning the first subset to the first computational tile; and 20. The method of claim 17, further comprising mapping information that is specific to the

19. 20. The method of claim 17, further comprising determining, by the tensor mapping unit, a mapping from a first set of addresses in a virtual address space to a second, different set of addresses in a physical address space.

20. the first tensor module is coupled to the processing engine and is external to each of the plurality of computational tiles; a second tensor module of the plurality of second tensor modules is located in each computational tile of the plurality of computational tiles included in the integrated circuit; each second tensor module configured to access the tile memory of the computational tile that includes that second tensor module; 18. The method of claim 17, wherein the second tensor module accesses the tile memory based on a particular subset of the plurality of tensor element positions assigned to that computational tile.

Images