Efficient Allocation of Memory in Computational Tiles of Neural Networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a large neural network. One of the methods includes obtaining data indicative of a neural network having a plurality of layers, assigning a subset of a plurality of computing units for performing, for each layer in the subset of the plurality of layers, at least in part, an inference computation associated with the layer, determining a memory size and a common memory address for each addressable memory unit of each computing unit assigned to the layer, and generating sharing instructions including memory allocation instructions. The memory allocation instructions, when executed by each of the subset of the plurality of computing units, cause the computing unit to store results of performing the inference computation associated with the layer in the determined common memory address having the determined memory size in the addressable memory units of the computing unit.

Description

[Background technology]

[0001] background This specification relates generally to neural networks, and more particularly to processing inputs to neural networks on hardware accelerators having multiple compute tiles.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output given a received input.

[0003] Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the inputs it receives according to the current values ​​of its respective set of network parameters.

[0004] Network parameters for a neural network are values ​​that affect the operations performed by the neural network and are also values ​​that are adjusted as part of training. For example, network parameters may include values ​​of the weight matrices and, in some cases, the bias vectors of a network layer of a neural network. Summary of the Invention

[0005] overview This specification generally describes techniques for generating instructions that cause computing units of a hardware computing system to efficiently allocate memory while processing inputs to particular types of neural network layers.

[0006] In general, one innovative aspect of the subject matter described herein can be embodied as a method that includes operations of obtaining data indicative of (i.e., defining) a neural network including a plurality of layers; selecting a subset of the plurality of layers from among the plurality of layers of the neural network based on the obtained data; assigning, for each layer in the subset of the plurality of layers, a subset of the plurality of computing units from among a plurality of computing units each including a respective addressable memory unit, to at least partially perform an inference computation associated with the layer; determining, in the subset of the plurality of computing units assigned to the layer, a memory size and a common memory address for each addressable memory unit of each computing unit; and generating sharing instructions that, when executed by each of the subset of the plurality of computing units, cause the computing unit to store a result of performing the inference computation associated with the layer in the determined common memory address having the determined memory size in the addressable memory of the computing unit.

[0007] An operation in a method embodying this innovative aspect of the subject matter further includes providing the shared instructions to a plurality of computing units. The method can be performed by a computer system, such as a hardware accelerator, including a plurality of computing units. The computer system can further include a controller for controlling the plurality of computing units to perform parallel processing based on the instructions transmitted by the controller to the plurality of computing units.

[0008] Another innovative aspect of the subject matter described herein may be embodied as a method that includes providing a set of instructions to a system including a plurality of computing units for performing inference calculations for a plurality of layers of a neural network, each computing unit including a respective addressable memory, the set of instructions including first memory allocation instructions associated with a first layer of the plurality of layers of the neural network and identifying a first memory address of the respective addressable memory and a first subset of the plurality of computing units, and second memory allocation instructions associated with a second layer of the plurality of layers of the neural network and identifying a second memory address of the respective addressable memory and a second subset of the plurality of computing units, the second memory address being different from the first memory address, and the second subset being different from the first subset.

[0009] The set of instructions causes the system to output, for each computing unit in the first subset, a result of an inference calculation associated with a first tier in the plurality of tiers to a respective memory address of the addressable memory of the computing unit based on the first memory address, and to output, for each computing unit in the second subset, a result of an inference calculation associated with a second tier in the plurality of tiers to a respective memory address of the addressable memory of the computing unit based on the second memory address.

[0010] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the methods.

[0011] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0012] A computing system that implements a neural network model in a hardware accelerator having multiple computing units (e.g., compute tiles) can use the described techniques to efficiently perform inference computations for the neural network model by issuing a single shared instruction to the computing units associated with a layer of the neural network model. The shared instruction identifies a shared memory address where the computing units store, fetch, and accumulate partial results computed by the computing units associated with that layer. Instead of using the described techniques, in some situations where a computing unit is assigned different memory addresses to store results from an inference computation at a particular layer, the system would need to issue multiple separate instructions to each of the computing units. However, a system employing the described techniques can issue a single shared instruction to send to each computing unit in a particular layer, thereby reducing the total number of instructions required by at least a factor N, where N is the total number of computing units used to perform inference computations at a particular network layer.

[0013] By issuing shared instructions, the system can reduce the overall instruction bandwidth required and reduce memory usage. Thus, in a situation where a system (which may be a hardware accelerator) includes a controller that controls it and the instruction bandwidth and memory of the controller's instruction memory are constrained, the system can directly load the shared instruction into the controller's instruction memory instead of fetching parts of separate instructions from the host to the controller at each of multiple times. In this way, the system can reduce or even eliminate the time wasted in transmitting multiple parts of separate instructions. In some implementations, the described technique can reduce the overall instruction bandwidth required by more than 50% when compared to conventional methods.

[0014] Furthermore, systems employing the described techniques are capable of performing inference calculations for fairly large neural network models. Previously, it was difficult or even impossible to compute large neural networks on certain hardware accelerators. The number and size of instructions required to allocate memory for inference calculations of large neural networks increases with the number of neural networks and network layers, so the overall instruction size can be quite significant. When the instruction size is large, the system needs to send multiple parts of separate instructions from the host to the controller multiple times, which can be inefficient and increase pause overhead. Furthermore, for large neural networks with repeating loops, it is error-prone for the system to correctly fetch multiple parts of separate instructions from the host to the controller.

[0015] However, when a system issues shared instructions as described herein, the size of the instructions can be significantly reduced, and thus the system can directly load all shared instructions to avoid fetching instructions multiple times from the host, reducing pause times (e.g., reducing the time spent loading shared instructions), and to avoid potential errors when fetching instructions to compute a neural network with an iterative loop.

[0016] Furthermore, the described techniques can reduce memory usage. For example, a system using shared instructions can reduce memory allocation for storing activation inputs, weight inputs, and hyperparameters for a particular machine learning model in a computing unit (e.g., a GPU or TPU), which can result in improved computational efficiency for performing inference calculations for the particular machine learning model, for example.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates an exemplary hardware computing system for performing inference calculations of a neural network model. [Figure 2A] FIG. 1 is a diagram of an example of a conventional memory allocation for layers and associated computation tiles of a neural network model. [Figure 2B] FIG. 1 illustrates an example of a common memory allocation for eligible layers and associated computation tiles of a neural network model. [Diagram 3] FIG. 1 illustrates an example allocation of additional memory addresses for computational tiles associated with a fully connected layer of a neural network model. [Figure 4] FIG. 2 is a diagram of an exemplary detailed architecture for computational tiles in the system. [Diagram 5] FIG. 1 illustrates an example process for memory allocation of computational tiles in a system for performing inference computations for neural network models. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Detailed Description This specification describes techniques for improving the efficiency of inference calculations in hardware computing systems that include multiple computing units when processing inputs in a neural network model.

[0020] Each computing unit of the hardware computing system is self-contained and can independently perform the computations required by at least a portion of a given layer of a multi-layer neural network. The described techniques can reduce the required instruction memory size and improve computational efficiency in the hardware computing system when performing inference computations for deep or large-scale neural network models.

[0021] A trained neural network model with multiple layers can be used to compute inferences, e.g., given an input, the neural network can compute an inference for that input.

[0022] A neural network computes an inference by processing an input through each of the neural network layers, in particular at least some of the neural network layers each having a respective set of weights, and each layer receives an input and processes the input according to the set of weights for that layer to generate an output.

[0023] The data input to a neural network layer, for example, either the input to the neural network or the output of one or more layers in the neural network, may be referred to as the activation input to that layer.

[0024] In some implementations, the layers of a neural network are arranged as a sequence, while in other implementations, the layers are arranged as a directed graph.

[0025] That is, any particular layer may receive multiple inputs, may generate multiple outputs, or both. The layers of a neural network may also be arranged such that the output of a layer can be fed back as an input to a previous layer.

[0026] Each layer of a neural network model may have a respective type, for example, as defined by the node connections between layers of the neural network.

[0027] As an example, one type of layer in a neural network model may be a fully connected layer. For a fully connected layer, all nodes in this type of layer are connected to all nodes in at least one adjacent layer, i.e., to at least one other layer in the neural network from which it receives node inputs. For example, when a given layer is a fully connected layer and the network layers are arranged in a sequence, each node in the given layer is connected to all nodes in the layer preceding the given layer in the sequence.

[0028] Fully connected layers can be found in different neural networks. For example, a fully recurrent neural network (FRNN) includes only a fully connected layer. The simplest form of a FRNN can be a multi-layer perceptron (MLP), which includes an input layer, an output layer, and one or more hidden layers, each of which is a fully connected layer. Furthermore, a conventional recurrent neural network can also include one or more fully connected layers. In a convolutional neural network (CNN), a fully connected layer is typically located as one of the last few CNN network layers. A fully connected layer in a CNN can receive node outputs from all nodes in the previous convolutional layer and generate node outputs for classifying or labeling input data. For example, a fully connected layer can be the penultimate layer followed by a SoftMax layer in a CNN.

[0029] As another example, another type of layer is an element-wise layer. An element-wise layer performs element-wise operations on input data, such as, for example, element-wise addition, element-wise multiplication, and element-wise nonlinear operations. In some implementations, each node of an element-wise layer is connected to exactly one of the nodes in an adjacent layer. That is, each node in an element-wise layer performs an operation on input received from a corresponding single node in an adjacent layer.

[0030] One exemplary element-wise layer may be a network layer in which each node in the layer includes a respective node operation of a respective activation function (e.g., Relu or Sigmoid function), where each node receives only one node input from an adjacent layer. Another exemplary element-wise layer may be a network layer for node residual connections. More specifically, each node in the layer includes a residual function that receives a single input from a node in an adjacent layer and outputs a residual that is added to the corresponding residual input to the subsequent layer. Optionally, each node in the layer may include a nonlinear function that is applied to the node residual.

[0031] The element-wise node operations may include any suitable operations, such as, for example, element-wise addition, subtraction, and multiplication. For example, an element-wise node output may be an addition of two corresponding elements following the node operation in an element-wise manner.

[0032] Neural networks may also include other layers of different types, such as average pooling layers and convolutional layers, where the nodes between layers are partially connected: each node in a layer is connected to a respective subset of the nodes in the adjacent layers.

[0033] To perform inference computations for a neural network model, i.e., to compute outputs for given inputs, a hardware computing system distributes some or all of the layers of the neural network across multiple computing units (also referred to as "computational tiles") within the hardware computing system, such that each computing unit can perform operations (e.g., tensor computations) for at least a portion of the layer assigned to it.

[0034] For certain types of layers, such as element-wise and fully-connected layers, the computation of the layer is distributed across each of a number of computing units, each of which outputs at least a respective partial result obtained from executing an associated inference computation of the layer and stores the respective partial result in one or more memory addresses in a respective addressable memory unit of the computing unit.

[0035] In some implementations, each computing unit may have a respective addressable memory unit that points to one or more memory addresses in a memory device.

[0036] To generate at least a portion of a layer output from partial results generated by computing units assigned to a layer, the system can generate separate instructions for each computing unit. These separate instructions may include data determining one or more respective memory addresses for storing the respective partial results, and data determining whether the respective partial results for a computing unit are used to generate at least a portion of the input to another layer. More specifically, because one or more computational tiles assigned to a subsequent layer may require copying or combining partial results from one or more computational tiles in a preceding layer, the system needs to generate separate instructions for tiles in the subsequent layer, each of which specifies a respective memory address where the associated partial result is stored for that tile in order to correctly fetch the partial result. As more computational tiles are assigned to a particular network layer, more separate instructions are needed to specify different memory addresses. Thus, the number and size of instructions rises with an increase in the total number of computing units assigned to the layer, and the instructions may exceed memory limits.

[0037] If the entire separate instructions exceed the memory limit of the system, the system will become less efficient in sending all the instructions from the host to the controller. More specifically, the system needs to transfer a part of the instructions from the host and copy a part of the instructions in the instruction memory of the computer included in the system every time. Furthermore, if the neural network modules are considerably large and include an iterative loop for performing the inference calculation, the system cannot perform the inference calculation for these neural network models using a conventional method. The iterative loop for performing the inference calculation of the neural network model exists in many neural network models deployed on neural network accelerator chips, either on-device or in the cloud. Therefore, finding a suitable method for efficiently performing the inference calculation of such neural networks is of great interest. Examples of neural network models whose calculations require an iterative loop include long-short-term memory (LSTM) models or recurrent neural network (RNN) models, such as WaveRNN, OCT, and RNN-T.

[0038] The techniques described herein below can address the challenges discussed above.

[0039] FIG. 1 illustrates an exemplary computing system 100 for performing inference calculations for neural network models.

[0040] System 100 includes a hardware accelerator 101 (also referred to in the following specification as an integrated circuit configuration or chip, but which in some implementations may also be implemented as multiple physically separate integrated circuits) and a host 108 located off-chip and configured to communicate with hardware accelerator 101 via a wired or wireless connection.

[0041] The host 108 may be implemented as one or more computer programs on one or more hardware devices that are located off-chip, i.e., not part of the hardware accelerator 101, and generates instructions for controlling the operation of the hardware accelerator 101.

[0042] As shown in FIG. 1, the hardware accelerator 101 includes a controller 102, a number of tiles 132-148 collectively referred to as computational tile sets 112, 114, and chip memory (not shown).

[0043] The controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in the instruction memory. The instruction memory 106 may store one or more machine-readable instructions executable by the one or more processors of the controller 102. The instruction memory 106 includes a memory of a particular size, such as, for example, 256 kB. The data memory 104 may be any of a variety of data storage media for storing and subsequently accessing various data related to computations within the system 100.

[0044] The controller 102 is capable of receiving instructions and data parameters from a host 108 and is configured to provide instructions to a number of computational tiles over an instruction bus 124 (described below).

[0045] The data parameters relate to the data that defines the neural network model and the input data to the neural network model.

[0046] In general, it is ideal for the controller 102 to receive all instructions from the host 108 at an initial time and store them in the instruction memory 106. In this way, during inference calculations, the controller 102 can avoid fetching parts of instructions during calculations, which may lead to errors introduced by iterative loops in the neural network. In rare but possible cases of neural networks that do not have iterative loops, the controller 102 can be configured to receive a portion of the total instructions to calculate these neural network models at one time and fetch other parts at different times. Optionally, the controller 102 in the system 100 can be configured to further receive more instructions from the host 108 and store them in the instruction memory 106 after at least some of the previously stored instructions have been executed.

[0047] The controller 102 may instruct one or more computational tiles to perform inference calculations for at least a portion of the neural network model.

[0048] Each computational tile is an individual computing unit that cooperates with other computational tiles in the system 100 to accelerate computation across one or more layers of a neural network model. As shown in FIG. 1, the computational tile set 112 includes computational tile 0, computational tile 1, computational tile 2, and computational tile 3 (i.e., tiles 132, 134, 136, and 138), each of which includes addressable memory units 152, 154, 156, and 158, respectively. The computational tile set 114 is composed of computational tiles 142-148, each of which includes addressable memory units 162, 164, 166, and 168, respectively. After performing inference computation, each computational tile stores partial results in a respective memory address within a respective addressable memory unit according to a respective instruction broadcast by the controller 102.

[0049] These instructions may each include a respective header (eg, a bitmap) that indicates which computational tile should execute the instruction.

[0050] The controller 102 can broadcast instructions for each computational tile. More specifically, the controller 102 can broadcast instructions received from the host 108 to each computational tile along the data path 118 using the instruction bus 124, i.e., instructions stored in the instruction memory 106 can be transmitted by the instruction bus 124, which provides communication starting from the controller 102 and via the data path 118 connecting each computational tile in the computational tile set 112, 114 in a ring back to the controller 102. While transmitting one or more instructions on the instruction bus 124, these instructions may be, for example, 32 bits wide, with the first 7 bits including header information indicating the instruction address / destination that will receive and execute these instructions. For example, these first 7 bits may include a data parameter representing a particular node ID. Each computational tile along the data path 118 may sequentially check the header of the instruction to determine whether the request by the host 108 was addressed to the computational tile that checks the header.

[0051] When the node ID in the header does not indicate that the destination is the inspecting tile, the inspecting tile copies the input command packet onto a command bus connecting to the next tile for inspection by the next tile. When the node ID in the header indicates that the destination is the inspecting tile, the inspecting tile performs the operation encoded in the input command packet.

[0052] The instructions may include data in a header (e.g., one or more node IDs) to identify a subset of computational tiles to perform computations within all tiles. The subset of computational tiles is assigned to perform computations within a particular layer of the neural network. Each computational tile of the subset performs inference computations for a respective non-overlapping portion of the assigned layer. For example, each computational tile of the same tile set (computational tiles 132, 134, 136, and 138 in tile set 112) may be assigned to the same layer of the neural network model and perform inference computations associated with that layer.

[0053] The instruction may further specify one or more memory addresses within the corresponding computational tile to output the data. To solve the above-mentioned problem with instructions having a size that exceeds the size limit of the instruction memory 106, the host 108 may generate a sharing instruction for each subset of computational tiles assigned to a particular network layer. The sharing instruction for a particular network layer may specify a memory size and a common memory address for each computational tile in the subset. The sharing instruction for each layer may have data (e.g., one or more node IDs) to identify each computational tile and cause the computational tiles to store partial results in their respective addressable memory units at their respective common memory addresses having the same memory size. In some implementations, the controller 102 may generate multiple instruction packets for each layer based on the respective sharing instruction for that layer, each of which has a node ID for a corresponding tile associated with the layer and a common memory address for the corresponding tile for storing partial results having the same memory size.

[0054] The controller 102 can, according to the instructions, combine the stored partial results generated by each of the associated computational tiles from a common memory address to generate a final output from the neural network for a given input. More specifically, the controller 102 can instruct one or more computational tiles in a subsequent layer to combine the partial results from the previous layer that are stored at a common address in the previous layer. After completing the inference calculations for the neural network, the controller 102 can instruct the computational tiles in the last layer of the neural network to generate a final output. The controller 102 can then provide the final output to the host 108.

[0055] Thus, to cause the hardware accelerator 101 to perform inference computations for a neural network deployed on the hardware accelerator, the host 108 generates and sends instructions to the controller 102. These instructions, once executed by one or more computational tiles in the hardware accelerator 101, can cause the one or more computational tiles to perform respective inference operations for at least a portion of the neural network in accordance with the instructions.

[0056] The instructions issued by the host 108 employing the described techniques may be share instructions for the computational tiles assigned to one or more eligible network layers. The details of using share instructions to allocate memory for each computational tile assigned to a network layer are described in more detail below.

[0057] FIG. 2A illustrates an example of conventional memory allocation for computational tiles 132, 134, 136, and 138 associated with layers of a neural network model.

[0058] The host 108 can generate and provide data to the controller 102 for allocating operations represented by each layer of the neural network to a respective subset of the total available computational tiles. Each tile of a respective subset of tiles can perform at least a non-overlapping portion of the operations for the associated layer. For example, the host 108 can determine and assign operations at a particular layer of the neural network to the computational tile set 112, which includes computational tiles 132, 134, 136, and 138, such that each computational tile performs at least a non-overlapping portion of the inference calculation.

[0059] Each computational tile may include a respective addressable memory unit, and each respective addressable memory unit may include multiple memory addresses. For example, tile 132 includes addressable memory units 152 having different memory addresses 202, such as memory addresses 202a, 202b, 202c, and 202d.

[0060] Conventionally, the controller 102 is configured to broadcast a separate instruction to each tile assigned to the same network layer. Each separate instruction may include a respective memory allocation instruction for the corresponding tile. The memory allocation instruction may identify a respective memory address for a respective addressable memory unit of the corresponding computational tile for storing a respective partial result. Each respective partial result is generated by the corresponding computational tile that executes a respective portion of an inference computation for the respective assigned layer given an input.

[0061] Referring to FIG. 2A as an example, a particular layer is assigned to a first computational tile set 112 including computational tiles 132, 134, 136, and 138. Each of the computational tiles performs at least a portion of the inference computation for that particular layer. These inference computations may include, for example, tensor computations of some of the input activations and corresponding layer weights. After obtaining respective partial results from the corresponding portion of the inference computation, each computational tile stores the respective partial results in a different memory address according to a respective conventional instruction for that tile. For example, as shown in FIG. 2A, the computational tile 132 outputs the respective partial results to memory address 202a of the addressable memory unit 152 according to a first conventional instruction. Similarly, computational tile 134 outputs to memory address 204b according to a second conventional instruction, computational tile 136 outputs to memory address 206d according to a third conventional instruction, and computational tile 138 outputs to memory address 208c according to a fourth conventional instruction.

[0062] Conventional instructions for a given neural network layer, when executed by tiles associated with that layer, cause those tiles to perform at least respective portions of the inference computation for that layer, generate respective partial results, and store those partial results at different memory addresses in respective addressable memory units of the tiles. Doing so may be beneficial because, for example, if a subsequent layer of a given layer is a fully connected layer, and thus each tile associated with that subsequent layer needs the entire output of the given layer, even to compute even a partial output of that subsequent layer, each of the one or more tiles associated with that subsequent fully connected layer can directly copy each of the stored partial results from the different memory addresses associated with the preceding layer to the same memory addresses in the local memory of that tile.

[0063] More specifically, in relation to FIG. 2A and according to conventional instructions, tiles 132, 134, and 136 may be assigned to a first layer, and tile 138 may be assigned to a fully connected layer following the first layer. Tiles 132, 134, and 136 may obtain respective partial results for performing inference calculations in the first layer and store their respective partial results at memory addresses 202a, 204b, and 206d, respectively. The respective partial results collectively include a full set of node outputs from the first layer. To obtain one or more input activations for a subsequent fully connected layer, tile 138 may directly copy the respective partial results from address 202a of tile 132 to addresses 208a, 204b to 208b, and 206 to 208d of tile 138, without allocating other memory addresses of tile 138 to store a full set of node outputs.

[0064] However, because one or more tiles in the first layer store partial results at different local memory addresses and one or more tiles in a subsequent layer (e.g., a fully connected layer) need to copy each of the stored partial results from the different local memory addresses, the host 108 needs to generate respective instructions that specify memory addresses for the tiles in the first layer to store each partial result, and respective instructions that specify respective memory addresses for each of the tiles in the subsequent layer from which to collect the partial results.

[0065] To speed up the execution of inference computations for a particular neural network, systems tend to use as many tiles as possible for parallelization, and the total instruction size therefore increases to different degrees based on the respective parallelization level.

[0066] Furthermore, as more tiles are assigned to a particular layer, or as more nodes are present in a subsequent fully connected layer, more separate instructions are required for the tiles associated with that particular layer to store partial results, and for the tiles associated with subsequent layers to copy or combine partial results from the particular layer. Thus, as discussed above, the size of the instructions to perform inference computations for a neural network increases with the number of computational tiles associated with each layer and the size of the layer (i.e., the number of nodes in the layer).

[0067] As the size of instructions increases, the efficiency of performing inference calculations for neural networks can decrease, especially for large neural networks.

[0068] However, the computing system 100 described below can solve this problem. 1, the computing system 100 may initially obtain data for the host 108. This data may represent a neural network having multiple layers and a layer type for each of the multiple layers, e.g., element-wise layer or fully connected layer.

[0069] The host 108 can determine and select a subset of layers of all layers of the neural network represented by the received data. For simplicity, this subset of layers of the neural network is referred to as eligible layers. Thus, eligible layers are layers that meet the eligibility criteria. Eligible layers include layers that are either element-wise type or fully-connected type.

[0070] The host 108 may generate respective sharing instructions for the computational tiles each associated with a respective eligible tier.

[0071] Each sharing instruction for the tiles associated with the eligible layer may include at least a memory allocation instruction that specifies a common memory address and a predetermined memory size for the associated tiles. Each computational tile may store its respective partial result in a common memory address having a predetermined memory size. The predetermined memory size may be determined by the host 108 or may be set by a user. The memory size may be determined based on the number of values ​​in the partial result that each tile needs to output after performing its assigned portion of the computation in the layer. For example, the memory size for an element-wise layer with at least five node computations, each assigned to its associated tile, is larger than an element-wise layer with only two node computations, each assigned to its associated tile. As another example, a fully connected layer may require a smaller memory size than an element-wise layer, depending on how many values ​​each associated tile needs to output.

[0072] In addition to storing each partial output in a common memory address for the tiles associated with the eligible layer, the host 108 also generates shared instructions for the tiles associated with the fully connected layer to accumulate stored data from the preceding layer, which may be any suitable eligible layer, such as, for example, a fully connected layer or an element-wise layer.

[0073] The host 108 may also generate sharing instructions for the fully-connected layer, further including a sharing accumulation instruction that specifies an additional memory address. The sharing accumulation instruction, when executed by the tiles, causes each tile associated with the fully-connected layer to obtain partial results representing all node results of the preceding layer stored at the common address, and accumulate the obtained partial results at the additional memory address. This is because even if a computational tile associated with the fully-connected layer is assigned to only a portion of the layer operations, the computational tile still needs to obtain the full set of results from the preceding layer.

[0074] The term "accumulation" refers to combining all node outputs from a preceding eligible layer into appropriate input activations for one or more nodes in a subsequent fully connected layer. More specifically, an accumulation instruction issued by the host 108, when executed by each tile associated with each node in the fully connected layer, causes the tile associated with each node in the fully connected layer to collect (e.g., copy and store) the entire set of node outputs from the common address associated with the preceding layer, and each to construct a respective input activation for each node in the fully connected layer based on the collected entire set of node outputs at the additional common address specified in the shared accumulation instruction. The respective input activations (or accumulated partial results) generated according to the accumulation instruction can be used by the respective tiles as inputs for the fully connected layer to generate partial results.

[0075] The results gathered by the tile for a fully connected layer are usually larger in size than the respective partial results stored in the common memory addresses in the respective addressable memories of each computational tile assigned to the preceding layer. Thus, the shared accumulation instruction generated by the system 100 can specify the additional common address having one or more additional common memory addresses each having a different size. For example, the additional common address can be a single additional common address having a larger memory size than the common address for storing the partial results for a layer. As another example, the additional common address can include one or more additional common addresses for a fully connected layer each having a different size. The total number of additional common addresses and the respective size for each additional common address can be determined based on the layer size of the preceding layer or based on the size of the gathered partial results from the preceding layer.

[0076] Each computational tile associated with the fully connected layer performs at least a respective portion of the inference computation of the fully connected layer to process the accumulated input activations according to a shared instruction issued by the host 108, and generates respective partial results for storage in a common memory address according to the shared instruction for the fully connected layer.

[0077] In the case of an element-wise layer, the sharing instruction does not need to specify another memory address besides the common memory address. Each tile associated with the element-wise layer does not need to accumulate the respective partial results associated with the preceding layer according to the sharing instruction issued by the host 108, nor does it need to accumulate the respective partial results associated with the current layer to form the input activations for the next layer. Instead, each node input activation of the element-wise layer is the node output activation from the preceding layer, and each output activation of that layer is the node input activation to the subsequent layer. Thus, the computational tiles of the next layer can directly copy the output activations stored at the common memory address of the corresponding computational tile associated with the preceding layer according to the sharing instruction.

[0078] FIG. 2B illustrates an example of a common memory allocation for computational tiles 132, 134, 136, and 138 associated with a qualified layer of a neural network model.

[0079] As shown in FIG. 2B, the host 108 generates a share instruction for each computational tile 132, 134, 136, and 138 assigned to the eligible layer. The share instruction specifies a common memory address for each computational tile to store the respective partial results at a common address in the respective addressable memory units 152, 154, 156, and 158. The common address may be a memory location on the chip memory that at least a respective portion in each respective addressable memory unit for each computational tile is configured to "point" to when the corresponding tile accesses data stored at the common address for the particular layer. For example, the computational tiles 132, 134, 136, and 138 each generate a respective partial result obtained by executing a respective portion of the inference computation in the associated layer and store the respective partial result at the common address. In some implementations, the respective memory addresses 202a for the tile 132, 204a for the computational tile 134, 206a for the computational tile 136, and 208a for the computational tile 138 all "point" to a common memory address. Because each computational tile stores its partial results at the common address, the stored partial results can be accessed by other computational tiles using shared instructions. Thus, the host 108 can reduce instruction size by generating shared instructions for the computational tiles to access the common memory address to fetch the stored partial results.

[0080] In some implementations, shared instructions for tiles associated with a layer may include data for both storage and access, further reducing the memory bandwidth requirements for receiving instructions from the host 108 and broadcasting those instructions using the controller 102.

[0081] FIG. 3 illustrates an exemplary allocation of additional memory addresses for computational tiles 132, 134, 136, and 138 associated with a fully connected layer of a neural network model.

[0082] As described above, for each of the fully-connected layers, in addition to the respective common addresses for the tiles to store the partial results after performing the respective portions of the computation in the particular fully-connected layer, the host 108 can generate a shared instruction that specifies an additional memory address for the tile associated with the particular fully-connected layer to combine the partial results from the preceding eligible layers. More specifically, each tile associated with the fully-connected layer can copy and combine the entire set of node outputs from the preceding layer in the additional memory address. The accumulated results are used as the respective node activation inputs for performing the inference computation in the fully-connected layer. Even if a tile associated with a fully-connected layer performs only a portion of the layer's operations, the tile is instructed to obtain all node outputs stored in the common addresses from the preceding layers and combine the node outputs in the layer's additional memory address.

[0083] In some implementations, the host 108 can identify each node output in the partial results stored in a common memory address for the preceding layer (e.g., by a number or ID associated with the node output), and consequently, tiles in the fully connected layer can copy each of the identified node outputs from the preceding layer and non-iteratively combine them in additional memory addresses.

[0084] Assuming that tiles 132, 134, 136, and 138 are associated with a fully connected layer, the layer preceding the fully connected layer is a qualifying layer. As shown in FIG. 3, each tile allocates additional memory addresses to its addressable memory units according to the shared instructions for those tiles. For example, memory unit 152 of tile 132 is configured to include additional memory address 302, memory unit 154 of tile 134 includes additional memory address 304, memory unit 156 of tile 136 includes additional memory address 306, and memory unit 158 ​​of tile 138 includes additional memory address 308. Each additional memory address of the respective memory units is associated with an additional common address on the chip memory.

[0085] According to the accumulation instructions of the shared instructions for the tiles associated with the fully connected layer, each tile obtains the entire set of node outputs from the partial results stored at the common address for the eligible previous layers and accumulates those node outputs in a respective additional memory address. For example, computational tile 132 copies the partial results from the previous eligible layer and combines them at memory address 302 of memory unit 152, computational tile 134 copies the partial results from the previous eligible layer and combines them at memory address 304 of memory unit 154, computational tile 136 copies the partial results from the previous eligible layer and combines them at memory address 306 of memory unit 156, and computational tile 138 copies the partial results from the previous eligible layer and combines them at memory address 308 of memory unit 158.

[0086] As a more specific example, FIG. 4 illustrates an exemplary detailed architecture for a computational tile 132 in system 100.

[0087] 4, an exemplary computational tile 132 may include narrow memory 425 units and wide memory units 412a, 412b, 412c, 412d (collectively, 412). The designations narrow and wide refer generally to the size of the width (in bits / bytes) of the memory units of the narrow memory 425 and the wide memory 412. In some implementations, the narrow memory 425 includes memory units each having a size or width of less than 16 bits, and the wide memory 412 includes memory units each having a size or width of less than 32 bits.

[0088] Generally, a computational tile 132 assigned to an eligible layer receives data, including input activations and parameters, from the host along data path 118. The computational tile 132 writes input activations to narrow memory 425 and parameters to wide memory 412 according to shared instructions for the eligible layer. In some implementations, narrow memory 524 may include a memory arbiter that is typically used to determine for each memory cycle which control device (e.g., TensorOp control or DMAOp control) is allowed to access narrow memory 210.

[0089] If the eligible layer is fully connected and the layer preceding the fully connected layer is also eligible, the computation tile 132 accumulates the entire set of node outputs from the partial results obtained for the preceding layer at an additional memory address indicated by the sharing instruction.

[0090] More specifically, the computational tiles 132 perform respective portions of the inference computation associated with a particular layer of the neural network through MAC operators and summation registers. The computational tiles 132 provide input activations for the layer along an input bus from the narrow memory 425 to one or more MAC operators. The computational tiles 132 also provide parameters from the wide memory unit 412 to one or more MAC operators. The one or more MAC operators and summation registers perform dot product computations and summations with the input activations and parameters.

[0091] The computational tiles 132 provide partial results generated from the MAC operator and the summing register along an output bus to a nonlinear unit. The nonlinear unit is configured to apply a nonlinear function, such as a Sigmoid or ReLU function, to the partial results to generate at least a portion of the output activations for a subsequent layer. The computational tiles 132 store the output activations at common addresses allocated in the narrow memory unit 425 according to a shared instruction for the layer. Other computational tiles associated with a subsequent layer can fetch the stored output activations from the narrow memory unit of the computational tiles 132. In some implementations, the shared instruction for a layer may include data that determines whether stored partial results from one or more computational tiles of the layer are used to generate input activations for a computational tile associated with a subsequent layer, and if so, the stored partial results will belong to the final result (e.g., input activations for a final output for a subsequent layer or neural network).

[0092] 5 illustrates an exemplary process 500 for memory allocation of computational tiles in system 100 for performing inference computations for neural network models. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, a hardware computational system such as system 100 of FIG. 1 that is appropriately programmed can perform process 500. As a more specific example, host 108 of system 100 of FIG. 1 can perform process 500 when appropriately programmed.

[0093] The host first obtains (502) data indicative of the neural network. The data indicative of the neural network includes data identifying the type of neural network (e.g., convolutional neural network or recurrent neural network), the total number of layers in the neural network (e.g., 10 hidden layers between the input layer and the output layer), the number of nodes in each layer (e.g., 10, 20, or 50 nodes, each of which includes at least a network operation), data representing the sequence of all layers, and data representing node connections between layers (e.g., whether a layer is fully connected, partially connected, or element-wise connected with one of the adjacent layers). In some implementations, this data may include parameters that further define the trained neural network, such as the learned set of weights for each node associated with each layer, data format requirements for the input layer or output layer, or both (e.g., data representing requirements for the size of input data for an input layer in the trained neural network, or data representing an output format, such as the number of output categories, for the trained neural network). In some implementations, the host can determine whether the data represents an untrained neural network or a trained neural network that is missing weights, and in response, the host can either train the neural network with the training examples or prompt a notice on a user interface indicating that the input neural network is untrained and request data representing the trained neural network or data representing the missing weights.

[0094] The controller 102 receives data representing the neural network from the host 108 and stores the data in a data memory 104 of the controller 102. In some implementations, the controller 102 can receive all of the data at once, or portions of the data at a time until all of the data is received.

[0095] The controller 102 also receives instructions from the host 108 and stores the received instructions in the instruction memory 106. Similarly, the controller can retrieve partial instructions from the host 108 at one of multiple times. The controller can also send data back to the host 108, including partial or final results after performing inference calculations for the neural network model.

[0096] After receiving the data, the host determines, for each layer of the neural network model, a respective layer type according to the received data, for example, the host determines whether the network layer is a fully connected layer, an element-wise layer, or any other type of layer according to the received data.

[0097] The host then selects (504) a (suitable) subset of the layers of the neural network based on the acquired data. Specifically, the host selects the subset of the layers based at least in part on the type of layers. The selected subset of the layers is also referred to herein as eligible layers. For example, the host may select all fully-connected layers from all layers to form one subset. As another example, the host may select both fully-connected layers and element-wise layers from all layers to form another subset. In some implementations, the host may select a first subset that includes all fully-connected layers of the neural network and a second subset that includes all element-wise layers of the neural network.

[0098] For each layer in the subset of layers, the host assigns (506) an (appropriate) subset of the computing units to at least partially perform the inference computation associated with that layer. For example, the node operations for the inference computation in a fully connected layer are distributed among 4 computation tiles. As another example, the node operations for the inference computation in an element-wise layer are distributed among 10 computation tiles.

[0099] Each computing unit includes a respective addressable memory unit, such that each computing unit can store partial results in the respective addressable memory unit at a given memory address. For example, with reference to FIG. 2B, computation 132 includes addressable memory unit 152 having four different memory addresses 202a-d, each "pointing" to a respective physical address on the chip memory. Each memory address 202a-d can be redirected to another physical memory address by instructions issued by the host.

[0100] Upon issuing the sharing instruction for the computational tiles assigned to the eligible layer, the host determines a memory size and a common memory address for each computing unit associated with the eligible layer (508). The determined memory size may be based at least in part on the type of network layer. For example, the determined memory size is larger for a fully connected layer than for an element-wise layer.

[0101] The memory size may also be a fixed size that is pre-determined for a given eligible network layer based on the hardware architecture, layer characteristics, and computational requirements. In some implementations, the memory size may instead be pre-set by the user through a user interface. The memory size may be 5Kb, 10Kb, or 1Mb, to name just a few examples.

[0102] The host 108 generates shared instructions including memory allocation instructions that, when executed by each computing unit associated with the layer, cause the computing units to store results of performing inference computations associated with the layer in a common memory address having a memory size in the addressable memory of the computing unit (510). For example, with reference to FIG. 2B, computation tiles 132, 134, 136, and 138 are associated with the element-wise layer, and each computation tile stores the partial results of the inference computation assigned to it in a respective memory address (e.g., 202a, 204a, 206a, and 208a) within a respective addressable memory unit (e.g., 152, 154, 156, and 158). Each memory address 202a, 204a, 206a, and 208a can be considered a common memory address. However, to be precise, each of the memory addresses 202a, 204a, 206a, and 208a has an address identifier that initially "points" to a common memory address of the chip, so that each data store / load process to the memory addresses 202a, 204a, 206a, and 208a is in effect a data store / load process to the common memory address. Alternatively, each of the memory addresses 202a, 204a, 206a, and 208a may initially "point" to a respective memory address, but may be changed to "point" to the common memory address due to a memory allocation instruction in the shared instruction.

[0103] The memory allocation instructions may further include data identifying whether the memory allocation instructions apply to one or more computing units assigned to the respective eligible tiers. By doing so, the controller 102 can selectively control the memory storage of each computational tile as needed according to the memory allocation instructions. For example, referring again to FIG. 2B, the data may specify that only computational tiles 132, 134, and 136 can store partial results in a common memory address having a predetermined memory size.

[0104] The data for identifying the computing unit may be binary data, and includes data representing the node ID of the computing tile and a status flag representing whether the memory allocation instruction is applicable to the computing tile, for example, 0 represents that the memory allocation instruction is not applicable to the computing unit having that node ID, and 1 represents that the instruction is applicable to the computing unit.

[0105] The memory allocation instructions may also include data tracking the partial results stored in the common memory address. The host may issue a share instruction that, when executed by the corresponding tile, causes the controller 102 to track the origin common memory address where each result is stored for a layer to form a final result. More specifically, when one or more computational tiles accumulate partial results to provide one or more layer outputs or a final output for the network, the one or more computational tiles can correctly identify which partial results to accumulate and where to accumulate the partial results. For example, referring again to FIG. 2B, the computational tile 132 stores the partial results at address 202a that "points" to the common memory address (or, equivalently, stores the partial results in the common memory address). After each computational tile executes its respective inference computation for the layer, the share instruction issued by the host can identify whether the partial results are to be used to generate the layer output. If so, the shared instruction issued by the host may further identify how the partial results should be processed in the arithmetic sequence.

[0106] As described above, the host can determine the type of the network layer. In response to determining that the layer is a fully-connected layer, and for each computing unit associated with the fully-connected layer, the host generates a sharing instruction, which further includes an accumulation instruction. The accumulation instruction can specify an additional memory address (i.e., a memory address different from the common memory address) used to accumulate the full set of node outputs from the preceding eligible layer. More specifically, when executed by each of the computing units assigned to the fully-connected layer, the accumulation instruction can cause each of the computational tiles to accumulate the full set of node outputs from the partial results stored at the common address associated with the preceding eligible layer at the additional common address.

[0107] The additional memory address may be an address accessible to the computing units assigned to the fully connected layer. For example, with reference to FIG. 3, computational tile 132 and computational tile 134 are assigned to one fully connected layer, and an instruction issued by the host, when executed, causes controller 102 to allocate memory address 302 to addressable memory unit 312 and memory address 304 to addressable memory unit 314. Both memory address 302 and memory address 304 point to an additional common memory address.

[0108] The computational tiles 132 and 134 can accumulate one or more related partial results obtained from a previous layer and store them in an additional common memory address. The computational tiles 132 and 134 can access data stored in the additional common memory address with each other.

[0109] In some implementations, the accumulation instruction further includes data that determines whether one or more of the stored partial results from the preceding layer are accumulated to an additional memory address by the computing unit associated with the fully connected layer. This may be useful for parallel calculation, especially when tiles redundantly calculate node outputs and store their respective partial results in a common address. Therefore, it is required that tiles in the system correctly fetch associated node outputs from the common address.

[0110] 1, assume that computational tile 148 is associated with a fully connected layer and is assigned to perform an operation on the first node of the layer. Computational tile 132 and computational tile 134 are each associated with an eligible preceding layer of the fully connected layer, where computational tile 132 is assigned to perform an operation on the first half of the nodes of the preceding layer, and computational tile 134 is assigned to perform an operation on the second half of the nodes of the preceding layer. If computational tile 148 is associated with a fully connected layer, the accumulation instruction includes data that determines that all relevant partial results stored in computational tile 132 and computational tile 134 should be accumulated in an additional memory address according to the sharing instruction (more specifically, the accumulation instruction) for the fully connected layer.

[0111] The neural network system implemented by the system 100 of FIG. 1 can be part of a computer vision system. For example, in a computer vision system for an autonomous vehicle or a robotic system. In real-world vision applications, unfamiliar objects are almost ubiquitous due to the so-called "long tail" of objects occurring in real scenes. As another example, the system can be used as part of a photo organization software where new categories may need to be created on the fly. In addition to object recognition, the system can perform other computer vision tasks, such as object detection and segmentation. In other domains, such as images and sounds, there may be rare and novel objects, and the system can be used in relation to such other domains as well.

[0112] For example, neural network systems may be used in language modeling systems, image / video processing systems, or action selection systems. For example, tasks may include classification tasks, such as image processing tasks, speech recognition tasks, natural language processing tasks, word recognition tasks, or optical character recognition tasks. Additionally, tasks may include reinforcement learning tasks, in which an agent interacts with one or more real or simulated environments to achieve one or more goals.

[0113] In the case of language modeling tasks or translation of text from a source language to a target language using neural networks, the system may be configured to receive an input sequence of source embeddings representing a source sequence of words in a source natural language and to generate an output sequence of target embeddings representing a target sequence of words that is a translation of the source sequence into the target natural language. More generally, the system can be applied to other sequence transfer applications in which a source sequence is mapped to a target sequence.

[0114] The input data may include, for example, one or more of image data, motion image / video data, motion data, voice data, audio data, electronic documents, data representing a state of an environment, and / or data representing an action. For example, image data may include color or monochrome pixel value data. Such image data may be captured from an image sensor, such as a camera or a LIDAR sensor. Audio data may include data defining an audio waveform, such as a series of values ​​in the time domain and / or frequency domain that define a waveform, which may represent a voice in a natural language. Electronic document data may include text data representing words in a natural language. Data representing a state of an environment may include any kind of sensor data, including, for example, data characterizing a state of a robot or vehicle, such as pose data and / or position / velocity / acceleration data, or data characterizing a state of an industrial plant or data center, such as sensed electronic signals, such as sensed current signals and / or temperature signals. Data representing an action may include, for example, position, velocity, acceleration, and / or torque control data, or data for controlling the operation of one or more items of equipment in an industrial plant or data center. These data may generally relate to a real or, for example, a simulated virtual environment.

[0115] The output data may similarly include any kind of data. For example, in a classification system, the output data may include classification labels for the input data items. In a regression task, the output data may predict the value of a continuous variable, such as a control variable for controlling an electronic or electromechanical system, such as a robot, a vehicle, a data center, or a plant. In another example of a regression task operating on image or audio data, the output data may define one or more locations in the data, such as, for example, the location of an object, the location of one or more corners of an object's bounding box, or the time location of a sound feature in an audio waveform. In a reinforcement learning system, the output data may include data representing an action, for example, as described above, performed by an agent operating in an environment, such as a mechanical agent, such as a robot or a vehicle.

[0116] Data representing an action may include, for example, data defining an action value (Q value) for that action, or data that parameterizes a probability distribution sampled to determine that action, or data that directly defines actions, for example, in a continuous action space. Thus, in a reinforcement learning system, a neural network system may directly parameterize a probability distribution for an action selection policy, or may learn to estimate the value of an action value function (Q value). In the latter case, multiple memories and respective output networks may share a common embedding network to provide a Q value for each available action.

[0117] Neural networks can be configured to accept any type of digital data input and generate any type of score, classification, or regression output based on that input.

[0118] For example, if the inputs to a neural network are images or features extracted from images, the output generated by the neural network for a given image may be a score for each of a set of object categories, each score representing the estimated likelihood that the image contains an image of an object belonging to that category.

[0119] As another example, if the input to a neural network is an Internet resource (e.g., a web page), a document, a portion of a document or features extracted from an Internet resource, or a portion of a document, the output generated by the neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, each score representing an estimated likelihood that the Internet resource, document, or portion of a document is related to that topic.

[0120] As another example, if the input to a neural network is impression context features for a particular advertisement, the output generated by the neural network may be a score representing the estimated likelihood that a particular advertisement will be clicked on.

[0121] As another example, if the inputs to the neural network are features of a personalized recommendation for the user, such as features characterizing the context for the recommendation or features characterizing previous actions taken by the user, the output generated by the neural network may be a score for each of a set of content items, each score representing an estimated likelihood that the user will respond favorably to the content item being recommended.

[0122] As another example, if the input to a neural network is a sequence of text in one language, the output generated by the neural network may be a score for each of a set of pieces of text in another language, each score representing the estimated likelihood that the piece of text in the other language is an appropriate translation of the input text into the other language.

[0123] As another example, if the input to a neural network is a sequence representing a spoken utterance, the output generated by the neural network may be a score for each of a set of pieces of text, each score representing the estimated likelihood that that piece of text is an accurate transcription for that utterance.

[0124] The subject embodiments and functional operations described herein can be implemented as digital electronic circuitry, as tangibly embodied computer software or firmware, as computer hardware including the structures disclosed herein and their structural equivalents, or as a combination of one or more of them. The subject embodiments described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0125] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. These processes and logic flows may also be performed by, and devices may be implemented as, special purpose logic circuitry, such as, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a general purpose graphics processing unit (GPGPU).

[0126] A computer suitable for executing a computer program may include or be based on, by way of example, a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive, transmit, or both transmit and receive data to and from one or more of these mass storage devices. However, it is not necessary for a computer to have such devices.

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including exemplary semiconductor memory devices, such as, for example, EPROM, EEPROM, and flash memory devices, as well as magnetic disks, such as, for example, internal hard disks or removable disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0128] Although the specification contains many specific implementation details, these should not be construed as limitations of any invention or claims, but as descriptions of features that may be specific to a particular embodiment of a particular invention. Also, certain features described in the specification in the context of separate embodiments may also be implemented in a combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0129] Similarly, although the figures show operations in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown, or in sequential order, or as requiring all of the operations shown, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0130] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. Obtaining data indicative of a neural network having a plurality of layers; for each layer in the subset of layers, allocating a subset of a plurality of computing units, each of the computing units including a respective addressable memory unit, to at least partially perform an inference computation associated with the layer; determining a memory size and a common memory address for the respective addressable memory units of each computing unit in the subset of the plurality of computing units assigned to the tier; generating shared instructions including memory allocation instructions; The memory allocation instruction, when executed by each of the subset of the plurality of computing units, causes the computing units to store results of performing inference computations associated with the layer at the determined common memory addresses having the determined memory size in the addressable memory units of the computing units.

2. The subset of the plurality of strata is selected from the plurality of strata, the selection comprising: The method of claim 1 , further comprising determining a layer type for each of the plurality of layers of the neural network based on the obtained data indicative of the neural network.

3. The method of claim 2 , wherein the selection is based at least in part on the determined layer type.

4. In response to determining that the layer type of the layer is a fully connected layer, determining an additional memory address that is different from the common memory address of the computing units, the sharing instructions further comprising an accumulation instruction; 3. The method of claim 2, wherein the accumulation instruction, when executed by each of the subset of the plurality of computing units for the layer, causes the computing unit to accumulate one or more results associated with another layer preceding the layer and store the accumulated results at a determined additional memory address in the addressable memory unit of the computing unit.

5. Determining the memory size for the respective addressable memory units of each computing unit in the subset of the plurality of computing units assigned to the tier includes:

3. The method of claim 2, further comprising determining the memory size for the respective addressable memory units of each computing unit in the subset of the plurality of computing units allocated to the tier based at least in part on the determined tier type of the respective tier.

6. for each layer of the plurality of layers other than the subset of the plurality of layers, allocating a second subset of a plurality of computing units, each computing unit including a respective addressable memory unit, to at least partially perform inference computations associated with the layer; and 2. The method of claim 1, further comprising: generating one or more memory allocation instructions for a corresponding computing unit in the second subset of the plurality of computing units, respectively.

7. The memory allocation instruction: The method of claim 1 , further comprising data identifying one or more computing units of the subset of the plurality of computing units to which the memory allocation instruction is to be applied.

8. The method of claim 7 , wherein the data identifying the one or more computing units is binary instruction data.

9. 2. The method of claim 1, wherein the memory allocation instructions further include, for each computing unit of the subset of the plurality of computing units, data tracking the results generated by the computing units that are stored in the respective common memory addresses.

10. 5. The method of claim 4, wherein the accumulation instruction further includes data specifying, for each computing unit of the subset of the plurality of computing units for the layer, whether each of the results associated with another layer preceding the layer is to be accumulated at the additional memory address in the addressable memory unit of the computing unit of the layer.

11. The method of claim 1 , further comprising providing the shared instruction to the multiple computing units.

12. 1. A method comprising: providing a set of instructions to a system including a plurality of computing units, each computing unit including a respective addressable memory, for performing inference calculations on a plurality of layers of a neural network; The set of instructions comprises: a first memory allocation instruction associated with a first layer in the plurality of layers of the neural network, the first memory allocation instruction identifying a first memory address of the respective addressable memory and a first subset of the plurality of computing units; The set of instructions further comprises: second memory allocation instructions associated with a second layer in the plurality of layers of the neural network, the second memory allocation instructions identifying a second memory address of the respective addressable memory and a second subset of the plurality of computing units; the second memory address is different from the first memory address, the second subset is different from the first subset, The set of instructions may include instructions for the system: causing each computing unit of the first subset to output a result of an inference computation associated with the first tier in the plurality of tiers to a respective memory address of an addressable memory of the computing unit based on the first memory address; and causing each computing unit of the second subset to output a result of an inference computation associated with the second layer in the plurality of layers to a respective memory address of the addressable memory of the computing unit based on the second memory address.

13. the first subset of the plurality of computing units corresponds to a subset of the plurality of computing units across which inference computations associated with the first tier in the plurality of tiers are distributed; 13. The method of claim 12, wherein the second subset of the plurality of computing units corresponds to a subset of the plurality of computing units across which inference computations associated with the second tier in the plurality of tiers are distributed.

14. the first memory allocation instruction further specifies a first memory size, and the second memory allocation instruction further specifies a second memory size; The set of instructions may further include causing the system to: for each computing unit of the first subset, allocating the first memory size to a respective memory address in the addressable memory of the respective computing unit based on the first memory address; and for each computing unit of the second subset, allocating the second memory size to the respective memory addresses in the addressable memory of the respective computing unit based on the second memory addresses.

15. The method of claim 14 , wherein the first memory size is greater than the second memory size.

16. The method of claim 12 , wherein the first layer in the plurality of layers comprises a fully connected layer and the second layer in the plurality of layers comprises an element-wise layer.

17. 13. The method of claim 12, wherein the set of instructions further comprises one or more memory allocation instructions associated with each of one or more tiers of the plurality of tiers that are different from the first tier and the second tier.

18. The method of claim 16 , wherein the set of instructions further comprises a first accumulation instruction associated with the first tier.

19. the first accumulation instruction associated with the first tier, when executed by each computing unit of the first subset, causes the system to allocate additional memory addresses associated with the respective computing unit to the first subset; 20. The method of claim 18, wherein the additional memory address is different from the first memory address.

20. 20. The method of claim 19, wherein the first accumulation instructions further include data for determining, for each computing unit of the first subset, whether each of the respective results of a speculation calculation associated with a preceding layer of the first layer is to be accumulated at a respective memory address of the computing unit based on the additional memory address.

21. In response to determining that a result of an inference calculation associated with the preceding layer of the first layer is to be accumulated, the first accumulation instruction associated with the first layer further comprises:

21. The method of claim 20, further comprising accumulating the results of inference calculations associated with the preceding layer in respective memory addresses of corresponding computing units based on the additional memory addresses.

Images