Efficiently allocating memory on neural network compute tiles

Abstract

To provide a method of generating instructions that cause computing units of a hardware computing system to efficiently allocate a memory while processing inputs to certain types of neural network layers.SOLUTION: A method includes: obtaining data indicating a neural network comprising a plurality of layers; to each layer in a subset of the plurality of layers, assigning a subset of a plurality of computing units to at least partially perform inference computations associated with the layer; determining a memory size and a common memory address for the respective addressable memory units of each computing unit assigned to the layer; and generating a shared instruction comprising a memory allocation instruction.SELECTED DRAWING: Figure 5

Description

Background Art

[0001] Background This specification generally relates to neural networks. In particular, this specification relates to processing inputs to neural networks on a hardware accelerator having a plurality of compute tiles. It relates to the processing of inputs to the network.

[0002] A neural network is a machine learning model that uses one or more layers of non-linear units to predict an output for a received input.

[0003] Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of the respective set of network parameters.

[0004] Network parameters for a neural network are values that affect the operations performed by that neural network and are also values that are adjusted as part of training. For example, network parameters can include the values of weight matrices and, in some cases, the values of bias vectors for the network layers of a neural network.

Summary of the Invention

[0005] Summary This specification generally describes techniques for generating instructions that cause a computing unit of a hardware computing system to efficiently allocate memory while processing inputs to a particular type of neural network layer.

[0006] Generally, one innovative aspect of the subject matter described in this specification is to obtain (i.e., define) data representing a neural network including a plurality of layers, select a subset of the plurality of layers based on the obtained data from among the plurality of layers of the neural network, for each layer in the subset of the plurality of layers, allocate a subset of a plurality of computing units from among the plurality of computing units each including an addressable memory unit, to at least partially execute an inference calculation associated with the layer, determine a memory size and a common memory address for the addressable memory unit of each computing unit in the subset of the plurality of computing units allocated to the layer, and generate a shared instruction including a memory allocation instruction to cause the computing unit to store, at the determined common memory address having the determined memory size in the addressable memory of the computing unit, a result of executing the inference calculation associated with the layer, when executed by each of the subset of the plurality of computing units. This can be embodied as a method including these operations.

[0007] The operations in a method of embodying this innovative aspect of the subject matter further include providing the shared instruction to the plurality of computing units. This method can be executed by a computer system such as a hardware accelerator including a plurality of computing units. The computer system may further include a controller, and the controller is for controlling the plurality of computing units to execute parallel processing based on instructions transmitted to the plurality of computing units by the controller. It is for controlling the plurality of computing units.

[0008] Another innovative aspect of the subject matter described in this specification can be embodied as a method that includes providing a set of instructions for performing inference calculations on multiple layers of a neural network to a system that includes multiple computing units. Each computing unit includes its own addressable memory. This set of instructions is associated with a first layer in multiple layers of the neural network and includes a first memory allocation instruction that identifies a first memory address of each addressable memory and a first subset of the multiple computing units, and is associated with a second layer in multiple layers of the neural network, and includes a second memory allocation instruction that identifies a second memory address of each addressable memory and a second subset of the multiple computing units. The second memory address is different from the first memory address, and the second subset is different from the first subset.

[0009] This set of instructions causes the system to output, for each computing unit in the first subset, the result of the inference calculation associated with the first layer in the multiple layers, based on the first memory address, to each memory address of the addressable memory of the computing unit, and for each computing unit in the second subset, the result of the inference calculation associated with the second layer in the multiple layers, based on the second memory address, to each memory address of the addressable memory of the computing unit.

[0010] Other embodiments of this aspect include a corresponding computer system, an apparatus, and a computer program recorded on one or more computer storage devices, each configured to perform the operations of the method.

[0011] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following effects.

[0012] having a plurality of computing units (e.g., compute tiles) A computing system that implements a neural network model in a hardware accelerator having a plurality of computing units (e.g., compute tiles) can efficiently perform inference calculations for the neural network model by issuing a single shared instruction to the plurality of computing units associated with a layer of the neural network model using the described techniques. The shared instruction specifies a shared memory address that the computing units store, fetch, and accumulate partial results computed by the computing units associated with that layer. Instead of using the described techniques, in some situations where the computing units are assigned a plurality of different memory addresses to store the results from the inference calculations in a particular layer, the system would need to issue a plurality of separate instructions to each of the plurality of computing units. However, a system that employs the described techniques can issue one shared instruction to send to each of the computing units in a particular layer, thereby reducing the total number of required instructions by at least a factor N, where N is the total number of computing units used to perform the inference calculations in a particular network layer.

[0013] This system can reduce the bandwidth of the overall instructions required and reduce memory usage by issuing a shared instruction. Thus, in situations where the system (which may be a hardware accelerator) includes a controller that controls it and the instruction bandwidth and memory of the instruction memory of the controller are constrained Instead, this system can directly load the shared instructions into the controller's instruction memory, rather than fetching parts of separate instructions from the host to the controller at each of multiple points in time. In this way, the system can reduce, and even eliminate, the time wasted in transmitting multiple parts of separate instructions. In some implementations, the techniques described can reduce the overall required instruction bandwidth by more than 50% compared to conventional methods.

[0014] Furthermore, a system employing the techniques described can perform inference calculations for relatively large neural network models. Previously, it was difficult, if not impossible, to compute large neural networks on certain hardware accelerators. The number and size of instructions required to allocate memory for inference calculations of large neural networks increase with the number of neural networks and network layers, so the overall instruction size can become extremely significant. When the instruction size is large, the system needs to send multiple parts of separate instructions from the host to the controller multiple times, which is inefficient and can increase the latency overhead. Furthermore, in the case of large neural networks with iterative loops, correctly fetching multiple parts of multiple, separate instructions from the host to the controller is error-prone for the system.

[0015] However, when the system issues the shared instructions described herein, it is possible to significantly reduce the instruction size. Thus, this system can avoid fetching instructions from the host multiple times to reduce the latency (e.g., reduce the time spent loading the shared instructions), and can directly load all shared instructions to avoid potential errors when fetching instructions for computing neural networks using iterative loops.

[0016] Furthermore, the described technology can reduce memory usage. For example, a system using shared instructions can reduce the memory allocation for storing activation inputs, weight inputs, and hyperparameters for a specific machine learning model in a computing unit (e.g., GPU or TPU), which can, for example, result in improving the computational efficiency of executing inference calculations for a specific machine learning model.

[0017] Details of one or more embodiments of the subject matter of this specification are given in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0019] Detailed Description This specification describes techniques for improving the efficiency of inference calculations in a hardware computing system that includes a plurality of computing units when processing inputs in a neural network model.

[0020] Each computing unit of the hardware computing system is self - contained and can independently perform the calculations required by at least a part of a given layer of a multi - layer neural network. The described techniques can reduce the required instruction memory size and improve the computing efficiency in the hardware computing system when performing inference calculations for deep or large neural network models.

[0021] A trained neural network model having multiple layers can be used to compute inferences. For example, when a certain input is given, the neural network can compute an inference for that input.

[0022] The neural network computes inferences by processing the input through each of the neural network layers. In particular, at least some of the neural network layers each have a respective set of weights. Each layer receives an input, processes the input according to the set of weights for that layer, and generates an output.

[0023] The data input to a neural network layer can be, for example, either an input to the neural network or an output of one or more layers in the neural network, and can be referred to as an activation input to that layer.

[0024] In some implementations, the layers of the neural network are arranged as a sequence. In other implementations, these layers are arranged as a directed graph.

[0025] That is, any particular layer may receive multiple inputs, may generate multiple outputs, or both. The layers of a neural network can also be arranged such that the output of one layer can be fed back as input to a previous layer.

[0026] Each layer of a neural network model can have a respective type, for example, as defined by the node connections between the layers of that neural network.

[0027] As an example, one type of layer in a neural network model can be a fully connected layer. In the case of a fully connected layer, all nodes in this type of layer are connected to all nodes in at least one adjacent layer, that is, the fully connected layer is connected to at least one other layer in the neural network from which it receives node inputs. For example, when a given layer is a fully connected layer and the network layers are arranged as a sequence, each node in the given layer is connected to all nodes in the layer preceding the given layer in the sequence.

[0028] Fully connected layers can be found in different neural networks. For example, a fully recurrent neural network (FRNN) contains only fully connected layers. The simplest form of an FRNN can be a multi-layer perceptron (MLP), which includes an input layer, an output layer, and one or more hidden layers, each of which is a fully connected layer. Further conventional recurrent neural networks can also include one or more fully connected layers. In a convolutional neural network (CNN), the fully connected layer is typically arranged as one of the last few CNN network layers. The fully connected layer in a CNN can receive node outputs from all nodes in the previous convolutional layer and generate node outputs for classifying or labeling the input data. For example, the fully connected layer can be the second-to-last layer followed by the SoftMax layer of the CNN.

[0029] As another example, another type of layer is an element-wise layer. An element-wise layer performs operations element-wise on the input data, such as element-wise addition, element-wise multiplication, and element-wise non-linear operations. In some implementations, each node in the element-wise layer is connected to only one of the nodes in the adjacent layer. That is, each node in the element-wise layer executes an operation on the input received from the corresponding single node in the adjacent layer.

[0030] One exemplary element-wise layer can be a network layer in which each node includes a respective node operation of a respective activation function (e.g., Relu or Sigmoid function) in that layer. In this case, each node receives only one node input from the adjacent layer. Another exemplary element-wise layer can be a network layer for node residual connections. More specifically, each node in the layer includes a residual function that receives a single input from a node in the adjacent layer and outputs a residual that is added to the corresponding residual input to the subsequent layer. Optionally, each node in the layer may also include a non-linear function applied to the node residual.

[0031] Element-wise node operations can include any suitable operations, such as element-wise addition, subtraction, and multiplication. For example, an element-wise node output can be the addition of two corresponding elements according to the node operation in an element-wise manner.

[0032] The neural network can also include other layers having different types, such as an average pooling layer and a convolutional layer, and the nodes between the layers are partially connected. Each node in the layer is connected to a respective subset of the nodes in the adjacent layer.

[0033] To perform inference calculations for a neural network model, i.e., to compute an output for a given input, a hardware computing system distributes some or all of the layers of the neural network across a plurality of computing units (also referred to as "computing tiles") within the hardware computing system such that each computing unit can execute operations (e.g., tensor calculations) for at least a portion of the assigned layer.

[0034] For certain types of layers, such as element-wise layers and fully connected layers, the layer calculations are distributed across each of the plurality of computing units. Each computing unit outputs at least its respective partial result obtained from the execution of the associated inference calculations of the layer and stores the respective partial results at one or more memory addresses in the respective addressable memory units of the computing unit.

[0035] In some implementations, each computing unit may have a respective addressable memory unit that is directed to one or more memory addresses in a memory device.

[0036] To generate at least a portion of the layer output from the partial results generated by the computing units assigned to the layer, the system can generate separate instructions for each computing unit. These separate instructions can include data that determines one or more respective memory addresses for storing the respective partial results, and data that determines whether the respective partial results for a computing unit are used to generate at least a portion of the input to another layer. More specifically, since one or more compute tiles assigned to a subsequent layer may need to copy or combine partial results from one or more compute tiles in a preceding layer, the system needs to generate separate instructions for the tiles in the subsequent layer, and each of these separate instructions identifies the respective memory address where the relevant partial result is stored for that tile in order to correctly fetch the partial result. As more compute tiles are assigned to a particular network layer, more separate instructions are needed to identify different memory addresses. Thus, the number and size of the instructions increase with an increase in the total number of computing units assigned to the layer, and the instructions may exceed the memory limit.

[0037] If all the individual instructions exceed the limits of the system's memory, the system will reduce the efficiency when sending all instructions from the host to the controller. More specifically, the system needs to transfer a part of the instructions from the host each time and copy a part of the instructions in the instruction memory of the computer included in the system. Further, when the neural network module is quite large and includes an iterative loop for performing inference calculations, the system cannot execute the inference calculations for these neural network models using conventional methods. The iterative loop for performing the inference calculations of the neural network model exists either on-device or in the cloud in many neural network models deployed on the neural network accelerator chip. Therefore, finding a method suitable for efficiently executing the inference calculations of such neural networks is a major concern. Examples of neural network models whose calculations require an iterative loop include long short-term memory (LSTM) models or recurrent neural network (RNN) models such as WaveRNN, OCT, and RNN-T.

[0038] The techniques described below in this specification are capable of addressing the issues described above.

[0039] FIG. 1 shows an exemplary computing system 100 for performing inference calculations of a neural network model.

[0040] System 100 includes a hardware accelerator 101 (also referred to as an integrated circuit configuration or chip in the following specification, but in some implementations, it can also be implemented as a plurality of physically separate integrated circuits), and a host 108 arranged off-chip and configured to communicate with the hardware accelerator 101 via a wired or wireless connection.

[0041] Host 108 can be implemented as one or more computer programs on one or more hardware devices that are located off-chip, i.e., not part of the hardware accelerator 101, and generates instructions for controlling the operations of the hardware accelerator 101.

[0042] As shown in FIG. 1, the hardware accelerator 101 includes a controller 102 and a plurality of tiles 132-148 collectively referred to as the compute tile sets 112, 114 and a chip memory (not shown).

[0043] The controller 102 generally includes a data memory 104, an instruction memory 106, and at least one processor configured to execute one or more instructions encoded in the instruction memory. The instruction memory 106 can store one or more machine-readable instructions executable by one or more processors of the controller 102. The instruction memory 106 includes a memory of a specific size, such as 256 kB, for example. The data memory 104 can be any of a variety of data storage media for storing and later accessing various data related to calculations within the system 100.

[0044] The controller 102 is capable of receiving instructions and data parameters from the host 108 and is configured to provide the instructions to the plurality of compute tiles through an instruction bus 124 (described later).

[0045] The data parameters relate to data that defines a neural network model and input data to the neural network model.

[0046] Generally, it is ideal for the controller 102 to receive all instructions from the host 108 at the initial time and store them in the instruction memory 106. In this way, during the inference calculation, the controller 102 can avoid fetching parts of instructions during the calculation that may lead to errors introduced by the iterative loop in the neural network. In the case of a neural network that has a rare but possible iterative loop, the controller 102 can be configured to receive at a certain point in time and calculate these neural network models, and receive a part of all the instructions for fetching other parts at different points in time respectively. Optionally, the controller 102 in the system 100 can be configured to receive more instructions from the host 108 and store them in the instruction memory 106 after at least a part of the previously stored instructions have been executed.

[0047] The controller 102 can instruct one or more computing tiles to perform inference calculations for at least a part of the neural network model.

[0048] Each computing tile is an individual computing unit that cooperates with other computing tiles in the system 100 to accelerate calculations across one or more layers of the neural network model. As shown in FIG. 1, the computing tile set 112 includes computing tile 0, computing tile 1, computing tile 2, and computing tile 3 (i.e., tiles 132, 134, 136, and 138), each of which includes an addressable memory unit 152, 154, 156, and 158 respectively. The computing tile set 114 is composed of computing tiles 142-148, which each include an addressable memory unit 162, 164, 166, and 168 respectively. After each computing tile executes the inference calculation, it stores the partial result at each memory address inside each addressable memory unit according to each instruction broadcast by the controller 102.

[0049] These instructions may each include a respective header (e.g., a bitmap) indicating which compute tile is to execute the instruction.

[0050] The controller 102 can broadcast the instructions for each compute tile. More specifically, the controller 102 can broadcast the instructions received from the host 108 to each compute tile along the data path 118 using the instruction bus 124, that is, the instructions stored in the instruction memory 106 can be transmitted by the instruction bus 124 that provides communication via the data path 118 that connects each compute tile in the compute tile sets 112, 114 back to the controller 102 starting from the controller 102. While transmitting one or more instructions on the instruction bus 124, these instructions can be, for example, 32-bit wide, but the first 7 bits include header information indicating the instruction address / destination that receives and executes these instructions. For example, these first 7 bits can include data parameters representing a specific node ID. Each compute tile along the data path 118 can sequentially inspect the header of the instruction to determine whether the request by the host 108 is addressed to the compute tile inspecting the header.

[0051] When the node ID of the header does not indicate that the destination is the tile being inspected, the tile being inspected copies the input instruction packet to the instruction bus connecting to the next tile for inspection by the next tile. When the node ID of the header indicates that the destination is the tile being inspected, the tile being inspected executes the operation encoded in the input instruction packet.

[0052] The command may include data in the header (e.g., one or more node IDs) to identify a subset of the compute tiles for performing computations among all the tiles. The subset of compute tiles is assigned to perform computations within a particular layer of the neural network. Each compute tile of the subset performs the inference computation for a respective non-overlapping part of the assigned layer. For example, each compute tile of the same tile set (compute tiles 132, 134, 136, and 138 in tile set 112) can be assigned to the same layer of the neural network model and perform the inference computation associated with that layer.

[0053] The command can further identify one or more memory addresses within the corresponding compute tile for outputting data. To address the issues described above for commands having a size that exceeds the limit of the size of the instruction memory 106, the host 108 can generate shared commands for each subset of compute tiles assigned to a particular network layer. The shared command for a particular network layer can identify the memory size and the common memory address for each compute tile in the subset. The shared command for each layer may have data (e.g., one or more node IDs) for identifying each compute tile, and cause the compute tiles to store partial results at their respective common memory addresses, each having the same memory size, in their respective addressable memory units. In some implementations, the controller 102 can generate a plurality of instruction packets for a layer based on each respective shared command for that layer, where each of the plurality of instruction packets has the node ID for the corresponding tile associated with that layer and the common memory address for the corresponding tile for storing partial results having the same memory size.

[0054] The controller 102 can combine the stored partial results generated by each of the associated computational tiles from a common memory address to generate the final output from the neural network for a given input according to the instructions. More specifically, the controller 102 can instruct one or more computational tiles in a subsequent layer to combine the partial results from a previous layer stored at a common address in the previous layer. After completing the inference calculation for the neural network, the controller 102 can instruct the computational tiles in the last layer of the neural network to generate the final output. The controller 102 can then provide the final output to the host 108.

[0055] Thus, the host 108 generates a plurality of instructions and transmits them to the controller 102 to cause the hardware accelerator 101 to execute the inference calculation for the neural network deployed on the hardware accelerator. Once executed by one or more computational tiles in the hardware accelerator 101, these plurality of instructions can cause each of the one or more computational tiles to execute the respective inference operations for at least a portion of the neural network according to those plurality of instructions.

[0056] The plurality of instructions issued by the host 108 that employ the described technique can be shared instructions for a plurality of computational tiles assigned to one or more eligible network layers. The details of allocating the memory of each computational tile assigned to a network layer using the shared instructions will be described in more detail later.

[0057] FIG. 2A illustrates an example of a conventional memory allocation for computational tiles 132, 134, 136, and 136 associated with the layers of a neural network model.

[0058] Host 108 can generate data for assigning the operations represented by each layer of the neural network to respective subsets of all available compute tiles and provide it to controller 102. Each tile of each subset of tiles can execute at least a non-overlapping portion of the operations for the associated layer. For example, host 108 can determine the operations in a particular layer of the neural network and assign them to a compute tile set 112 including compute tiles 132, 134, 136, and 138, such that each compute tile executes at least a non-overlapping portion of the inference computation.

[0059] Each compute tile may include a respective addressable memory unit, and each respective addressable memory unit may include a plurality of memory addresses. For example, tile 132 includes an addressable memory unit 152 having different memory addresses 202, such as memory addresses 202a, 202b, 202c, and 202d.

[0060] Conventionally, controller 102 is configured to broadcast separate instructions to each tile assigned to the same network layer. Each of the separate instructions may include a respective memory allocation instruction for the corresponding tile. The memory allocation instruction can identify the respective memory addresses for the respective addressable memory units of the corresponding compute tile for storing the respective partial results. Each of the respective partial results is generated by the corresponding compute tile that executes each part of the inference computation for the layer assigned when the input is provided.

[0061] Referring to FIG. 2A as an example, a particular layer is assigned to a first set of computing tiles 112 that includes computing tiles 132, 134, 136, and 138. Each of the computing tiles executes at least a part of the inference calculation for that particular layer. These inference calculations may include, for example, tensor calculations of a part of the input activation and the corresponding layer weights. After obtaining the respective partial results from the corresponding parts of the inference calculations, each computing tile stores the respective partial results into different memory addresses according to the respective conventional instructions for that tile. For example, as shown in FIG. 2A, computing tile 132 outputs the respective partial results to the memory address 202a of the addressable memory unit 152 according to the first conventional instruction. Similarly, computing tile 134 outputs to memory address 204b according to the second conventional instruction, computing tile 136 outputs to memory address 206d according to the third conventional instruction, and computing tile 13 8 outputs to memory address 208c according to the fourth conventional instruction.

[0062] The conventional instructions for a given neural network layer, when executed by the tiles associated with that layer, cause those tiles to execute at least respective parts of the inference calculation for that layer, generate respective partial results, and store those partial results at different memory addresses in the respective addressable memory units of the tiles. Doing so can be beneficial because, for example, if a subsequent layer of a given layer is a fully connected layer and thus each tile associated with that subsequent layer requires the entire output of the given layer even if it computes only a partial output of that subsequent layer, each of one or more tiles associated with that subsequent fully connected layer can directly copy each of the stored partial results from different memory addresses associated with the preceding layer to the same memory address in the local memory of that tile.

[0063] More specifically, with reference to FIG. 2A and according to conventional instructions, tiles 132, 134, and 136 can be assigned to the first layer, and tile 138 can be assigned to the fully connected layer following the first layer. Tiles 132, 134, and 136 can obtain their respective partial results for performing inference calculations in the first layer and store their respective partial results in memory addresses 202a, 204b, and 206d, respectively. The respective partial results collectively include the entire set of node outputs from the first layer. To obtain one or more input activations for the subsequent fully connected layer, tile 138 can directly copy the respective partial results from the address 202a of tile 132 to the address 208a of tile 138, from 204b to 208b, and from 206 to 208d, without allocating other memory addresses of tile 138 to store the entire set of node outputs.

[0064] However, since one or more tiles in the first layer store partial results in different local memory addresses and one or more tiles in a subsequent layer (e.g., a fully connected layer) need to copy each of the stored partial results from those different local memory addresses, host 108 needs to generate respective instructions for specifying the memory addresses for the tiles in the first layer for storing the respective partial results and respective instructions for specifying the respective memory addresses for each of the tiles in the subsequent layer for collecting the partial results therefrom.

[0065] To speed up the execution of inference calculations for a particular neural network, the system tends to use as many tiles as possible for parallelization. The total size of the instructions thus increases to different extents based on each parallelization level.

[0066] Furthermore, as more tiles are assigned to a particular layer or more nodes are present in subsequent fully-connected layers, more distinct instructions are required for the tiles associated with a particular layer to store partial results and for the tiles associated with subsequent layers to copy or combine partial results from a particular layer. Thus, as described above, the size of the instructions for performing inference calculations for a neural network increases with the number of computational tiles associated with each layer and the size of the layer (i.e., the number of nodes in the layer).

[0067] As the size of the instructions increases, the efficiency of performing inference calculations for a neural network can be impaired, particularly in the case of large-scale neural networks.

[0068] However, the computing system 100 described below can solve this problem. In connection with FIG. 1, the computing system 100 can first obtain data for the host 108. This data can represent a neural network having a plurality of layers and the type of layer for each of the plurality of layers, such as layer-by-element or fully-connected layers.

[0069] The host 108 can determine and select a subset of layers from all the layers of the neural network indicated by the received data. For simplicity, this subset of layers of the neural network is referred to as eligible layers. Thus, an eligible layer is a layer that meets the eligibility criteria. Eligible layers include layers that are either of the layer-by-element type or the fully-connected type.

[0070] The host 108 can generate respective shared instructions for the computational tiles associated with each respective eligible layer.

[0071] Each shared instruction for tiles associated with a qualified layer may include at least a memory allocation instruction that specifies a common memory address and a predetermined memory size for the associated tiles. Each computing tile can store its respective partial result at a common memory address having a predetermined memory size. The predetermined memory size can be determined by the host 108 or set by the user. The memory size can be determined based on the number of values in the partial results that each tile needs to output after executing the portion of the computation assigned in the layer. For example, the memory size for a layer with at least five node computations and elements associated with each of the tiles assigned to it is larger than that for a layer with only two node computations and elements associated with each of the tiles assigned to it. As another example, a fully connected layer may require a smaller memory size than a per-element layer, depending on how many values each associated tile needs to output.

[0072] In addition to storing each partial output at a common memory address for tiles associated with a qualified layer, the host 108 also generates a shared instruction for tiles associated with a fully connected layer to accumulate the stored data from the previous layer. The previous layer can be any suitable qualified layer, such as a fully connected layer or a per-element layer.

[0073] The host 108 can also generate a shared instruction for a fully connected layer that further includes a shared accumulation instruction that specifies an additional memory address. When executed by a tile, the shared accumulation instruction causes each tile associated with the fully connected layer to obtain a partial result representing all the node results of the previous layer stored at the common address and accumulate the obtained partial results at the additional memory address. This is because, even when the computing tiles associated with the fully connected layer are assigned only to a part of the layer operation, the computing tiles still need to obtain the entire set of results from the previous layer.

[0074] The term "accumulation" refers to combining all the node outputs from a preceding eligible layer into appropriate input activations for one or more nodes in a subsequent fully-connected layer. More specifically, when an accumulation instruction issued by host 108 is executed by each tile associated with each node in the fully-connected layer, each tile associated with each node in the fully-connected layer is caused to gather (e.g., copy and store) the entire set of node outputs from a common address associated with the preceding layer, and based on the entire gathered set of node outputs at an additional common address specified in a shared accumulation instruction, each constructs its respective input activation for each node in the fully-connected layer. Each input activation (or accumulated partial result) generated according to the accumulation instruction can be used by each tile as input for the fully-connected layer to generate a partial result.

[0075] The results gathered by tiles for a fully-connected layer are typically larger in size than the respective partial results stored at a common memory address in the respective addressable memory of each computational tile assigned to the preceding layer. Thus, the shared accumulation instruction generated by system 100 can specify an additional common address having one or more additional common memory addresses of different sizes. For example, the additional common address can be a single additional common address having a larger memory size than the common address for storing partial results for a layer. As another example, the additional common address can include one or more additional common addresses for a fully-connected layer, each having a different size. The total number of additional common addresses and the respective size for each of the additional common addresses can be determined based on the layer size of the preceding layer or based on the size of the partial results gathered from the preceding layer.

[0076] Each computational tile associated with the fully connected layer executes at least respective portions of the inference computation of the fully connected layer to process the aggregated input activations, in accordance with shared instructions issued by the host 108, and generates respective partial results for storage at a common memory address, in accordance with the shared instructions for the fully connected layer.

[0077] For the per-element layer, the shared instructions need not specify another memory address in addition to the common memory address. Each tile associated with the per-element layer need not aggregate the respective partial results associated with the previous layer, nor aggregate the respective partial results associated with the current layer to form the input activations for the next layer, in accordance with the shared instructions issued by the host 108. Instead, each node input activation of the per-element layer is a node output activation from the previous layer, and each output activation of that layer is a node input activation to the subsequent layer. Thus, the computational tiles of the next layer can directly copy the output activations stored at the common memory address of the corresponding computational tiles associated with the previous layer, in accordance with the shared instructions.

[0078] FIG. 2B illustrates an example of a common memory allocation for computational tiles 132, 134, 136, and 138 associated with a qualified layer of a neural network model.

[0079] As shown in FIG. 2B, host 108 generates shared instructions for each of the computing tiles 132, 134, 136, and 138 assigned to the eligible layer. The shared instructions identify a common memory address for each computing tile for storing the respective partial results at a common address in the respective addressable memory units 152, 154, 156, and 158. The common address can be a memory location on the chip memory, which is configured such that when the corresponding tile accesses the data stored at the common address for a particular layer in each of the respective addressable memory units for each computing tile, at least each part in each of the respective addressable memory units "points" thereto. For example, computing tiles 132, 134, 136, and 138 each generate a respective partial result obtained by executing a respective part of the inference calculation in the associated layer, and store the respective partial result at the common address. In some implementations, each memory address 202a for tile 132, 204a for computing tile 134, 206a for computing tile 136, and 208a for computing tile 138 all "point" to the common memory address. Since each computing tile stores its respective partial result at the common address, the stored partial results can also be accessed by other computing tiles using the shared instructions. Thus, host 108 can reduce the instruction size by generating shared instructions for the computing tiles to access the common memory address for fetching the stored partial results. is possible.

[0080] In some implementations, the shared instructions for the tiles associated with a layer can include data for both storage and access, which further reduces the memory bandwidth requirements for receiving instructions from host 108 and broadcasting those instructions using controller 102.

[0081] FIG. 3 illustrates an exemplary allocation of additional memory addresses for compute tiles 132, 134, 136, and 138 associated with the fully connected layers of the neural network model.

[0082] As described above, for each of the fully connected layers, in addition to the respective common addresses for the tiles that store partial results after each part of the computation in a particular fully connected layer, host 108 can generate a shared command that specifies additional memory addresses for the tiles associated with a particular fully connected layer in order to combine the partial results from the preceding eligible layer. More specifically, each tile associated with a fully connected layer can copy the entire set of node outputs from the preceding layer and combine them at the additional memory address. The combined results are used as the respective node activation inputs for performing inference computations in the fully connected layer. Even if a tile associated with a fully connected layer only executes a part of the layer's operations, this tile is instructed to obtain all of the node outputs stored at the common address from the preceding layer and combine those node outputs at the additional memory address of the layer.

[0083] In some implementations, host 108 can identify each node output in the partial results stored at the common memory address for the preceding layer (e.g., by the number or ID associated with the node output), and as a result, the tiles in the fully connected layer can copy each of the identified node outputs from the preceding layer and non-redundantly combine them at the additional memory address.

[0084] Assuming that tiles 132, 134, 136, and 138 are associated with a certain fully-connected layer, the layer preceding that fully-connected layer is a qualified layer. As shown in FIG. 3, each tile distributes additional memory addresses to respective addressable memory units according to a shared command for these tiles. For example, memory unit 152 of tile 132 is configured to include additional memory address 302, memory unit 154 of tile 134 includes additional memory address 304, memory unit 156 of tile 136 includes additional memory address 306, and memory unit 158 of tile 138 includes additional memory address 308. Each additional memory address of each memory unit is associated with an additional common address on the chip memory.

[0085] According to the integration instruction among the shared instructions for the tiles associated with the fully-connected layer, each tile obtains the entire set of node outputs from the partial results stored at the common address for the qualified preceding layer, and accumulates those node outputs at their respective additional memory addresses. For example, calculation tile 132 copies the partial results from the preceding qualified layer and combines them at memory address 302 of memory unit 152, calculation tile 134 copies the partial results from the preceding qualified layer and combines them at memory address 304 of memory unit 154, calculation tile 136 copies the partial results from the preceding qualified layer and combines them at memory address 306 of memory unit 156, and calculation tile 138 copies the partial results from the preceding qualified layer and combines them at memory address 308 of memory unit 158.

[0086] As a more specific example, FIG. 4 illustrates an exemplary and detailed architecture for calculation tile 132 in system 100.

[0087] As shown in FIG. 4, an exemplary computing tile 132 may include narrow-width memory 425 units and wide-width memory units 412a, 412b, 412c, 412d (collectively, 412). The designation of narrow-width and wide-width generally refers to the size (bits / bytes) of the memory units of narrow-width memory 425 and wide-width memory 412. In some implementations, narrow-width memory 425 includes memory units each having a size or width of less than 16 bits, and wide-width memory 412 includes memory units each having a size or width of less than 32 bits.

[0088] Generally, a computing tile 132 assigned to a qualified layer receives data including input activations and parameters from a host along data path 118. The computing tile 132 writes the input activations to the narrow-width memory 425 and the parameters to the wide-width memory 412 according to a shared instruction for the qualified layer. In some implementations, narrow-width memory 524 may include a memory arbiter typically used to determine for each memory cycle which control device (e.g., TensorOp control or DMAOp control) is allowed to access narrow-width memory 210.

[0089] If the qualified layer is a fully connected layer and the layer preceding the fully connected layer is also a qualified layer, then the computing tile 132 aggregates the entire set of node outputs from the partial results obtained for the preceding layer at an additional memory address indicated by the shared instruction.

[0090] More specifically, the compute tile 132 executes each part of the inference computation associated with a particular layer of the neural network by means of MAC operators and an addition register. The compute tile 132 provides the input activations for the layer, along the input bus, from the narrow memory 425 to one or more MAC operators. The compute tile 132 also provides the parameters, from the wide memory unit 412, to one or more MAC operators. The one or more MAC operators and the addition register perform an inner product calculation and an addition using the input activations and the parameters.

[0091] The compute tile 132 provides the partial results generated from the MAC operators and the addition register, along the output bus, to the non-linear unit. The non-linear unit is configured to apply a non-linear function, such as the Sigmoid or ReLU function, to the partial results and generate at least a part of the output activations for a subsequent layer. The compute tile 132 stores the output activations at a common address allocated to the narrow memory unit 425 according to the shared instructions for the layer. Other compute tiles associated with the next layer can fetch the stored output activations from the narrow memory unit of the compute tile 132. In some implementations, the shared instructions for a layer may include data for determining whether the stored partial results from one or more compute tiles of that layer are used to generate the input activations for the compute tiles associated with a subsequent layer, and if so, the stored partial results will belong to the final result (e.g., the input activations for the final output for a subsequent layer or the neural network).

[0092] FIG. 5 illustrates an exemplary process 500 for memory allocation of compute tiles in a system 100 for performing inference calculations for a neural network model. For convenience, process 500 is described as being executed by a system of one or more computers located at one or more locations. For example, a hardware computing system such as system 100 of FIG. 1 programmed appropriately can execute process 500. As a more specific example, the host 108 of system 100 of FIG. 1 can execute process 500 when appropriately programmed.

[0093] The host first obtains data representing the neural network (502). The data representing the neural network includes data identifying the type of neural network (e.g., a convolutional neural network or a recurrent neural network), the total number of layers in the neural network (e.g., 10 hidden layers between the input layer and the output layer), the number of nodes in each layer (e.g., each having at least 10, 20, or 50 nodes including network operations), data representing the sequence of all layers, and data representing the node connections between layers (e.g., whether the layer is a fully connected layer, a partially connected layer, or an element-by-element connected layer with one of the adjacent layers). In some implementations, this data may further include parameters that further define the trained neural network, such as the learned set of weights for each node associated with each layer, data format requirements for the input layer and / or output layer or both (e.g., data representing requirements regarding the size of the input data for the input layer in the trained neural network, or data representing the output format such as the number of output categories for the trained neural network). In some implementations, the host can determine whether the data represents a neural network that has not been trained or a neural network that has been trained but lacks weights. In response, the host can either train the neural network using examples for training or prompt a notification on the user interface indicating that the input neural network has not been trained, and request data representing the trained neural network or data representing the missing weights.

[0094] Controller 102 receives data representing a neural network from host 108 and stores the data in the data memory 104 of controller 102. In some implementations, controller 102 can receive all the data at once or receive a portion of the data at a time until all the data is received.

[0095] Controller 102 also receives instructions from host 108 and stores the received instructions in instruction memory 106. Similarly, the controller can retrieve partial instructions from host 108 at each of multiple times. The controller can also send back to host 108 data including partial results or final results after performing inference calculations for the neural network model.

[0096] After receiving the data, the host determines the type of each layer of the neural network model according to the received data. For example, the host determines whether a network layer is a fully connected layer, a per-element layer, or any other type of layer according to the received data.

[0097] The host then selects (504) a (suitable) subset of the multiple layers of the neural network based on the acquired data. Specifically, the host selects a subset of the multiple layers based at least in part on the type of layer. The selected subset of the multiple layers is also referred to herein as eligible layers. For example, the host can select all fully connected layers from all the layers to form a certain subset. As another example, the host can select both fully connected layers and per-element layers from all the layers to form another subset. In some implementations, the host can select a first subset including all fully connected layers of the neural network and a second subset including all per-element layers of the neural network.

[0098] For each layer of the subset of layers, the host allocates (506) a (suitable) subset of the plurality of computing units to at least partially execute the inference calculations associated with that layer. For example, the node operations for inference calculations in a fully connected layer are distributed among 4 computing tiles. As another example, the node operations for inference calculations in an element-wise layer are distributed among 10 computing tiles.

[0099] Each computing unit includes its own addressable memory unit, and as a result, each computing unit can store partial results in its own addressable memory unit at a given memory address. For example, in relation to Figure 2B, computation 132 includes an addressable memory unit 152, and the addressable memory unit 152 has four different memory addresses 202a - d, each of which "points to" a respective physical address on the chip memory. Each memory address 202a - d can be redirected to another physical memory address by an instruction issued from the host.

[0100] When issuing a shared instruction for the computing tiles assigned to an eligible layer, the host determines (508) the memory size and the common memory address for each computing unit associated with that eligible layer. The determined memory size can be based at least in part on the type of network layer. For example, the determined memory size is larger for a fully connected layer than for an element-wise layer.

[0101] The memory size can also be a predetermined fixed size for a given eligible network layer, based on the hardware architecture, the characteristics of the layer, and the computational requirements. In some implementations, the memory size can instead be preset by the user through a user interface. The memory size can be, to give just a few examples, 5Kb, 10Kb, or 1Mb.

[0102] Host 108 generates a shared command including a memory allocation command, and when executed by each computing unit associated with a layer, causes the computing unit to store, in a common memory address having a memory size in the addressable memory of the computing unit, the result of executing the inference calculation associated with the layer (510). For example, in relation to FIG. 2B, computing tiles 132, 134, 136, and 138 are each associated with a layer on a per-element basis, and each computing tile stores a partial result of the assigned inference calculation at a respective memory address (e.g., 202a, 204a, 206a, and 208a) within a respective addressable memory unit (e.g., 152, 154, 156, and 158). Each of the memory addresses 202a, 204a, 206a, and 208a can be considered a common memory address. However, precisely, each of the memory addresses 202a, 204a, 206a, and 208a has an address identifier that initially "points to" the common memory address of the chip, and thus, each data storage / load process to the memory addresses 202a, 204a, 206a, and 208a is effectively a data storage / load process to the common memory address. Instead, each of the memory addresses 202a, 204a, 206a, and 208a may initially "point to" a respective memory address. However, it may be changed to "point to" the common memory address due to the memory allocation command in the shared command.

[0103] The memory allocation command may further include data for identifying whether the memory allocation command is applied to one or more computing units assigned to respective eligible layers. By doing so, the controller 102, according to the memory allocation command, as needed Accordingly, the memory storage of each computing tile can be selectively controlled. For example, referring again to FIG. 2B, it is possible to identify that data can store partial results at a common memory address having a predetermined memory size only in computing tiles 132, 134, and 136.

[0104] Data for identifying a computing unit can be binary data and includes data representing a node ID of a computing tile and a status flag indicating whether a memory allocation is applicable to the computing tile. For example, 0 indicates that a memory allocation instruction is not applicable to the computing unit having that node ID, and 1 indicates that the instruction is applicable to that computing unit.

[0105] The memory allocation instruction may also include data for tracking partial results stored at a common memory address. The host can issue a sharing instruction that, when executed by the corresponding tile, causes the controller 102 to track a starting common memory address, which is the memory address where each result is stored for a certain layer and to which each of those results belongs to form the final result. More specifically, when one or more compute tiles accumulate partial results to provide one or more layer outputs or the final output for the network, the one or more compute tiles can correctly identify which partial results should be accumulated and where to accumulate the partial results. For example, referring again to FIG. 2B, the compute tile 132 stores (or, similarly, stores the partial result at) the partial result at the address 202a that the common memory address “points to”. After each compute tile has executed its respective inference calculation for the layer, the sharing instruction issued by the host can identify whether the partial result will be used to generate the layer output. If so, the sharing instruction issued by the host can further identify how the partial result should be processed in the arithmetic sequence.

[0106] As described above, the host can determine the type of network layer. In response to determining that the layer is a fully connected layer and for each computing unit associated with that fully connected layer, the host generates a sharing instruction that further includes an accumulation instruction. The accumulation instruction can identify an additional memory address (i.e., a memory address different from the common memory address) used to accumulate the entire set of node outputs from the previous eligible layer. More specifically, when executed by each of the computing units assigned to that fully connected layer, the accumulation instruction can cause each of the compute tiles to accumulate the entire set of node outputs from the partial results stored at the common address associated with the previous eligible layer to that additional common address.

[0107] The additional memory address can be an address accessible to the computing units allocated to the fully connected layer. For example, in relation to FIG. 3, compute tiles 132 and 134 are allocated to one fully connected layer, and instructions issued by the host, when executed, cause controller 102 to allocate memory address 302 to memory unit 312 addressable by memory address 302 and memory address 304 to memory unit 314 addressable by memory address 304. Both memory address 302 and memory address 304 refer to an additional common memory address.

[0108] Compute tiles 132 and 134 can accumulate one or more related partial results obtained from a preceding layer and store them at the additional common memory address. Compute tile 132 and compute tile 134 can access each other to the data stored at the additional common memory address.

[0109] In some implementations, the accumulation instruction further includes data for determining whether one or more of the stored partial results from the preceding layer are accumulated by the computing units associated with the fully connected layer at an additional memory address. This can be useful for parallel computing, particularly when tiles redundantly compute node outputs and store each partial result at a common address. Thus, it is necessary for the tiles in the system to correctly fetch the relevant node outputs from the common address.

[0110] Referring back to FIG. 1, assume that compute tile 148 is associated with a fully connected layer and is assigned to execute an operation on the first node of this layer. Compute tiles 132 and 134 are each associated with a respective eligible preceding layer of the fully connected layer. In this case, compute tile 132 is assigned to execute an operation on the first half of the nodes of the preceding layer, and compute tile 134 is assigned to execute an operation on the second half of the nodes of the preceding layer. When compute tile 148 is associated with the fully connected layer, the aggregation instruction includes data that determines that all relevant partial results stored in compute tiles 132 and 134 should be aggregated to an additional memory address according to a shared instruction (more specifically, an aggregation instruction) for the fully connected layer.

[0111] The neural network system implemented by the system 100 of FIG. 1 can be part of a computer vision system. For example, in a computer vision system for an autonomous vehicle or a robotic system. In real-world vision applications, unfamiliar objects are present almost everywhere due to the so-called "long tail" of objects that occur in the real scene. As another example, this system can be used as part of photo editing software that may need to create new categories on the fly. In addition to object recognition, this system can perform other computer vision tasks such as object detection and segmentation. In other areas, rare and novel objects such as images and sounds may also exist, and the system can also be used in relation to such other areas.

[0112] For example, a neural network system can be used in a language modeling system, an image / video processing system, or an action selection system. For example, tasks can include classification tasks such as image processing tasks, speech recognition tasks, natural language processing tasks, word recognition tasks, or optical character recognition tasks. Further, the tasks can include reinforcement learning tasks, in which an agent interacts with one or more real or simulated environments to achieve one or more goals.

[0113] In the case of a language modeling task or translation from a source language of text to a target language using a neural network, the system can be configured to receive an input sequence of source embeddings representing a source sequence of words in the source natural language and generate an output sequence of target embeddings representing a target sequence of words that is a translation of the source sequence into the target natural language. More generally, the system can be applied to other sequence transduction applications where a source sequence is mapped to a target sequence.

[0114] The input data can include, for example, one or more of image data, moving image / video data, motion data, speech data, audio data, electronic documents, data representing the state of an environment, and / or data representing an action. For example, the image data can include color or monochrome pixel value data. Such image data can be captured from an image sensor such as a camera or a LIDAR sensor. The audio data can include data defining an audio waveform, such as a series of values in a time domain and / or a frequency domain that defines the waveform, which can represent speech in a natural language. The electronic document data can be It may include text data representing words in a natural language. Data representing the state of an environment may include any kind of sensor data, such as pose data and / or position / velocity / acceleration data that characterize the state of a robot or vehicle, or data representing the state of an industrial plant or data center, such as sensed current signals and / or temperature signals. Data representing an action may include, for example, position, velocity, acceleration, and / or torque control data, or data for controlling the operation of one or more items of equipment in an industrial plant or data center. These data may generally relate to the real world or, for example, a simulated virtual environment.

[0115] The output data may similarly include any kind of data. For example, in a classification system, the output data may include classification labels for the input data items. In a regression task, the output data may predict the values of continuous variables, such as control variables for controlling an electronic or electromechanical system, such as a robot, vehicle, data center, or plant. In another example of a regression task operating on image or audio data, the output data may define one or more positions in the data, such as the position of an object, the position of one or more corners of the object's bounding box, or the temporal position of a sound feature in an audio waveform. In a reinforcement learning system, the output data may include, for example, data representing an action, as described above, where the action is an action performed by an agent operating in an environment with respect to a mechanical agent, such as a robot or vehicle.

[0116] Data representing an action may include, for example, data defining an action value (Q-value) for the action, or data parameterizing a probability distribution sampled to determine the action, or data directly defining an action in a continuous action space, for example. Thus, in a reinforcement learning system, a neural network system can directly parameterize a probability distribution for an action selection policy, or can learn to evaluate the value of an action value function (Q-value). In the latter case, multiple memories and their respective output networks can share a common embedding network to provide Q-values for each available action.

[0117] A neural network can be configured to receive any type of digital data input and generate any type of score, classification, or regression output based on that input.

[0118] For example, if the input to the neural network is an image or features extracted from an image, the output generated by the neural network for the given image can be scores for each of a set of target categories, where each score represents the estimated probability that the image contains an image of an object belonging to that category.

[0119] As another example, if the input to the neural network is an Internet resource (e.g., a web page), a document, a part of a document or features extracted from an Internet resource, or a part of a document, the output generated by the neural network for the given Internet resource, document, or part of a document can be scores for each of a set of topics, where each score represents the estimated probability that the Internet resource, document, or part of a document is related to that topic.

[0120] As another example, if the input to the neural network is the feature of the impression context for a specific advertisement, the output generated by the neural network can be a score representing the estimated probability that the specific advertisement will be clicked.

[0121] As another example, if the input to the neural network is the feature of a personalized recommendation for the user, such as a feature characterizing the context for a recommendation or a feature characterizing the user's previous actions made by the user, the output generated by the neural network can be a score for each of a set of content items, and each score represents the estimated probability that the user will respond favorably to being recommended the content item.

[0122] As another example, if the input to the neural network is a sequence of text in one language, the output generated by the neural network can be a score for each of a set of pieces of text in another language, and each score represents the estimated probability that the piece of text in the other language is an appropriate translation of the input text into the other language.

[0123] As another example, if the input to the neural network is a sequence representing a spoken utterance, the output generated by the neural network can be a score for each of a set of pieces of text, and each score represents the estimated probability that the piece of text is an accurate transcription of the utterance.

[0124] The described embodiments of the subject matter and the functional operations described herein can be implemented as digital electronic circuitry, tangibly embodied as computer software or firmware, as computer hardware including the structures disclosed herein and their structural equivalents, or as one or more combinations thereof. The described embodiments of the subject matter can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, such as an electrical, optical, or electromagnetic signal generated by a machine, that is generated to encode information and transmitted to an appropriate receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or include, a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0125] The processes and logical flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. These processes and logical flows can also be performed by, for example, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a general-purpose graphics processing unit (GPGPU), etc., dedicated logic circuitry, and similarly, the apparatus can be implemented as dedicated logic circuitry.

[0126] A computer suitable for the execution of a computer program includes, by way of example, general-purpose and / or special-purpose microprocessors or both, or any other kind of central processing unit, and can be based thereon. Generally, the central processing unit receives instructions and data from a read-only memory, or a random access memory, or both. Essential elements of a computer are a central processing unit for executing or performing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive, transmit, or both receive and transmit data between one or more of such mass storage devices. However, it is not necessary for a computer to have such devices.

[0127] A computer-readable medium suitable for storing computer program instructions and data includes all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as, for example, EPROM, EEPROM, and flash memory devices, and magnetic disks such as internal hard disks or removable disks. The processor and the memory can be supplemented by, or incorporated in, a dedicated logic circuit configuration.

[0128] While this specification contains many details of specific implementations, these should not be construed as limiting any invention or the scope of any claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, the specific plurality of features described herein in the context of separate embodiments may be implemented as a combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may be implemented separately in multiple embodiments, or in any suitable partial combination. Further, while multiple features may be described above as operating in a particular combination and even initially claimed as such, one or more features from the claimed combination may in some cases be excluded from the combination, and the claimed combination may be directed to a partial combination or a variation of a partial combination.

[0129] Similarly, in the drawings, a plurality of operations are shown in a particular order, but this should not be understood as requiring that these operations be performed in the particular order shown or in a sequential order to achieve the desired result, nor that all of the illustrated operations must be performed. In certain circumstances, multitasking and parallel processing may be advantageous. Further, the separation of the various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the program components and systems described are generally understood to be capable of being integrated into a single software product or packaged into multiple software products.

[0130] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve the desired result. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. Obtaining data indicating a neural network having a plurality of layers; For each layer in a subset of the plurality of layers, Allocating a subset of the plurality of computing units from among a plurality of computing units each including an addressable memory unit, to at least partially execute an inference calculation associated with the layer; Determining, in the subset of the plurality of computing units allocated to the layer, a memory size and a common memory address for the respective addressable memory units of each computing unit; Generating a shared command including a memory allocation command; The memory allocation command, when executed by each of the subset of the plurality of computing units, causes the computing unit to store, at the determined common memory address having the determined memory size in the addressable memory of the computing unit, a result of executing the inference calculation associated with the layer. A method.

2. The subset of the plurality of layers is selected from the plurality of layers, and the selection includes: Determining a layer type of each of the plurality of layers of the neural network based on the obtained data indicating the neural network. The method according to claim 1.

3. The selection is at least partially based on the determined layer type. The method according to claim 2.

4. In response to determining that the layer type of the layer is a fully connected layer, determining an additional memory address different from the common memory address of the computing unit, and the shared command further includes an aggregation command, The aggregation command, when executed by each of the subset of the plurality of computing units for the layer, causes the computing unit to aggregate one or more results associated with another layer preceding the layer, and stores the aggregated results at the determined additional memory address in the addressable memory of the computing unit. The method according to claim 2.

5. determining, in the subset of the plurality of computing units assigned to the layer, the memory size for the respective addressable memory unit of each computing unit comprises The method of claim 2, comprising determining, in the subset of the plurality of computing units assigned to the layer, the memory size for the respective addressable memory unit of each computing unit, based at least in part on the determined layer type of each layer. **Claim 6** for each layer of the plurality of layers that is separate from the subset of the plurality of layers assigning a second subset of the plurality of computing units to at least partially perform an inference calculation associated with the layer, from among the plurality of computing units each including a respective addressable memory unit; and The method of claim 1, further comprising, in the second subset of the plurality of computing units, generating, for each corresponding computing unit, one or more memory allocation instructions. **Claim 7** The memory allocation instructions The method of claim 1, wherein the memory allocation instructions further include data identifying one or more computing units of the subset of the plurality of computing units to which the memory allocation instructions are applied. **Claim 8** The method of claim 7, wherein the data identifying the one or more computing units is binary instruction data. **Claim 9** The method of claim 1, wherein the memory allocation instructions further include data for tracking, for each computing unit of the subset of the plurality of computing units, a common memory address of each of the respective stored results generated by the computing unit. **Claim 10** The integration command further includes data for each computing unit of the subset of the plurality of computing units with respect to the layer, specifying whether each of the results associated with the other layer preceding the layer is to be integrated at the additional memory address in the addressable memory of the computing unit of the layer, according to the method of claim 4.

11. The method according to claim 1, further comprising providing the sharing command to the plurality of computing units.

12. A method comprising: providing a set of instructions for performing inference calculations on a plurality of layers of a neural network to a system including a plurality of computing units, each computing unit including a respective addressable memory; the set of instructions includes: a first memory allocation command associated with a first layer in the plurality of layers of the neural network, the first memory allocation command identifying a first memory address of the respective addressable memory and a first subset of the plurality of computing units; the set of instructions further includes: a second memory allocation command associated with a second layer in the plurality of layers of the neural network, the second memory allocation command identifying a second memory address of the respective addressable memory and a second subset of the plurality of computing units; the second memory address is different from the first memory address, and the second subset is different from the first subset; the set of instructions causes the system to: for each computing unit of the first subset, output the result of the inference calculation associated with the first layer in the plurality of layers to each memory address of the addressable memory of the computing unit based on the first memory address; and for each computing unit of the second subset, output the result of the inference calculation associated with the second layer in the plurality of layers to each memory address of the addressable memory of the computing unit based on the second memory address.

13. The first subset of the plurality of computing units corresponds to the plurality of computing units in which the inference calculations associated with the first layer in the plurality of layers are distributed. and The method according to claim 12, wherein the second subset of the plurality of computing units corresponds to the subset of the plurality of computing units in which the inference calculations associated with the second layer in the plurality of layers are distributed. **Claim 14** The first memory allocation instruction further specifies a first memory size, and the second memory allocation instruction further specifies a second memory size. The set of instructions further causes the system to allocate the first memory size to each respective memory address in the addressable memory of each computing unit of the first subset, based on the first memory address, for each computing unit of the first subset; and to allocate the second memory size to each respective memory address in the addressable memory of each computing unit of the second subset, based on the second memory address, for each computing unit of the second subset. The method according to claim 12. **Claim 15** The method according to claim 14, wherein the first memory size is larger than the second memory size. **Claim 16** The method according to claim 12, wherein the first layer in the plurality of layers includes a fully connected layer, and the second layer in the plurality of layers includes an element-wise layer. **Claim 17** The method according to claim 12, wherein the set of instructions further includes one or more memory allocation instructions associated with each of one or more layers of the plurality of layers that are different from the first layer and the second layer. **Claim 18** The method according to claim 16, wherein the set of instructions further includes a first integration instruction associated with the first layer. **Claim 19** When the first integration instruction associated with the first layer is executed by each computing unit of the first subset, the system is caused to allocate to the first subset additional memory addresses associated with each respective computing unit, The method according to claim 18, wherein the additional memory address is different from the first memory address. **Claim 20** The method according to claim 19, wherein the first integration instruction further includes data for determining, for each computing unit of the first subset, whether each result of the inference calculation associated with the previous layer of the first layer is integrated into each respective memory address of the computing unit, based on the additional memory address.

21. In response to determining that the results of the inference calculations associated with the previous layer of the first layer are to be integrated, the first integration instruction associated with the first layer further causes the system to integrate the results of the inference calculations associated with the previous layer into respective memory addresses of the corresponding computing units, based on the additional memory address, as claimed in claim 20.

Images