Implementation of neural networks in multi-core hardware

Abstract

Implementations of neural networks in multi-core hardware are disclosed. Multi-core hardware implementations of deep neural networks are disclosed. Multiple layers of the network are arranged in multiple layer groups. Input data for the network comprises a multi-dimensional tensor comprising one or more traversal dimensions, which are dimensions traversed by a stride in at least one layer of a first layer group. The hardware implementation is configured to split the input data for the first layer group into at least a first tile and a second tile along at least one of the traversal dimensions, each tile comprising a plurality of data elements in each of the one or more traversal dimensions. A first core is configured to evaluate the multiple layer groups depth-first based on the first tile. A second core is configured to evaluate the multiple layer groups depth-first based on the second tile.

Description

TECHNICAL FIELD

[0001] The present invention relates to implementations of neural networks in multi-core hardware. BACKGROUND

[0002] A deep neural network (DNN) is an artificial neural network that can be used for machine learning applications. In particular, DNNs can be used for signal processing applications, including image processing and computer vision applications.

[0003] DNNs have been implemented in applications where power resources are not a significant factor. Nonetheless, DNNs have applications in many different technical fields where hardware resources for implementing DNNs are constrained in terms of power consumption, processing capability, or silicon area. Therefore, there is a need for hardware configured to implement a DNN (or at least a portion thereof) in an efficient manner, e.g., in a manner that requires less silicon area or less processing power to operate. Moreover, DNNs can be configured in a variety of different ways for a variety of different applications. Therefore, there is also a need for hardware for implementing DNNs with flexibility to support a variety of DNN configurations. As DNNs become more complex in structure and are required to process larger amounts of data in shorter time periods (e.g., for inference in real-time applications), there is an increasing need for hardware with greater processing capability while maintaining or improving area efficiency and power efficiency. SUMMARY

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] A multi-core hardware implementation of a deep neural network is disclosed. A plurality of layers of the network are arranged in a plurality of layer groups. Input data for the network comprises a multi-dimensional tensor comprising one or more traversal dimensions, which are dimensions traversed by a stride in at least one layer of a first layer group. The hardware implementation is configured to split the input data for the first layer group into at least a first tile and a second tile along at least one of the traversal dimensions, each tile comprising a plurality of data elements in each of the one or more traversal dimensions. A first core is configured to evaluate the plurality of layer groups depth-first based on the first tile. A second core is configured to evaluate the plurality of layer groups depth-first based on the second tile.

[0006] According to one aspect, a method for implementing a deep neural network in multi-core hardware is provided.

[0007] According to the method, groups of layers of the neural network can be evaluated in a depth-first manner by splitting the data into tiles whose evaluation is distributed over multiple cores. Here, depth-first means that a given core evaluates multiple consecutive groups of layers based on a given tile before continuing any other evaluation. For example, a first core evaluates a first group of layers based on a first tile, and then a second group of layers based on a first output tile of the first group of layers, before the first core evaluates any other tile or group of layers. A second core acts accordingly. Each output tile of each group of layers acts as an input tile for the next group of layers, up to the last group of layers. Splitting the data into tiles along one or more of the dimensions of traversal can enable the groups of layers to be processed in a depth-first manner by layer fusion, while reducing the size requirement of the one or more second memory devices. The size reduction is possible because the one or more second memory devices do not need to store the entire tensors output by each group of layers. They can store a subset of the tensors corresponding to that tile.

[0008] Evaluating a group of layers means evaluating each layer of the group of layers. A group of layers can be defined such that all layers of a single group of layers can be evaluated in a single hardware pass. For example, all layers of a single group of layers can be evaluated in a hardware pipeline of a given core without needing to write any data to the one or more second memory devices or the one or more first memory devices. In some cases, a group of layers can consist of a single layer of the neural network. In other cases, a group of layers can consist of multiple layers. For example, a group of layers can include any one or any two or more of: a single convolutional layer, a single pooling layer, a single activation layer, a single normalization layer, and a single element-wise operation layer. In some cases, each of the first group of layers and the second group of layers can include a convolutional layer. Optionally, a group of layers can include at most one convolutional layer.

[0009] The cost of accessing the one or more second memory devices can be less than the cost of accessing the one or more first memory devices. The cost can be in terms of time or power consumption. For example, the one or more first memory devices can be provided as off-chip memory; the one or more second memory devices can be provided as on-chip memory.

[0010] The dimensions of traversal can include an X dimension (columns) and a Y dimension (rows), and optionally a batch dimension B. The input data can include additional dimensions, such as a channel dimension C. The input data can be processed by a number of filters to produce a corresponding number of output channels F.

[0011] The step of splitting the input data can be pre-planned before starting the evaluation of the groups of layers. In other words, the splitting can be planned before the training phase or the inference phase of the neural network starts. The splitting can be pre-planned based on the parameters of the layers in the respective groups of layers and the tensor sizes at the input and output of each layer.

[0012] The plurality of layer groups can form a tile group. A tile group is a group of layer groups that are processed together depth-first per tile. A tile group starts with an initial layer group and ends with a final layer group. The output tile of the final layer group in the tile group can be stored in one or more first memory devices.

[0013] The deep neural network can be a convolutional neural network.

[0014] The method can also include storing the first and second output tiles of the final layer group of the plurality of layer groups in one or more first memory devices. In this way, intermediate data generated while evaluating the layer groups is stored in one or more second memory devices, and output data generated by evaluating the final layer group is stored in one or more first memory devices.

[0015] The method can optionally also include analyzing at least one characteristic of the input data for the first layer group, and determining that the input data can be partitioned into tiles along the at least one of the traversal dimensions based on the analysis, where the partitioning of the input data for the first layer group is performed in response to the determination. The at least one characteristic can optionally include a size of the input data in the at least one traversal dimension, and determining that the input data can be partitioned into tiles can include determining that the size in the at least one traversal dimension is greater than a threshold size.

[0016] If the size in the at least one traversal dimension is not greater than the threshold size, then the evaluation of the layer group can be partitioned along one of the other (non-traversal) dimensions. In other words, while processing of larger tensors can be divided among the cores along one or more traversal dimensions, smaller tensors can be divided among the cores along other dimensions such as a batch or channel dimension.

[0017] The size of the tiles can vary within and between layer groups. In particular, the size of the head tile and the tail tile of the input data of a layer group can be different than the other tiles of that layer group. Due to the effects of data dependencies, the head tile in an earlier layer group of a tile group can be larger than the head tile in a later layer group of the tile group. For a given layer group, the other tiles (i.e., those tiles between the head tile and the tail tile) can have a uniform size. These combined factors can mean that the tail tile tends to shrink towards earlier layer groups.

[0018] The head tile of the initial layer group can be larger than all other tiles of that layer group. This can be due to the head tile “growing” backwards through the layer group due to data dependencies.

[0019] The tile size can be chosen such that the size of the head tile in the final group of layers is smaller than some or all of the other tiles in the final group of layers. This can help to balance the load between different cores by slowing down the "growth" of the head tile going backwards through the group of layers. If the head tile in any given group of layers is too large, then that tile can delay the overall evaluation (see also the note below regarding synchronization between cores). Reducing the size of the initial tile in the final group of layers reduces the size of all groups of layers (because of the data dependencies), thus helping to avoid situations where the initial tile is much larger (and takes much longer to process) than the other tiles.

[0020] The input data for the first group of layers can be split into a first number of tiles, and the input data for the second group of layers can be split into a second number of tiles, where the second number differs from the first number by at most one. In particular, in some cases (depending on the characteristics of the network), a group of layers can need one more tile than its previous group of layers. As explained above, this is due to the different sizes of the initial and tail tiles. Since the size of the tail tile can shrink towards earlier groups of layers, it can sometimes disappear completely.

[0021] Splitting the input data for the first group of layers can comprise splitting it into a first tile, a second tile, and a third tile, the method optionally further comprising: assigning the third tile to the first core; and after using the first core to evaluate each group of layers based on the first tile, using the first core to evaluate each group of layers based on the third tile.

[0022] When a group of layers is split into more tiles than the number of cores in the multi-core hardware, this can be done. The number of tiles can be chosen such that it is an integer multiple of the number of cores (for at least one of the groups of layers). The evaluation of all groups of layers based on the first and second tiles is referred to as a first multi-core tile propagation process. Then, the evaluation of all groups of layers based on the third tile represents a second multi-core tile propagation process (at least part of). Note that if the third tile is the tail tile of the multiple tiles, then the second core can be idle during the second multi-core tile propagation process.

[0023] The one or more second memory devices can include a local memory of the first core that is accessible only by the first core, a local memory of the second core that is accessible only by the second core, and a shared memory that is accessible by the first core and the second core. The method can also include identifying a one-time portion of the first output tile of the first layer group that is needed only to evaluate the second layer group based on the first tile, and identifying an overlap portion of the first output tile of the first layer group that is needed to evaluate the second layer group based on the first tile and also needed to evaluate the second layer group based on the second tile, the method optionally including when storing the first output tile of the first layer group in the one or more second memory devices: writing the one-time portion of the first output tile to the local memory of the first core; and writing the overlap portion of the first output tile to the shared memory.

[0024] As discussed above, the third tile can be assigned to the first core. The method can also include identifying a one-time portion of the second output tile of the first layer group that is needed only to evaluate the second layer group based on the second tile, and identifying an overlap portion of the second output tile of the first layer group that is needed to evaluate the second layer group based on the second tile and also needed to evaluate the second layer group based on the third tile. The method can include when storing the second output tile of the first layer group in the one or more second memory devices: writing the one-time portion of the second output tile to the local memory of the first core; and writing the overlap portion of the second output tile to the shared memory.

[0025] The overlap portion of the second output tile can be saved in the shared memory for a longer time than the overlap portion of the first output tile. In particular, the overlap portions can be kept in the shared memory (only) as long as they are needed. The overlap portion of the first output tile can be kept until it is used by the second core when evaluating the second layer group based on the second output tile. The overlap portion of the second output tile can be kept until it is used by the first core when evaluating the second layer group based on the third output tile.

[0026] The first layer group can include a convolutional layer, the method further including storing weight values and / or bias values of the convolutional layer in the shared memory. This can allow each of the cores to access the coefficient (weight and / or bias) data to evaluate the convolutional layer for its respective one or more tiles.

[0027] The method can also include waiting for the first core to complete evaluation of the first layer group for the first tile and waiting for the second core to complete evaluation of the first layer group for the second tile, and then using the second core to evaluate the second layer group for the second tile.

[0028] This can represent the minimal synchronization condition to satisfy the data dependency requirements, in particular when the first output tile comprises overlapping data. It reflects the fact that the second kernel will need (i) the overlapping data of the first output tile of the first layer group, and (ii) the second output tile of the first layer group, in order to evaluate the second layer group based on the second tile. The presence of overlapping data will depend on the structure of the network. For example, a convolution layer will generate overlapping data if the kernel size is larger than the stride (ignoring the dilation size); a convolution transpose will generate overlapping data if the stride is larger than 1; and a pooling layer will generate overlapping data if the kernel size or the stride is larger than 1.

[0029] The method can comprise waiting for all the kernels to complete the evaluation of the first layer group before any kernel starts evaluating the second layer group.

[0030] In principle, the minimal synchronization condition allows each kernel to continue its processing of a tile in the depth direction as long as the necessary overlapping data from a previous tile is available. Allowing the kernels to "run free" in this way can result in some kernels being significantly ahead of others in their processing. While in principle this can be a good way to exploit the potential of the kernels, in practice the inventors have recognized that this does not generally result in an overall gain in performance. After a certain number of layer groups ("tile groups") have been processed in the depth direction, the output data can be written to one or more first memory devices. A tile group can comprise a subset of the layers of the neural network, or it can comprise all the layers of the neural network. A tile group is completed only when all the constituent tiles have completed. Thus, the hardware is inevitably still limited by the slowest path of the parallel evaluation operations on the tiles. At the same time, there can be a significant management overhead to support the free running. Therefore, the hardware implementation can be simplified by synchronizing all the kernels after each layer group, without slowing down the processing. On the other hand, in some applications (e.g. speech processing) it can be advantageous for some output data to be available as early as possible. In this case, the reduction in the latency to produce the first output that can be achieved by free running can justify the increased management overhead.

[0031] The input data for the first layer group optionally comprises a first traversal dimension and a second traversal dimension, wherein the input data can be stored in one or more first memory devices ordered first according to the first dimension, and wherein the input data can be partitioned along the second dimension.

[0032] For example, the input data can be stored row-wise, in other words ordered according to the X dimension. In this case, the input data can be partitioned into tiles in the Y dimension. Alternatively, the input data can be stored column-wise (ordered first according to the Y dimension), and it can be partitioned into tiles along the X dimension.

[0033] When input data elements are tightly packed into memory, and there is no one-to-one correspondence between data elements and burst sizes for reading and writing memory, it can be advantageous to split in this way. It can allow data to be split by reading a contiguous burst from memory for each tile. Likewise, when reassembling output tiles of the final layer group, it can be possible to write each output tile by a single contiguous burst.

[0034] In some embodiments, input data can be split only along the second dimension, whereby the tiles have the actual size of the original tensor in the first dimension.

[0035] If the size of the input data in the second dimension is not greater than a threshold size, the input data can be split along the first dimension or another dimension.

[0036] According to another aspect, there is provided a hardware implementation of a deep neural network.

[0037] The first and second cores can be configured to operate in parallel, with the first core evaluating the first layer group based on the first tile and the second core evaluating the first layer group based on the second tile.

[0038] The one or more second memory devices can comprise: a local memory of the first core, accessible only by the first core; a local memory of the second core, accessible only by the second core; and a shared memory, accessible by the first and second cores. The splitter can be configured to: identify a one-shot portion of the first output tile of the first layer group, which is only needed to evaluate the second layer group based on the first tile; and identify an overlap portion of the first output tile of the first layer group, which is needed to evaluate the second layer group based on the first tile and also based on the second tile, and the first core can be configured to, when storing the first output tile of the first layer group in the one or more second memory devices: write the one-shot portion to its local memory; and write the overlap portion to the shared memory.

[0039] The second core can be configured to retrieve the overlap portion from the shared memory when evaluating the second layer group based on the second tile.

[0040] Storing the overlap data so that it is accessible by the second core can avoid the need to recalculate the overlap data when evaluating the second layer group based on the second tile.

[0041] Alternatively, the need to store overlapping data can be avoided by computing the overlapping data in both the first core and the second core. In other words, all data required by a core for a subsequent layer group can be computed by that core. In this case, the first output tile of the first layer group and the second output tile of the first layer group will contain a common portion (corresponding to the overlapping portion). This approach results in some duplication of computation between cores operating in parallel; however, it has the advantage that no core at the end of a layer group needs to wait for any other core - that is, each core has complete freedom to run freely, independent of the computation being performed by any other core. The first output tile of the first layer group can be stored entirely in the local memory of the first core; and the second output tile of the first layer group can be stored entirely in the local memory of the second core.

[0042] Each core can comprise a plurality of processing elements configured to process input data to evaluate a layer of a neural network.

[0043] There is also provided an artificial intelligence accelerator system comprising a hardware implementation as outlined above. There is also provided an artificial intelligence accelerator system configured to perform a method as outlined above. Such an artificial intelligence accelerator system can be embodied in hardware on an integrated circuit.

[0044] There is also provided a method of manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system.

[0045] There is also provided a method of manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system, the method comprising: processing a computer readable description of the artificial intelligence accelerator system using a layout processing system so as to generate a circuit layout description embodying an integrated circuit of the artificial intelligence accelerator system; and manufacturing the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system.

[0046] There is also provided computer readable code configured so as to cause a method to be performed when the code is run. There is also provided a computer readable storage medium having said computer readable code encoded thereon.

[0047] There is also provided an integrated circuit definition data set which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture an artificial intelligence accelerator system.

[0048] There is also provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an artificial intelligence accelerator system, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the artificial intelligence accelerator system.

[0049] There is also provided a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an artificial intelligence accelerator system, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process the computer-readable description of the artificial intelligence accelerator system using a layout processing system so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system.

[0050] There is also provided an integrated circuit manufacturing system configured to manufacture an artificial intelligence accelerator system.

[0051] There is also provided an integrated circuit manufacturing system comprising: a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an artificial intelligence accelerator system; a layout processing system configured to process the computer-readable description so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and an integrated circuit generation system configured to manufacture the artificial intelligence accelerator system from the circuit layout description.

[0052] The layout processing system can be configured to determine position information of logical components of a circuit derived from the integrated circuit description so as to generate the circuit layout description of the integrated circuit embodying the artificial intelligence accelerator system.

[0053] Computer program code for performing any of the methods described herein can be provided. A non-transitory computer-readable storage medium can be provided having stored thereon computer-readable instructions which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0054] As will be apparent to those skilled in the art, the above features can be suitably combined and can be combined with any of the aspects of the examples described herein. BRIEF DESCRIPTION OF DRAWINGS

[0055] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0056] Figure 1 is a schematic diagram of an example deep neural network (DNN);

[0057] Figure 2 is a schematic diagram of example data in a DNN;

[0058] Figure 3 is a block diagram of an example hardware implementation of a convolutional layer of a DNN;

[0059] Figure 4 is a block diagram of an example convolution engine of Figure 3 is a block diagram of an example convolution engine of

[0060] Figure 5 This is a schematic diagram illustrating the data flow according to an exemplary hardware implementation.

[0061] Figure 6 This shows the data dependencies across multiple layers of the DNN for the first block of data;

[0062] Figure 7 The second block of data is shown. Figure 6 Data dependencies within multiple layers;

[0063] Figure 8 It is a concept map of overlapping data concepts;

[0064] Figure 9 An exemplary allocation of memory associated with the hardware implementation of a DNN is illustrated schematically;

[0065] Figure 10 A DNN with four layers and no branches is shown;

[0066] Figure 11A An exemplary grouping of layers in a DNN with four layers having branches is shown;

[0067] Figure 11B Showing the target Figure 11A An exemplary grouping of alternatives to DNNs;

[0068] Figure 12 This is a block diagram based on the example of a multi-core hardware implementation.

[0069] Figure 13 This illustrates the process according to the first aspect: when the input data is segmented into tiles and these tiles are processed in a depth-first manner... Figure 12 An example of execution order in multi-core hardware;

[0070] Figures 14A-14B This illustrates synchronization between cores in another example, where the input data is segmented into tiles and processed in a depth-first manner;

[0071] Figure 15 This is a flowchart illustrating a method for implementing a convolutional neural network in multi-core hardware when input data is segmented into tiles;

[0072] Figure 16 An example of the execution order when the evaluation of a layer is divided into a propagation process, according to the second aspect, is shown;

[0073] Figure 17 This is a flowchart illustrating how to implement a convolutional neural network in multi-core hardware when the evaluation of a layer is divided into a propagation process;

[0074] Figure 18 A computer system in which a graphics processing system is implemented is shown; and

[0075] Figure 19 An integrated circuit manufacturing system for generating an embodiment of a graphics processing system is shown.

[0076] The drawings show various examples. A skilled artisan will understand that the boundaries shown by the elements in the drawings (e.g., boxes, groups of boxes, or other shapes) represent one example of boundaries. In some examples, it can be the case that an element can be designed as multiple elements, or that multiple elements can be designed as one element. Where appropriate, common reference numerals have been used throughout the various figures to designate similar features. DETAILED DESCRIPTION

[0077] The following description is presented to enable any person skilled in the art to make and use the application. The present application is not limited to the embodiments described herein, and various modifications can be made by persons skilled in the art to the disclosed embodiments. Embodiments are now described, by way of example only.

[0078] A deep neural network (DNN) is a type of artificial neural network that includes multiple interconnected layers that can cause the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An example DNN 100 that includes multiple layers 102-1, 102-2, 102-3 is shown. Each layer 102-1, 102-2, 102-3 receives input data and processes that input data according to the layer to produce output data. The output data is either provided as input data to another layer or output as final output data of the DNN. For example, in the DNN 100 of Figure 1 In the DNN 100, the first layer 102-1 receives original input data 104 of the DNN 100 and processes the input data according to the first layer 102-1 to produce output data. The output data of the first layer 102-1 becomes input data to the second layer 102-2, which processes the input data according to the second layer 102-2 to produce output data. The output data of the second layer 102-2 becomes input data to the third layer 102-3, which processes the input data according to the third layer 102-3 to produce output data. The output data of the third layer 102-3 is output as output data 106 of the DNN.

[0079] The processing performed on the input data to a layer depends on the type of the layer. For example, each layer of a DNN can be one of a number of different types. Exemplary DNN layer types include, but are not limited to, convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It will be apparent to those skilled in the art that these are exemplary DNN layer types, and that this is not an exhaustive list, and that other DNN layer types can exist.

[0080] A convolutional layer is configured to convolve input data using weights associated with the layer. In particular, each convolutional layer is associated with a plurality of weights w1...wn, which can also be referred to as filter weights or coefficients. The weights can be grouped to form or define one or more filters or kernels. i

[0081] Referring to Figure 2 , the figure illustrates an example of the format of data 200 used in a DNN. As can be seen in Figure 2 , the data 200 used in a DNN can be arranged in p data planes, with each plane having dimensions of x x y. The DNN can include one or more convolutional layers, with each convolutional layer being associated with a plurality of filters, each filter including a plurality of weights. Each filter has dimensions of m x n x p (i.e., each filter includes a set of m x n x p weights w), and is applied to the input data according to a convolution operation across a number of steps, referred to as strides, s and t, as shown in Figure 2 . The group of data elements to which the filter is applied at each stride is referred to as a window. Each filter produces one output plane. The number of filters and the number of weights per filter can vary between convolutional layers. Convolutional Neural Networks (CNNs) are an effective method of image recognition and classification, which is a special type of DNN, generally including a plurality of convolutional layers.

[0082] ​Before describing the multi-core hardware implementations of the subject matter of the present application, a single-core example will be described. A (single-core) hardware implementation of a convolutional layer can include a hardware module or block (which can be referred to herein as a "processing element") that is configured to compute the sum of the products between the weights forming all or a portion of a filter and the input data values forming all or a portion of a window (which can be referred to as a filter window computation). Since typically a large number of such filter window computations are performed, some hardware implementations can include multiple such processing elements so that more than one filter window computation can be performed in parallel. Preparing each processing element to perform a filter window computation involves reading from one or more first memory devices the appropriate input data and weights for each filter window computation and providing it to one of the processing elements. Typically, a large amount of data needs to be transferred from the memory to the processing elements. If this cannot be done efficiently, it can result in high memory bandwidth requirements and high power consumption for providing the input data and weights to the processing elements. This is especially true when the memory is "off-chip" memory, i.e., implemented in a different integrated circuit or semiconductor die than the processing elements.

[0083] Accordingly, described herein are hardware implementations of neural networks and methods for processing data in such hardware implementations, in which input data and weights are provided to the processing elements in an efficient manner to reduce the memory bandwidth and power consumed required to provide the input data and weights to the processing elements. In particular, blocks (also referred to as "tiles") of data can be processed in a manner along the depth direction, through multiple layers of the neural network. Intermediate outputs of layers within the multiple layers can be stored in one or more second memory devices. The size of the one or more second memory devices can be smaller than the size of the one or more first memory devices used to batch store the data, but can be accessed in a faster and / or more energy-efficient manner. In one example, at least one of the one or more second memory devices is provided on-chip (i.e., in the same integrated circuit or semiconductor die as the processing elements).

[0084] Reference is now made to Figure 3 which illustrates an exemplary single-core hardware implementation 300 of a convolutional layer of a DNN. The hardware implementation 300 includes a plurality of convolution engines 302, a plurality of accumulators 304, an accumulation buffer 306, a coefficient buffer 308, and an input buffer 310. Each convolution engine 302, along with its respective accumulator 304 and its share of resources in the accumulation buffer 306, represents a processing element 318. The hardware implementation also includes a double data rate (DDR) SDRAM 312 as off-chip memory; on-chip memory (OCM) 314, and an output buffer 316.

[0085] Each convolution engine 302 includes hardware logic configured to receive a set of weights (e.g., {wl...w8}) representing all or a portion of a filter and a set of input data values (e.g., {dl...d8}) representing all or a portion of a window and perform multiply-accumulate computations on the received weights and input data values, as shown in Figure 4 In some examples, as shown in Figure 4 Each convolution engine 302 can include multiple multipliers 402, each configured to multiply a weight (wl i ) and a corresponding input data value (dl i ) to produce a multiplication output value. The multipliers 402 are followed by a plurality of adders 404 forming an adder tree to compute a sum of the multiplication outputs. In the example shown in Figure 4 The convolution engine 302 includes eight multipliers 402, but in other examples, there can be more or fewer multipliers. For example, in some cases, there can be 128 multipliers. In general, if there are Z multipliers 402, then the adder tree includes Z-1 adders 404.

[0086] In some cases, the multiply-accumulate computations are pipelined. For example, the multipliers 402 and the adders 404 can be divided into multiple pipeline stages, with register stages (not shown) before the first pipeline stage and between each pair of pipeline stages. For example, the multipliers can form a first pipeline stage, and the adders can be divided into tiers, with subsequent pipeline stages including one or more tiers of adders.

[0087] Figure 3 The example hardware implementation 300 includes four convolution engines 302; however, it will be apparent to those skilled in the art that the methods and principles described herein are applicable to hardware implementations having any number of convolution engines.

[0088] Because the convolution engines can take more than one hardware propagation to generate a complete filter result (e.g., because the convolution engines can receive and process only a portion of the filter weights and / or a portion of the input data values of the window in a cycle), the hardware implementation can include multiple accumulators 304. Each accumulator 304 receives the output of one convolution engine 302 and adds that output to the previous convolution engine output related to the same filter. Because the convolution engines can not generate or produce outputs related to the same filter in consecutive cycles, partial results for one or more filters can be stored in a cumulative buffer 306, which can then provide the appropriate partial results to the accumulators in each cycle. In some examples, the cumulative buffer 306 can be capable of storing partial results related to 128 different filters.

[0089] Coefficient buffer 308 includes memory (not shown) to store a plurality of weights related to a convolutional layer (or a fully connected layer) of the DNN and hardware logic (not shown) to provide the weights to convolution engine 302 for processing in a predetermined order in a plurality of cycles. The plurality of weights can include all weights related to the convolutional layer or only a portion of the weights related to the convolutional layer. Although coefficient buffer 308 is shown as a single module, coefficient buffer 308 can be implemented, for example, by a plurality of coefficient buffers each forming a bank of memory.

[0090] Input buffer 310 includes memory (not shown) to store a plurality of input data values related to a convolutional layer (or a fully connected layer) of the DNN and hardware logic (not shown) to provide the input data values to convolution engine 302 for processing in a predetermined order in a plurality of cycles. The plurality of input data values can include all input data values related to the convolutional layer or only a portion of the input data values related to the convolutional layer. Although input buffer 310 is shown as a single module in Figure 3 , input buffer 310 can be implemented, for example, by a plurality of input buffers each forming a bank of memory.

[0091] DDR 312 is coupled to on-chip memory 314 for providing weight data to on-chip memory 314. DDR 312 is also coupled to input buffer 310 for providing a block of input data to input buffer 310. On-chip memory 314 is coupled to coefficient buffer 308 for providing weight data to coefficient buffer 308. On-chip memory 314 is also coupled to input buffer 310 for providing intermediate output data (including input data for a subsequent layer) to input buffer 310. On-chip memory 314 is also coupled to DDR 312 for providing a block of output data for an ending layer of a plurality of layers to DDR 312. Accumulation buffer 306 is coupled to output buffer 316 to allow output buffer to receive intermediate output data for a layer within the plurality of layers and output data for the ending layer. Output buffer 316 is coupled to on-chip memory 314 for providing the intermediate output data and the output data for the ending layer to on-chip memory 314.

[0092] The various connections are shown separately in Figure 3 the example; however, in some embodiments, some or all of them can be provided by one or more shared bus connections. It should also be understood that other connections can be provided, as Figure 3alternatives or supplements to the connections shown. For example, the output buffer 316 can be coupled to the DDR 312 for providing output data directly to the DDR 312. As a further example, the DDR 312 can be coupled to the coefficient buffer 308 for providing weight data directly to the coefficient buffer 308. Likewise, in some cases, not all of the connections shown are necessary. For example, the DDR 312 does not necessarily have to be coupled to the input buffer 310, which can instead obtain input data from the DDR 312 via the on-chip memory 314. Figure 3 All of the connections shown are necessary. For example, the DDR 312 does not necessarily have to be coupled to the input buffer 310, which can instead obtain input data from the DDR 312 via the on-chip memory 314.

[0093] Figure 5 is a simplified schematic to better illustrate the data flow according to one example of a hardware implementation similar to Figure 3 Figure 5 A plurality of layers of a neural network (consisting of two layers 502-1, 502-2) is shown. These can be any two consecutive layers of the network. Note also that the plurality of layers is not limited to two layers. The figure shows input data 504 to a first layer 502-1 of the plurality of layers. The intermediate output data 506 of the first layer 502-1 forms the input data to a second layer 502-2 (the end layer in the example of the two layers). The figure also shows output data 508 of the second layer 502-2 (the end layer). The DDR 312 is a first memory device with a relatively high read / write cost. The on-chip memory 314 is a second memory device with a relatively low read / write cost. For example, reading / writing a given amount of data from the DDR can consume more time and energy than reading / writing that amount of data from the on-chip memory 314. It will be appreciated that DDR memory is merely one example of a mass memory storage device, and that other memory / storage technologies can also be used, such as flash memory.

[0094] According to Figure 3 and Figure 5 The DDR 312 stores input data for the first layer 502-1 of the plurality of layers 502-1, 502-2. The DDR 312 also stores weight data representing weights of the plurality of layers 502-1, 502-2. In the example of the present invention, the weight data is transferred from the DDR 312 to the on-chip memory 314. The coefficient buffer 308 reads the weight data from the on-chip memory 314 and provides it to the processing element 318. The input buffer 310 reads a first subset of input data from the DDR 312 representing a first block of the input data 504. This first subset of the input data 504 (also referred to as the “head tile”) is read from the DDR 312 while the weight data is being transferred from the DDR 312 to the on-chip memory 314. The input buffer 310 provides the first subset of the input data 504 to the processing element 318. The processing element 318 processes the first subset of the input data 504 using the weight data to generate the intermediate output data 506 of the first layer 502-1. The processing element 318 provides the intermediate output data 506 to the on-chip memory 314. The on-chip memory 314 stores the intermediate output data 506. The input buffer 310 reads a second subset of input data from the DDR 312 representing a second block of the input data 504. The input buffer 310 provides the second subset of the input data 504 to the processing element 318. The processing element 318 processes the second subset of the input data 504 using the weight data to generate the intermediate output data 506 of the first layer 502-1. The processing element 318 provides the intermediate output data 506 to the on-chip memory 314. The on-chip memory 314 stores the intermediate output data 506. This process is repeated for each block of the input data 504 until the output data 508 of the second layer 502-2 is generated. Figure 5 ​The input data 504 is shown in gray shading and can include one or more data planes. The input buffer 310 provides a first subset of the input data 504 to the processing element 318. The processing element 318 processes the weight data and the first subset of the input data 504 to evaluate each of a plurality of layers to compute a first block of output data 508 of a final layer. The first block of output data 508 is written to the output buffer 316. Figure 5 The output data 508 is shown in gray shading and can include one or more data planes. The output buffer 316 can write the first block of output data 508 to on-chip memory 314. The first block of output data 508 can be transferred from the on-chip memory to the DDR 312. Alternatively, the output buffer can write the output data directly into the DDR memory 312 (as shown by the dashed arrow in Figure 3 and Figure 5 ).

[0095] As shown in Figure 5 , the data input to and output from each layer is arranged in one or more planes (also referred to herein as input data sets and output data sets). In the example shown in Figure 5 , the input data 504 includes p in = 3 data planes. The intermediate output data 506 of the first layer 502-1 includes p1= 3 data planes. The output data 508 of the second layer 502-2 includes p = 2 data planes. The number of planes depends on the weight data. In particular, the weights that make up each filter take a predetermined number of input planes as input data, and each filter outputs a plane of output data. Thus, the number of data planes produced by a given layer is equal to the number of filters applied by that layer. Each data plane can represent a feature map. In the example shown in Figure 5 , the three planes of input data 504 can represent three feature maps generated by a previous layer of the neural network (if the first layer 502-1 in the figure is not the first layer of the neural network). Alternatively (if the first layer 502-1 is the first layer of the neural network), the three planes of input data 504 can represent three input data planes. For example, input data with multiple planes often occurs in image processing tasks. Such input planes can include, but are not limited to: red, green, and blue (RGB) planes; or hue, saturation, and value (HSV) planes. The input planes can also include depth data as one of the input planes.

[0096] To compute the first block of output data, processing element 318 must evaluate each of the plurality of layers (as described above). In particular, coefficient buffer 308 reads the weight data for the first layer 502-1 from on-chip memory 314. Processing element 318 processes the weight data for the first layer 502-1 and the first subset of input data 504 to evaluate the first layer 502-1 of the plurality of layers, thereby computing output data 506 for the first layer. This is a block of output data for the first layer 502-1 corresponding to the first block of input data 504. In other words, it is a subset of the complete set of output data for the neural network of the first layer 502-1. This block / subset is shown in grey shading in Figure 5 Output buffer 316 writes the block of output data for the first layer 502-1 to on-chip memory 314. It represents the intermediate output data that will be required as input data to the next layer (second layer 502-2). To evaluate the next layer (second layer 502-2), input buffer 310 reads the block of output data 506 for the first layer 502-1 stored therein by output buffer 316 from on-chip memory 314. Coefficient buffer 308 reads the weight data for the second layer 502-2 from on-chip memory 314. Processing element 318 processes the weight data for the second layer 502-2 and the block of output data 506 for the previous (first) layer to evaluate the second layer 502-2, thereby computing output data for the second layer 502-2. For any subsequent internal layers of the plurality of layers, the process continues— saving the intermediate output data in on-chip memory 314 each time, and reading this intermediate output data to provide input data to the next layer. This continues until the final layer is reached. In the example of Figure 5 the second layer 502-2 is the final layer. Thus, the output for the second layer 502-2 is the first block of output data 508. As already explained above, output buffer 316 can write the first block of output data 508 to on-chip memory 314. The first block of output data 508 can be transferred from this on-chip memory to DDR 312. Alternatively, output buffer 316 can write the output data directly to DDR 312 (dashed arrow).

[0097] In this manner, processing a subset of the input data (a tile) through multiple layers of the neural network can efficiently utilize the processing resources of the processing elements 318. Writing intermediate output data to and reading it from the on-chip memory 314 to process the next layer reduces the need to read and write data from the off-chip DDR 312, thereby helping to reduce power consumption. Note that the on-chip memory 314 can be much smaller than the off-chip DDR 312, as during each propagation pass, the on-chip memory 314 only needs to store data associated with the current tile of input data, the current tile of output data, and the associated tile of intermediate output data for the internal layers. This is in stark contrast to conventional approaches of evaluating layers of a neural network, which independently process each layer in its entirety, and where the memory needs to be able to store the full input data and full output data for the layer at any given time. This requires a much larger memory, which is typically impractical to implement on-chip. Thus, conventional approaches in practice require the use of off-chip memory to store the intermediate output data. As off-chip read and write operations typically consume more energy than on-chip read and write operations, reducing the amount of data written to off-chip memory can significantly save power consumption.

[0098] The size of the tile to be processed during each propagation pass can be selected based on the available storage capacity of the on-chip memory 314, the number of layers in the plurality of layers, and the structure of the neural network. An exemplary approach to memory management in the on-chip memory 314 will be described below. First, it is useful to consider how the tile of output data for a layer in the plurality of layers depends on the intermediate output data of the previous layer and the input data of the first layer. Reference is made to Figure 6 This can be understood. The figure illustrates the input data to three layers (not explicitly shown) of a neural network; the output data of the first layer (here labelled layer 0); the output data of the second layer (labelled layer 1); and the output data of the final layer. For simplicity, it can be convenient to consider the output data of the final layer as a single feature map created by a single filter. Likewise, it can be convenient to consider the other layers as convolutional layers with a single filter, thereby creating a single output feature map, as shown in simplified form in the figure. Of course, the scope of the present disclosure is not limited to this approach, these are merely simplifications made for ease of understanding. At each layer, a single filter can be applied to multiple input channels (feature maps), or multiple filters can be applied to one or more input channels, thereby processing and / or generating multiple data planes. The approaches and hardware implementations according to the present disclosure can equally be used in such scenarios.

[0099] From the first block of output data in the end layer (tile 1), it is possible to trace back through the network to find the intermediate and input data on which this output block depends. There is a block of data in the layer 1 output (tile 1) that corresponds to tile 1 in the end layer in the sense that tile 1 in layer 1 is needed to compute tile 1 in the end layer. It is assumed that the filter has a kernel size larger than 1. Because of such a kernel size, tile 1 in the layer 1 output is larger than tile 1 in the end layer output. In other words, tile 1 in the end layer output depends on some additional data in the layer 1 output. For example, for a kernel of 3x3 and a stride of 1, one extra row of output data is needed in the layer 1 output. Figure 6 The dashed lines in the figure indicate this expansion of the amount of data needed. In this figure, each tile is separated by a solid line.

[0100] Next, consider which data in the layer 0 output is necessary to compute tile 1 in the layer 1 output. For diversity considerations, it will be assumed that the stride of layer 1 is larger than 1. This means that the layer 1 output is smaller than the layer 0 output. There is a block in layer 0, tile 1, that corresponds to tile 1 in layer 1. Because the kernel size is also larger than 1; therefore the amount of data needed at the border is further increased. Therefore, tile 1 in the layer 0 output is larger than tile 1 in the layer 1 output.

[0101] The same happens when tracing back from layer 0 to the input data. There is a block in the input data, tile 1, that corresponds to tile 1 in layer 0. It is assumed that the stride in layer 0 is 1; therefore the size of the input data is about the same as the size of the layer 0 output (depending on padding). Again, because the kernel size is larger than 1, additional data is needed at the border. Therefore, tile 1 in the input data is again larger than tile 1 in the layer 0 output. Tile 1 in the input data constitutes the first subset (block 1) of the input data. Tile 1 in the output data of the end layer constitutes the output block.

[0102] From the above, it can be concluded that the total amount of data needed for the first tile increases from the end block to the input block. This depends specifically on the parameters of all intermediate layers (kernel size, stride, dilation). Figure 6 Also shown in the figure is the "overlap". This is the data that can be reused in the subsequent propagation, i.e. in the computation of the subsequent tile. The overlap data is indicated by the horizontal dashed lines in each layer data. Overlap 1 in the layer 1 output is the part of tile 1 that can be (repeatedly) used to compute the subsequent block (tile 2) of the end layer output. Overlap 1 in the layer 0 output is the part of tile 1 that can be (repeatedly) used to compute tile 2 of the layer 1 output. The size of the overlap depends only on the parameters of the next layer (kernel size, dilation). For example, if the stride = 1 and the dilation = 1, then the size of the overlap (the number of rows of the overlap) is the kernel size minus 1.

[0103] Immediately Figure 6 ,Figure 7 The data dependencies of the second tile are shown. The second tile of the output data of the final layer (tile 2) is adjacent to the first tile (tile 1). As with tile 1, tile 2 depends on the number of intermediate data (eventually input data) that is added when traversing the layers backwards. However, some of the data dependencies are the same as for tile 1 in the output data of the final layer. In particular, when evaluating layer 1 of tile 1 (tile 1), overlap 1 in the output of layer 1 has already been computed. The only additional data that needs to be computed is the tile in layer 1, i.e. tile 2. Likewise, moving backwards to the layer 0 output, the only additional data that needs to be computed is the tile in layer 0, i.e. tile 2. Finally, moving backwards to the input data, the only additional input data that needs to be read from memory is tile 2 (tile 2). As Figure 6 is shown, Figure 7 Each tile in is separated by a solid line. Overlap data is separated by a dashed line. The dashed line indicates the upper limit of the data dependencies (i.e. the first row of data needed to compute the tile in the next layer). The slanted dashed line indicates the lower limit of the data dependencies (i.e. the last row of data needed to compute the tile in the next layer). Thus, for example, tile 2 in the layer 1 output depends on overlap 1 and tile 2 in the layer 0 output. Tile 2 contains overlap 2, which will be reused in the next pass to compute tile 3 of the subsequent layer, and so on. Note that if the overlap data of the first pass is preserved, the amount of “fresh” data needed by tile 2 does not increase when traversing these layers backwards (except for the cases indicated by a stride greater than 1). The same is true for all subsequent passes (tiles) after tile 2. In other words, for all passes after the first pass (operating on the head tile), a smaller amount of input data needs to be retrieved from memory and a smaller amount of intermediate data needs to be computed at each layer.

[0104] This leads to an important observation: the overlap data constitutes a non- one-time portion of the data. The rest of each tile constitutes a one-time portion of the data. This means that the overlap data is not only needed for computing the output data of the final layer for the current tile, but the overlap data is also needed later for computing the output data of the final layer for a subsequent tile. In contrast, the rest of the data in each tile is necessary for computing the output data of the final layer for the current tile, but can be deleted / overwritten after that.

[0105] To consider this in detail with a simplified one-dimensional example, refer to Figure 8 . Figure 8 The illustration in shows individual data elements in the input data and output data of each layer of three layers (layer 0, layer 1 and the final layer) of a neural network. Each circle represents a data element (which can be, for example, a pixel in an image or feature map). Each element depends on a certain subset of elements in the previous layer. Figure 8The connecting lines in indicate the elements of the previous layer on which a given element depends. Thus, for example, the 3rd element in the output of the final layer depends on the 2nd, 3rd, and 4th elements in the output of layer 1. The fourth element in the output of the final layer depends on the 3rd, 4th, and 5th elements in the output of layer 1. In other words, in this example, the kernel size of the final layer is 3, and the stride is 1. The kernel size of layer 1 is 3, and the stride is 2. The kernel size of layer 0 is 3, and the stride is 1. Since the stride of both layer 0 and the final layer is 1, the output of layer 0 is approximately the same size as the input data, and the output of the final layer is approximately the same size as the output of layer 1 (depending on padding needs). Meanwhile, since the stride of layer 1 is 2, the output of layer 1 is approximately half the output of layer 0. The dashed lines represent the boundaries between blocks (tiles). That is, the 3rd data element shown in the final layer is the final data element of the first block (tile 1). As an example, consider the 4th element in the output of the final layer. It can be seen that each element in the output of the final layer depends on: 3 elements in the output of layer 1; 7 elements in the output of layer 0; and 9 elements in the input data. The elements in tile 1 (except for the overlap 1 element) are shown in diagonal shading. Three such elements (1-3) are shown in the output of the final layer by way of example.

[0106] Due to the data dependencies, some elements can be discarded after the final layer output of the current block (tile 1) has been computed. These elements are referred to as “one-shot” elements. Some other elements will also be used to compute the final layer output of the next block (tile 2). These elements are referred to as “non-one-shot” elements. In particular, the overlap 1 element (shown in Figure 8 in solid black) is a non-one-shot element. Elements 7 and 8 in the input data are used to compute elements 6 and 7 in the output of layer 0 on the first pass through the layers to compute the final layer output data for the current block (tile 1). However, these elements are also needed to compute elements 8 and 9 in the output of layer 0. These elements 8 and 9 are not needed when computing the first block during the first pass, but will be needed when computing the second block during the second pass. Thus, to minimize memory access overhead, it is desirable to keep elements 7 and 8 of the input data in on-chip memory 314 to avoid having to read these elements from DDR 312 again during the second pass. Elements 7 in the output of layer 0 and elements 3 and 4 in the output of layer 1 are also overlap 1 elements, as these will also be needed when performing the computation during the second pass (element 7 in the output of layer 0 will be needed to compute element 5 in the output of layer 1; and elements 3 and 4 in the output of layer 1 will be needed to compute elements 4 and 5 in the output of the final layer).

[0107] Now consider Figure 8The remaining elements of tile 1 are shown with diagonal shading. Element 6 of the input data is used to compute elements 5, 6, and 7 of the layer 0 output. However, each of these three elements must be computed in the first pass of the propagation process for the computation of the first output tile (tile 1). Element 6 of the input data is not needed in any additional computation in the second pass of the propagation process; therefore, this element is "one-time" data. Similarly, element 6 in the layer 0 output is used to compute element 4 in the layer 1 output, but this element is not needed in the second pass of the propagation process (provided that element 4 in the layer 1 output is retained). The elements of tile 2 (except for the overlap 2 elements) are shown as hollow circles. The overlap 2 elements are shown with gray shading. These elements will be reused in the computation of tile 3, and therefore are non-one-time data. Some of the elements of tile 3 are shown with horizontal hatching at the bottom of the figure.

[0108] The inventors of the present invention have recognized that one-time data is only used to compute the next layer in the current pass of the propagation process through the plurality of layers. Once the next layer in the current pass of the propagation process is computed, the one-time data from the previous layer can be deleted / overwritten. The non-one-time data (overlaps) must be retained for the second pass of the propagation process so that it can be used in the computation of the second tile. This recognition can reduce unnecessary read / write operations from / to the DDR 314, and can also reduce unnecessary recomputation of intermediate output data.

[0109] Reference will now be made to Figure 9 A memory management strategy according to one example of a hardware implementation will now be described. The inventors have recognized that one-time data is relatively large, but is only valid for a short time. In contrast, non-one-time data (overlaps) is relatively small, but is valid for the entire propagation process. Figure 9 One example memory allocation for the on-chip memory 314 that takes advantage of these characteristics is shown. In this example, it is assumed that six layers of a neural network (numbered 0, 1, 2, 3, 4, 5) are to be processed together. In one branch, each layer is connected to its previous layer. In a second branch, layer 5 is connected to layer 1. This means that the output data of layer 1 is part of the input data for layer 5.

[0110] The memory management policy divides the on-chip memory 314 into three segments. A first segment 802 (which can be referred to as "scratch" memory) is used to store one-time data. A second segment 804 (which can be referred to as "heap" memory) is used to store non-one-time data. A third segment 806 (coefficient memory) is used to store weight data for each layer. Within the first segment 802, multiple slots A, B, C are defined. In a given pass (for a given block to compute output data), the method proceeds as follows. The one-time output of layer 0 is written to slot A. The non-one-time output of layer 0 is written to the beginning of the second segment 804. During evaluation of layer 1, the one-time output of layer 1 is written to slot B, and the non-one-time output of layer 1 is written to the second segment 804, e.g., appended to the non-one-time output of layer 0. At this point, the one-time data from layer 0 is no longer needed. Thus, during evaluation of layer 2, the contents of slot A can be overwritten with the one-time output of layer 2. The non-one-time output of layer 2 is again appended to the data in the second segment 804. However, when layer 3 is evaluated, neither slot B nor slot A should be overwritten. Due to branching, the contents of slot B (the one-time output of layer 1) are again needed to evaluate layer 5. Of course, the contents of slot A are immediately needed to evaluate the current layer (layer 3). Thus, the one-time output of layer 3 is written to slot C. The one-time output of layer 4 is written to slot A, overwriting the contents of this slot that are no longer needed for further computation. To evaluate layer 5, the one-time output data from layer 1 is read from slot B, and the one-time output data from layer 4 is read from slot A. (The non-one-time output data from layers 1 and 4 is read from the second segment 804.)

[0111] In this way, the slots used to store one-time data are frequently reused during a given pass, thereby helping to reduce the total amount of on-chip memory 314 occupied at any given time. The size of each slot can be selected based on the maximum size of one-time data to be stored in the slot during a pass. Note that in this example, the second segment 804 is double buffered. The non-one-time output data (overlaps) for all layers of a current block are stored in one set of memory locations. During the next pass (for computation of the next block), the overlaps are retrieved from this set of memory locations as needed. The overlaps for the next block are stored in a second set of memory locations. For subsequent passes (subsequent blocks), the hardware implementation alternates between using the first set of memory locations and the second set of memory locations, i.e., reads from one set while writing to the other.

[0112] The one-time and non-one-time portions of the data can be identified by analyzing the structure of each of the plurality of layers. Similarly, the operation of assigning the one-time data of each layer to a particular slot in the first section 802 of the on-chip memory 314 can be done by analyzing the overall layer structure of the plurality of layers. This analysis can be done dynamically by software at runtime, or when mapping a given neural network to a given hardware implementation. Alternatively, this can be defined manually by the designer of the hardware implementation. However, the use of software facilitates mapping a variety of different neural networks to a given hardware implementation.

[0113] As mentioned above, the plurality of layers of the neural network that are evaluated together in each propagation (also referred to herein as a "tile group") can comprise all layers of the neural network, but this is not required. The plurality of layers can consist of a subset of the layers of the entire neural network. Depending on the structure of the neural network, different groupings of layers can be found to be advantageous. Figure 10 A simple neural network is shown, which consists of four layers, with no branching, each depending only on the output of the previous layer. This leaves a high degree of freedom in choosing how to group the layers together. For example, the layers can be grouped into one or more groups of layers according to any of the following grouping strategies:

[0114] 1-4

[0115] 1-2, then 3-4

[0116] 1-2, then 3, then 4

[0117] 1-3, then 4

[0118] 1, then 2-4

[0119] 1, then 2-3, then 4

[0120] 1, then 2, then 3-4

[0121] In general, subject to the size of the on-chip memory 314, it can be preferable to group as many layers together as possible.

[0122] Figure 11A A slightly more complex network structure is shown. Again, there are four layers, each depending on the previous layer. However, there is an additional branch from layer 2 to layer 4. In this case, it can be advantageous to group the layers 1 and 2 together, and the layers 3 and 4 together. Figure 11AIn this case, the network is divided into two groups, each group having two layers (Layer 1 to Layer 2 and Layer 3 to Layer 4), indicated by the dashed boxes in the figure. Note, however, that this can not be optimal in terms of memory access requirements. The output of Layer 1 to Layer 2 will be written to DDR 312. When evaluating Layer 3 (the first layer of Layer 3 to Layer 4), this output will be read from DDR 312; however, the evaluation of Layer 4 also requires the output of Layer 2. Thus, the same data must be read again from DDR 312 in order to evaluate Layer 4. Conversely, Figure 11B An alternative grouping of the layers is shown, which can be more efficient in terms of memory access requirements. Here, Layer 1 is handled by itself, and Layers 2 to 4 are grouped into multiple layers. In this way, both branches are captured within multiple layers. A single read of the output data of Layer 1 from DDR 312 is made, as input data for the first layer of the multiple layers (Layer 2). The output data of Layer 2 is handled in the multiple layers using the memory management strategy described above with reference to Figure 9

[0123] First, reference is made to the above Figure 2 , which gives an example of a convolutional layer of a neural network operating on two-dimensional (2D) image data in multiple p-planes or feature maps. It should be understood, however, that the present disclosure is not limited to this manner. The above principles are applicable to 1D, 2D, or higher dimensional data. When applied to 2D (or higher dimensional) data, multiple ways can be employed to construct the blocks (tiles) of input data and output data. Each block can be defined in one dimension, two dimensions, or more. For example, for 2D image data, blocks defined in one dimension can form 2D stripes (in the horizontal or vertical direction). Overlapping data can be defined above and below (for horizontal stripes) or to the left and right (for vertical stripes) of the stripes. With this in mind, the figures of Figure 5 , Figure 6 and Figure 7 are drawn. (These figures are also applicable to 1D blocks defined in 1D data.) It should be understood that while Figures 5-8 indicates that the data is divided into blocks in the Y (height) dimension, this is not limiting. As mentioned above, the same approach can be applied in the X (width) dimension. In some cases, however, it can be more preferable to split the tiles in one dimension over the other. For example, if the data is stored in memory by rows, and the individual data elements are not aligned neatly with the burst size used to read / write the memory, it can be preferable to split the tiles in the Y (height) dimension.

[0124] ​Alternatively, for 2D image data, blocks can be defined in two dimensions to form a 2D rectangle of data. Overlapping data can then be defined above, below, to the left, and to the right of this rectangle. The overlapping data from the current block may be useful for computing multiple subsequent blocks of output data, such as the block to the right of the current block, the block below the current block, and the block to the lower right of the current block (assuming the blocks are processed in raster scan order). The same principle can be extended to blocks in more dimensions that have overlapping data in multiple directions in multidimensional space.

[0125] In the example above, the depth-first processing of the neural network on a tile-by-tile basis is discussed layer by layer. This applies to... Figure 3 Implementation 300 is primarily designed to process a single convolutional layer. However, it is possible to provide a more refined hardware implementation that adds additional processing modules to implementation 300, enabling the processing of multiple layers in a single pass through the hardware implementation. For example, pooling modules, element-wise operation modules, normalization modules, and activation modules could be provided (see, for example, GB 2566702 A). This could, for instance, allow the execution of a convolutional layer, followed by an activation function, and then a pooling operation in a single pass through the hardware implementation. In some cases, data can be piped from one layer / operation to the next without being written to on-chip memory. When the hardware implementation is capable of performing multiple sequential operations in a single hardware pass, it makes sense to group layers processed together by the hardware. To accommodate this, the concept of a "layer group" is introduced. A layer group is a set of layers that can be processed together by the hardware implementation in a single pass through the hardware. A layer group can consist of a single layer (as referenced above). Figure 3 (as described in the example); however, a layer group can consist of more than one layer if the hardware has the necessary functionality.

[0126] As mentioned above, segmenting data into tiles can provide a way to facilitate parallel multi-core implementations of convolutional neural networks. Now, reference will be made to... Figures 12-15 Describe an example of a multi-core implementation based on the first aspect. Figure 12 A block diagram of hardware implementation 600 is shown. It includes three cores 350-0, 350-1, and 350-2. Each core has a corresponding local on-chip memory (OCM) 314a, which can only be accessed by that core. Hardware implementation 600 also includes a shared on-chip memory 314b. All cores are able to access (i.e., read and write) the shared OCM 314b via a high-bandwidth interconnect 610. A partitioning unit 620 is provided, which is configured to manage the allocation of work among the cores.

[0127] Figure 13One example of using the different cores 350 of the hardware implementation 600 to process tiles in a depth-first manner is shown schematically. The individual tiles are similar to those described earlier. In this example, the tiles are partitioned along the Y dimension of the input data tensor. This is convenient when the data is stored in memory row- first, i.e. prioritizing the X dimension. As shown, the neural network comprises five groups of layers (labelled LG0 to LG4). Each group of layers is indicated as comprising three layers, represented by the circles in the diagram. (It will be appreciated that this is not limiting - different groups of layers can have different numbers of layers.) The groups of layers are to be evaluated by the hardware implementation in two “tile groups” (labelled Tile Group 0 and Tile Group 1). A tile group consists of a group of groups of layers that will be evaluated together in a depth-first manner after splitting the data into tiles. Input data for the first group of layers in a tile group is read from the DDR 312, and output data from the final group of layers in a tile group is written to the DDR 312. Within a tile group, intermediate data (in tiles) is written by each group of layers to the OCM 314, and read by the next group of layers from the OCM 314. The intermediate data can be stored partly in the local OCM 314a, and partly in the shared OCM 314b. This will be explained in more detail below.

[0128] As shown, in this example, the input data stored in the DDR 312 is split into six tiles (labelled Tile 0 to Tile 5) for evaluation of Tile Group 0. There are only three cores 350; therefore, not all tiles can be processed in parallel in this tile group. Instead, the first three tiles are each assigned to one of the three cores for a first multi-core tile propagation process (multi-core tile propagation process 0). Here, a multi-core tile propagation process refers to the evaluation of all groups of layers of a tile group for N consecutive tiles, where N is the number of cores. Tile Group 0 consists of the first three groups of layers LG0-LG2. In multi-core tile propagation process 0, core 0 evaluates groups of layers LG0-LG2 based on tile 0; core 1 evaluates these groups of layers based on tile 1; and core 2 evaluates these groups of layers based on tile 2. This is indicated by the solid arrows in Figure 13 . Next, as indicated by the dashed arrows, the hardware implementation 600 proceeds to multi-core tile propagation process 1, in which core 0 evaluates groups of layers LG0-LG2 based on tile 3; core 1 evaluates groups of layers based on tile 4; and core 2 evaluates groups of layers based on tile 5. This completes the evaluation of the first tile group. Note that Figure 13 is primarily intended to show the order of execution of the cores on the tiles. This diagram does not provide detailed instructions of which parts of the data are stored in which memory. Further details are provided below in Figure 14A and Figure 14B to explain how the data is arranged in the memories 312 and 314.

[0129] During each multi-kernel tile propagation, the output tile of LG0 is written to OCM 314 and read from OCM 314 as the input tile of LG1. Similarly, the output tile of LG1 is written to OCM 314 and read from OCM as the input tile of LG2. The output data of the first tile group, i.e., the output tile of LG2, is written to DDR 312 before hardware implementation 600 continues to evaluate tile group 1.

[0130] Tile group 1 consists of two layer groups, LG3-LG4. As shown, for this tile group, the data is split into only three tiles. The number of tiles can be chosen based on the size of the input data of the tile group, which depends on the size on OCM 314. The number of tiles can also be influenced by the number and configuration of layers within the tile group, as these will affect the amount of data that needs to be stored in OCM 314 during the evaluation of the tile group for each tile. Tile group 1 can be evaluated in a single multi-kernel tile propagation, as the number of tiles is not greater than the number of cores. Core 0 evaluates layer groups LG3-LG4 based on tile 0; core 1 evaluates layer groups LG3-LG4 based on tile 1; and core 2 evaluates layer groups LG3-LG4 based on tile 2. The output tile of LG3 is written to OCM 314 and read from OCM 314 as the input tile of LG4. The output tile of LG4 is written to DDR 312.

[0131] Figures 14A-14B It is shown (for different examples) how tiles can be assigned to and processed by multiple cores. Figure 14A The output data of the three layer groups LG0-LG2, which are split into tiles, is shown. Here, the notation “LGxTn” denotes “layer group x, tile n”. As explained above with reference to Figure 5 The size of the tiles in each layer is defined by the data dependencies in the neural network layer, as explained above with reference to

[0132] The tiles are processed in a multi-core architecture with four cores (C0-C3). In Figure 14A and Figure 14BOf the two, tiles are shaded / hatched depending on which core processes them. Figure 14B How layer groups are processed over time by the cores is shown, with arrows indicating data dependencies between tiles. During the first multi-core tile propagation, the first four tiles are processed by the four cores C0-C3, respectively. In the first layer group, the head tile LG0T0 is larger than the other tiles, so core C0 takes longer to evaluate the layer group based on this tile than the other cores C1-C3 take to evaluate the layer group based on the other tiles. In the example of the present invention, each core waits until all cores have finished evaluating the layer group. Thus, cores C1-C3 wait for C0 to finish the head tile LG0T0. This provides a simple and effective way to achieve synchronization between cores. After evaluation of the first layer group, all four cores continue evaluation of the second layer group, followed by the third layer group. From Figure 14B It can be seen in that during evaluation of the final layer group in this multi-core tile propagation, the head tile LG2T0 is evaluated faster than the other tiles. This is because the head tile LG2T0 has been chosen to be smaller than the subsequent tiles LG2T1-LG2T7. This is done to avoid the head tile becoming too large in the initial layer. The aim is to try to balance the workload between the cores, and to try to avoid one core finishing its work significantly earlier or later than any other core.

[0133] As explained above with reference to Figure 7 and Figure 8 Evaluation of a layer group for a given tile (except for the head tile) depends on overlapping data from a previous output tile in a previous layer group. For example, evaluation of LG1T1 depends on both LG0T1 and LG0T0. Likewise, evaluation of LG1T2 depends on LG0T2 and LG0T1, and so on. This dependency between tiles during the multi-core tile propagation is shown by the solid arrows in Figure 14B . The dependency means that each core needs to access overlapping data produced by another core. To this end, the overlapping (non-ephemeral) data for each tile is stored in the shared OCM 314b. The remaining data (i.e. the ephemeral data) can be stored in the local OCM 314a of each core, as it does not need to be accessed by another core. This can avoid the need for each core to recompute the overlapping data, while also avoiding the potentially expensive overheads involved in writing the overlapping data to the DDR 312. At the same time, access to the local OCM 314a can be less costly than to the shared OCM 314b, providing improved efficiency compared to an implementation where all data is stored in the shared OCM.

[0134] It is noted that it is not necessary in principle for each core to wait for all cores to complete an LG. For example, in principle, the evaluation of LG1T2 could start immediately after the completion of LG0T1 and LG0T2. Similarly, the evaluation of LG1T0 could start at any time (i.e., an indeterminate amount of time) after the completion of the evaluation of LG0T0. This “free-running” approach could indeed be employed in other implementations. However, the present implementation has the advantage of simplicity, as it avoids the additional complexity necessary to track free-running.

[0135] The end of the first multi-kernel tile propagation process is indicated by the black, thick vertical line in Figure 14B . Next, the cores conduct a second multi-kernel tile propagation process to evaluate the layer groups based on the remaining tiles. With respect to the second multi-kernel tile propagation process, several points are noted. For the second and third layer groups LG1-LG2, it is understood that the fifth tile relies on overlapping data produced when the fourth tile was evaluated in the previous multi-kernel tile propagation process. In other words, while overlapping data from LG0T0, LG0T1, and LG0T2 is used by the cores “immediately” in the evaluation of LG1T1, LG1T2, and LG1T3, respectively, overlapping data from LG0T3 needs to be preserved in the shared OCM 314b until the second multi-kernel tile propagation process, at which time it is used by core CO in the evaluation of LG1T4. Similarly, overlapping data from LG1T3 is preserved in the shared OCM 314b until it is used by core CO in the evaluation of LG2T4. These dependencies between tiles processed in different multi-kernel tile propagation processes are illustrated by the dashed arrows in Figure 14B . Another point to note is that the evaluation of the second layer group based on the tail tile LG1T7 completes faster than the other tiles in that layer group. This is because the tile is smaller, scaled down compared to the tail tile of the final layer group LG2T7 (see Figure 14A ). Moreover, core C3 is idle during the evaluation of LG0 in the second multi-kernel tile propagation process, as that layer group has only seven tiles. It is understood that, due to the overlap at the boundaries between tiles, dependencies between tiles, and the corresponding need for synchronization between cores, arise. In the absence of overlap (e.g., a series of convolutions with a 1x1x1 kernel), no synchronization is needed, and the cores can be allowed to free-run (without any complexity penalty). Generally speaking, for convolutions, the formula for determining the size of any overlap in the Y dimension is: overlap = max(0, (kernel height - 1) * dilation height - (stride height - 1)).

[0136] This approach will be described in more detail with reference to Figure 15 . In step 710, the partitioner unit 620 defines a plurality of layer groups and arranges them in tile groups. The arrangement in tile groups can be as described above with reference to Figures 10-11BThe latter is done. In step 712, input data for the first layer group is provided in the DDR 312. In step 714, the splitter unit 620 analyzes the input data for the first layer group. In particular, it checks whether the size of the input data in the Y dimension is above a threshold. If so, it determines (step 716) that the input data can be split into tiles along the Y dimension. In other words, if the input data has a size in the Y dimension that is large enough, it will be split into tiles. In response to the determination in step 716, the splitter unit splits the data into tiles (step 718) and assigns each tile to one of the available cores (step 719). If there are more tiles than cores, the splitter unit 620 will control the cores to perform several multi-core tile propagation processes. In general, the splitter unit tries to split the input data into tiles so that all cores are occupied. Thus, it can try (if possible) to split the input data into a number of tiles that is an integer multiple of the number of cores. The number of tiles (and the number of multi-core tile propagation processes) will be influenced by the capacity of the OCM 314.

[0137] In step 720, at least two cores evaluate the first layer group based on at least two respective tiles. That is, each core evaluates the first layer group based on one tile. The splitter unit is configured to identify the one-time portion and the non-one-time (overlapping) portion of each output tile. It instructs the cores to store these different portions in different places. Thus, the first core 350-0 writes the one-time portion of the first output tile to its local OCM 314a (step 722) and writes the overlapping portion of the first output tile to the shared OCM 314b (step 724). At the same time, the second core 350-1 writes the one-time portion of the second output tile to its local OCM 314a (step 722) and writes the overlapping portion of the second output tile to the shared OCM 314b (step 724). Only two instances of steps 720-724 are shown in the flowchart, which means that there are two cores; however, it will be understood that there are as many parallel instances of these steps as there are cores that are occupied. Note that steps 722 and 724 can also be performed in the opposite order or in parallel.

[0138] In step 728, any core that finishes its evaluation before the other cores waits until all cores have finished their evaluation of the first layer group before proceeding. The method proceeds to step 730. If there are further layer groups in the current tile group, each core continues to evaluate the next layer group based on its tile. When evaluating the second layer group and subsequent layer groups, each core reads the necessary one-time data from the previous layer group from its own local OCM 314a and reads the necessary overlapping data from the previous layer group from the shared OCM 314b.

[0139] The loop continues until all layer groups of the current tile group have been evaluated by each core based on the respective tile. When (at step 730) there are no further layer groups in the current tile group, the method proceeds to step 732, which checks whether there are further tiles to process. This represents the end of a multi-core tile propagation process. If there are further tiles to process, the cores will continue a second multi-core tile propagation process in order to evaluate all layer groups of the current tile group for the next group of tiles. The method continues to loop within the multi-core tile propagation process until all tiles have been processed. Then, when there are no more tiles to process in step 732, the evaluation of the tile group is complete and the method ends. The method can be repeated for subsequent tile groups as necessary.

[0140] It should be noted that in steps 714 and 716, the splitter unit 620 determines that the input data can be split into tiles based on the size of the input data, in particular in the Y dimension. The splitting into tiles depends on whether the tensor size is large enough. However, even for relatively small tensors, it can be advantageous to split the processing among multiple cores. According to a second aspect, a method for doing so is provided. Examples of this method will now be described with reference to Figure 12 、 Figure 16 and Figure 17 According to this example, the evaluation can be split among cores along one of the other dimensions. This can include splitting the evaluation of layer groups along the X dimension as one of the traversal dimensions. Likewise, it can include splitting the evaluation of layer groups across one of the non-traversal dimensions, in particular across the input channels C, or across the output channels F. Typically, when splitting the evaluation into a propagation process, the input data for a layer group is not split among the cores, but is stored in the shared OCM 314b and accessed by each of the cores 350 from there. This is different from the use of tiles in the first aspect (as exemplified by Figures 13-15 When splitting along the C dimension, each core evaluates layer groups for a respective channel (or group of channels) of the input data. When splitting along the F dimension, each core evaluates layer groups for a respective filter (or group of filters), producing a respective output channel or group of output channels. When splitting along the X dimension, each core evaluates layer groups for a respective column (or group of columns) of the input data.

[0141] Figure 16An example of splitting the evaluation of multiple groups of layers into multiple propagation processes, across multiple cores, evaluating one group of layers at a time is shown schematically. There are five groups of layers LG0-LG4. Group LG2 includes two layers; the other groups each include three layers. The input data for the first group of layers is stored in DDR 312. In this example, the first group of layers LG0 is split into three propagation processes to be evaluated by different cores. The split is along the X dimension. Each of the three cores C0-C2 evaluates a respective one of the three propagation processes. In each propagation process, the respective core processes a portion of the input data to produce a portion of the output data. In this example, it is assumed that there is at most one convolutional layer in each group of layers. Thus, there is no issue of overlapping data. Depending on the kernel size and stride, different cores can access overlapping portions of the input data, but they generate different portions of the output data. Unlike the case of Figure 13 , in this example, the data is not split into separate tiles. (In the current example, it would be difficult to do this in the X dimension, as the data is arranged row-first, and the data elements in the X dimension do not align with bursts of data in memory.) Instead, each core accesses the portion of the data that it needs from the shared OCM 314b. As already noted, each core generates a respective portion of the output data. When all of the cores have evaluated the first group of layers, the portions of the output data are combined in OCM 314.

[0142] The next two groups of layers LG1-LG2 are not split; thus, each is evaluated by one core in one propagation process. The subsequent group of layers LG3 is split across the output channels (F dimension) into six propagation processes. As there are more propagation processes than cores here, the cores C0-C2 first evaluate the group of layers for the first three propagation processes in the F dimension (labeled 0, 1, and 2 in the figure). Then, the cores C0-C2 evaluate the group of layers for the second set of three propagation processes (labeled 3, 4, and 5 in the figure) to complete the group of layers. The final group of layers LG4 is again processed by core C0 in a single propagation process. The outputs of the first four groups of layers LG0-LG3 are combined in OCM 314; the output of the final group of layers LG4 is written to DDR 312.

[0143] As shown by this example, the split is flexible, as different groups of layers can be split along different dimensions (e.g., compare LG0 and LG3). This is achieved by combining the full output of each group of layers in OCM 314, which makes the split of one group of layers independent of the split of other groups of layers. This is another difference from the tile-based split of Figure 13 , in which each core operates on a specific tile, and the correspondence between tiles is preserved when the groups of layers are evaluated depth-first. Figure 13

[0144] Reference will be made to Figure 17 ​The method is described in more detail. Steps similar to those in Figure 15 Steps similar to those in steps 710’ will be given the same reference numerals. In step 710’, the splitter unit 620 defines a plurality of layer groups. In step 712’, input data for a first layer group is provided in the DDR 312. In step 714’, the splitter unit 620 analyses the input data to check whether it can be split into tiles. In particular, the splitter unit checks whether the size of the input data in the Y dimension is above a threshold. In step 746, the splitter unit 620 determines that the size is not above the threshold - thus, the data is not suitable for tiling. In response to this determination, the splitter unit 620 splits the evaluation of the layer group into propagation processes (step 748) and assigns each propagation process to a core (step 749). In step 750, the cores evaluate the layer group of their respective propagation process to generate a respective portion of the output data. If it is determined in step 752 that there are further propagation processes to complete in the current layer group, the cores evaluate the layer group for the necessary further propagation processes (as in the example of layer group LG3 in Figure 16 When it is determined in step 752 that there are no further propagation processes to evaluate for the current layer group, the method proceeds to step 754 and the portions of output data from each of the propagation processes are combined to produce the output data for the layer group. In step 756, this output data is written to the OCM 314. In step 758, it is checked whether there is another layer group to evaluate. If so, the method proceeds again to step 748 to split the evaluation of the next layer group into propagation processes. Note that although only two instances of step 750 are shown in the flowchart, this implies that there are two cores; however, it will be appreciated that there are as many parallel instances of this step as there are cores occupied.

[0145] The output data for each layer group is written to the shared OCM 314b so that it is accessible to all of the cores to evaluate the next layer group. The manner in which the evaluation is split in step 748 and the manner in which the portions of output data are combined in step 754 will depend on the characteristics of the layers in the layer group and the dimension along which the evaluation is split. In some cases, the evaluation can be split by input channel C so that each core evaluates a layer group for a respective channel (or group of channels) of the input data. For example, if the layer group comprises a convolutional layer with one filter and each core processes a different input channel, this will mean that each core will apply one channel of the filter kernel to a respective channel of the input data. The respective output portions produced by the different cores will then need to be combined by element-wise summation to produce the output of the convolution. This can be done by the element-wise operation module of one of the cores 350 (if available) or by another processing module separate from the cores 350. The weights (and biases) for different channels of the kernel can be stored in the local OCM 314a of each respective core as each core only needs to access the weights for the channel (or channels) that it is responsible for evaluating.

[0146] If the layer group consists of element-wise operations, and each core processes a different input channel, then the corresponding output portions produced by different cores can be combined by concatenating them in the input channel dimension (P dimension). Note that because the step of combining the corresponding output portions consists of concatenating them, this can be achieved by each core simply writing its output portion to the appropriate location in the shared OCM 314b. In other words, the "combination" in step 754 and the "writing" in step 756 can be performed together, and can be performed by each core immediately following the "evaluation" step 750.

[0147] The parameters controlling the element-wise operation of each input channel can be segmented and stored in a separate local OCM 314a. Similarly, for other operations that can be segmented by input channels, the parameters controlling the processing of each channel can be segmented into separate local OCM 314a.

[0148] If a layer group comprises convolutional layers with multiple output channels (filters), and the evaluation is split across kernels according to the output channels (F dimension), then each kernel will be responsible for evaluating a given filter or filter group. The weights (and biases) of the relevant filters can be stored in the local OCM 314a of the corresponding kernel.

[0149] On the other hand, if the layer group includes convolutional layers and the evaluation is performed across the kernel along one of the traversal dimensions (e.g., the X dimension), the weights and biases are stored in a shared OCM 314b, where they are accessible to all kernels.

[0150] Some layer partitioning across multiple kernels may be difficult, inefficient, or impossible. For example, it should be noted that... Figure 16 Layers LG1, LG2, and LG4 in the code are not partitioned across cores at all—they are all executed by a single core (which may leave other cores idle if no other work is occupying them). For example, this approach can be used to process fully connected layers that include a filter.

[0151] Now we will understand, Figure 15 and Figure 17 The methods can be combined advantageously. Figure 15 This method can be used to handle large input data tensors (i.e., if the size exceeds a threshold in step 714 when analyzing the size of the input data tensor). And... Figure 17 This method can be used to process small input data tensors (if the size is below a threshold in step 714'). In other words, the threshold used in step 714 and the threshold used in step 714' can be the same, allowing the hardware implementation 600 to follow... Figure 15 The method or following Figure 17 The method.

[0152] The processing system can comprise a hardware implementation as described above, or can be configured to perform a method as described above. The processing system can be an artificial intelligence accelerator system, such as a neural network accelerator (NNA), or a graphics processing system / graphics processing unit (GPU).

[0153] Figure 18 A computer system in which a processing system according to the present disclosure can be implemented is shown. The computer system comprises a CPU 902, an NNA 904, a memory 906 and other devices 914, such as a display 916, a speaker 918 and a camera 919. The processing block 910 (corresponding to the hardware implementation 600) is implemented on the NNA 904. In other examples, the processing block 910 can be implemented on the CPU 902. The components of the computer system can communicate with each other over a communication bus 920. The storage 912 (corresponding to the DDR 312) is implemented as part of the memory 906.

[0154] Although Figure 18 An implementation of an artificial intelligence accelerator system is shown, but it will be appreciated that a similar block diagram can be drawn for a graphics processing system, for example, by replacing the CPU 902 or the NNA 904 with a GPU, or by adding a GPU as an additional unit. In this case, the processing block 910 can be implemented in the GPU.

[0155] Figure 12 The hardware implementation 600 of the processing system is shown as comprising a number of functional blocks. This is merely schematic and is not intended to define a strict division between the different logical elements of such entities. Each functional block can be provided in any suitable way. It will be appreciated that intermediate values formed by the hardware implementation described herein need not be physically generated by the hardware implementation at any point in time, and can merely represent logical values that conveniently describe the processing performed by the hardware implementation between its input and output terminals.

[0156] The hardware implementations described herein can be realized in hardware on an integrated circuit. The hardware implementations described herein can be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The term "module," "functionality," "component," "element" "unit," "block," and "logic" can be used generically herein to represent software, firmware, hardware, or any combination thereof. In the case of a software implementation, the module, functionality, component, element, unit, block, or logic represents program code that performs specified tasks when executed on a processor. The algorithms and methods described herein can be performed by one or more processors executing code as described above. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disc, flash memory, hard disk storage, and other storage devices that can use magnetic, optical, and other technologies to store instructions or other data and that can be accessed by a machine.

[0157] The terms computer program code and computer readable instructions as used herein refer to any kind of executable code, including code expressed in machine language, interpreted language, or scripting language, for execution by a processor. Executable code includes binary code, machine code, byte code, code that defines integrated circuits (such as a hardware description language or netlist), and code expressed in a programming language such as C, C++, C#, Java, Python, JavaScript, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when executed, processed, interpreted, compiled, run, in a virtual machine or other software environment, causes a processor of a computer system that supports the executable code to perform tasks specified by the code.

[0158] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or collection or part thereof, that has processing capability to enable it to perform instructions. A processor can be any kind of general- or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system can include one or more processors.

[0159] ​The present application also intends to encompass software, such as a hardware description language (HDL) software, defining configurations of hardware as described herein, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer readable storage medium can be provided on which a computer readable program code in the form of an integrated circuit definition data set is encoded, which when processed (i.e., run) in an integrated circuit manufacturing system configures the system to manufacture a hardware implementation configured to perform any of the methods described herein, or to manufacture a hardware implementation comprising any of the apparatus described herein. The integrated circuit definition data set can be, for example, an integrated circuit description.

[0160] Thus, a method of manufacturing a hardware implementation as described herein at an integrated circuit manufacturing system can be provided. Further, an integrated circuit definition data set can be provided, which when processed in an integrated circuit manufacturing system causes the method of manufacturing a hardware implementation to be performed.

[0161] The integrated circuit definition data set can be in the form of computer code, for example as a netlist, code for configuring a programmable chip, as a hardware description language defining hardware suitable for manufacture at any level of integration, including as register transfer level (RTL) code, as a high level circuit representation (such as Verilog or VHDL), and as a low level circuit representation (such as OASIS (RTM) and GDSII). Higher level representations of hardware logically defining hardware suitable for manufacture in an integrated circuit, such as RTL, can be processed on a computer system configured for generating a manufacturing definition of an integrated circuit in the context of a software environment comprising definitions of circuit elements and rules for combining these elements to generate a manufacturing definition of an integrated circuit defined by the representation. As is common for software executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) can be required to configure the computer system to generate a manufacturing definition of an integrated circuit, to execute code defining an integrated circuit to generate a manufacturing definition of said integrated circuit.

[0162] Reference will now be made to Figure 19 Examples of processing an integrated circuit definition data set at an integrated circuit manufacturing system to configure the system to manufacture a hardware implementation will now be described.

[0163] Figure 19An example of an integrated circuit (IC) fabrication system 1002 configured to fabricate hardware implementations as described in any of the examples herein is shown. In particular, the IC fabrication system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC fabrication system 1002 is configured to receive an IC definition data set (e.g., defining a hardware implementation as described in any of the examples herein), process the IC definition data set, and generate an IC from the IC definition data set (e.g., embodying a hardware implementation as described in any of the examples herein). The processing of the IC definition data set configures the IC fabrication system 1002 to fabricate an integrated circuit embodying a hardware implementation as described in any of the examples herein.

[0164] The layout processing system 1004 is configured to receive and process the IC definition data set to determine a circuit layout. Methods of determining a circuit layout from an IC definition data set are known in the art and can involve, for example, synthesizing RTL code to determine a gate level representation of the circuit to be generated, e.g., in terms of logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, a circuit layout can be determined from the gate level representation of the circuit. This can be done automatically or with user involvement in order to optimize the circuit layout. When the layout processing system 1004 has determined the circuit layout, it can output a circuit layout definition to the IC generation system 1006. The circuit layout definition can be, for example, a circuit layout description.

[0165] The IC generation system 1006 generates an IC from the circuit layout definition, as is known in the art. For example, the IC generation system 1006 can implement a semiconductor device fabrication process that generates an IC, which can involve a multi-step sequence of photolithographic and chemical processing steps during which an electronic circuit is gradually formed on a wafer made of semiconductor material. The circuit layout definition can be in the form of a mask that can be used in a photolithographic process to generate an IC from the circuit definition. Alternatively, the circuit layout definition provided to the IC generation system 1006 can be in the form of computer readable code that can be used by the IC generation system 1006 to form a suitable mask for generating an IC.

[0166] The different processes performed by the IC fabrication system 1002 can all be performed at one location, e.g., by one party. Alternatively, the IC fabrication system 1002 can be a distributed system, such that some processes can be performed at different locations, and can be performed by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesis of RTL code representing the IC definition data set, to form a gate level representation of the circuit to be generated; (ii) generation of a circuit layout based on the gate level representation; (iii) formation of a mask from the circuit layout; and (iv) fabrication of an integrated circuit using the mask.

[0167] In other examples, processing of an integrated circuit definition data set at an integrated circuit fabrication system can configure the system to fabricate a hardware implementation without processing the IC definition data set to determine a circuit layout. For example, an integrated circuit definition data set can define a configuration of a reconfigurable processor, e.g., an FPGA, and processing of the data set can configure the IC fabrication system to generate the reconfigurable processor with the defined configuration, e.g., by loading configuration data to the FPGA.

[0168] In some embodiments, an integrated circuit fabrication definition data set, when processed in an integrated circuit fabrication system, can cause the integrated circuit fabrication system to generate an apparatus as described herein. For example, an integrated circuit fabrication definition data set can define a circuit to be fabricated, and processing of the data set can cause the integrated circuit fabrication system to fabricate the circuit as described herein with reference to Figure 19 Configuration of an integrated circuit fabrication system in the manner described can result in an apparatus as described herein being fabricated.

[0169] In some examples, an integrated circuit definition data set can include software to be run on hardware defined at the data set, or in combination with hardware defined at the data set. In Figure 19 In the example shown, the IC generation system can be further configured by the integrated circuit definition data set to load firmware onto an integrated circuit when the integrated circuit is fabricated, in accordance with program code defined in the integrated circuit definition data set, or otherwise to provide the integrated circuit with program code for use with the integrated circuit.

[0170] Compared to known implementations, the concepts set forth in this application can result in performance improvements in the implementation of apparatuses, devices, modules, and / or systems (as well as in the methods implemented herein). Performance improvements can include one or more of improved computational performance, reduced latency, increased throughput, and / or reduced power consumption. During fabrication of such apparatuses, devices, modules, and systems (e.g., in integrated circuits), trade-offs can be made between performance improvements and physical implementation, improving the fabrication process. For example, a trade-off can be made between performance improvements and layout area, matching the performance of known implementations but using less silicon. This can be done, for example, by reusing functional blocks in serial fashion or sharing functional blocks among elements of the apparatuses, devices, modules, and / or systems. Conversely, the concepts set forth in this application that result in improvements in the physical implementation of apparatuses, devices, modules, and systems, such as reduced silicon area, can be traded off for performance improvements. This can be done, for example, by fabricating multiple instances of a module within a predefined area budget.

[0171] Applicant hereby expressly disclaims any right in the individual features of the application, independently of the combination of features, and explicitly not to append hereto any limitation implying the non-applicability of such individual features to the genre of applications disclosed herein or to the patentability of such individual features as set forth in any jurisdiction where such claim of disclaimer or disclaiming such patentability can be construed as invalid. The applicant reserves the right, in the Japanese translation of the specification as filed initially in English, to append a disclaimer to the translation of any individual feature of the application, independently of the combination of features, and explicitly not to append hereto any limitation implying the non-applicability of such individual features to the genre of applications disclosed herein or to the patentability of such individual features as set forth in any jurisdiction where such claim of disclaimer or disclaiming such patentability can be construed as invalid.

Claims

1. A method for implementing a deep neural network in multi-core hardware (600), the deep neural network comprising a plurality of layers, the multi-core hardware comprising at least a first core (350-0) and a second core (350-1), the method comprising: defining (710) a plurality of layer groups (LG) comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein output data of the first layer group comprises input data for the second layer group; providing (712) input data for the first layer group in one or more first memory devices (312), the input data comprising a multi-dimensional tensor comprising one or more traversal dimensions, the one or more traversal dimensions being dimensions traversed by a stride in at least one layer of the first layer group; segmenting (718) the input data for the first layer group into at least a first tile and a second tile along at least one of the traversal dimensions, each tile comprising a plurality of data elements in each of the one or more traversal dimensions; allocating (719) the first tile to the first core and the second tile to the second core; using the first core, evaluating (720) the first layer group based on the first tile to generate a first output tile of the first layer group; using the second core, evaluating (720) the first layer group based on the second tile to generate a second output tile of the first layer group, each of the first and second output tiles of the first layer group comprising a plurality of data elements in each of the one or more traversal dimensions; storing (722, 724) the first and second output tiles of the first layer group as input data for the second layer group in one or more second memory devices (314); using the first core, evaluating (720) the second layer group based on the first output tile of the first layer group to generate a first output tile of the second layer group; using the second core, evaluating (720) the second layer group based on the second output tile of the first layer group to generate a second output tile of the second layer group, each of the first and second output tiles of the second layer group comprising a plurality of data elements in each of the one or more traversal dimensions.

2. The method of claim 1, wherein: (A) the plurality of layer groups comprises a third layer group comprising one or more consecutive layers of the neural network, the method further comprising storing (722, 724) the first and second output tiles of the second layer group as input data for the third layer group in the one or more second memory devices; and / or (B) the method further comprising storing the first and second output tiles of a final layer group of the plurality of layer groups in the one or more first memory devices (312).

3. The method of any preceding claim, further comprising: analyzing (714) at least one characteristic of the input data for the first group of layers; and based on the analyzing, determining (716) that the input data can be tiled along the at least one of the traversal dimensions, wherein the tiling (718) of the input data for the first group of layers is performed in response to the determining, and optionally wherein the at least one characteristic comprises a size of the input data in the at least one traversal dimension, and wherein determining (716) that the input data can be tiled comprises determining that the size in the at least one traversal dimension is greater than a threshold size.

4. The method according to any of the preceding claims, wherein, the input data for the first group of layers is tiled into a first number of tiles, and the input data for the second group of layers is tiled into a second number of tiles, wherein the second number differs from the first number by at most one.

5. The method according to any one of the preceding claims, wherein, tiling (718) the input data for the first group of layers comprises tiling the input data for the first group of layers into the first tile, the second tile, and a third tile, the method further comprising: allocating (719) the third tile to the first core; and after evaluating (720) each group of layers based on the first tile using the first core, evaluating (720) each group of layers based on the third tile using the first core.

6. The method according to any one of the preceding claims, wherein, the one or more second memory devices comprise: a local memory (314a) for the first core that is accessible only by the first core; a local memory (314a) for the second core that is accessible only by the second core; and a shared memory (314b) that is accessible by the first core and the second core, the method further comprising: identifying a one-time portion of the first output tile of the first group of layers that is needed only to evaluate the second group of layers based on the first tile; and identifying an overlap portion of the first output tile of the first group of layers that is needed to evaluate the second group of layers based on the first tile and also needed to evaluate the second group of layers based on the second tile, the method comprising, when storing the first output tile of the first group of layers in the one or more second memory devices: writing (722) the one-time portion of the first output tile to the local memory of the first core; and writing (724) the overlap portion of the first output tile to the shared memory, and optionally wherein the first group of layers comprises a convolutional layer, the method further comprising storing weight values and / or bias values of the convolutional layer in the shared memory (314b).

7. The method of any preceding claim, comprising: waiting (728) for the first core to complete evaluation of the first group of layers on the first tile, and waiting (728) for the second core to complete evaluation of the first group of layers on the second tile, The second kernel is then used to evaluate (720) the second layer group for the second tile, and optionally All kernels are waited for (728) to complete evaluation of the first layer group before any kernel starts evaluating (720) the second layer group.

8. The method of any of the preceding claims, wherein, The input data for the first layer group includes a first traversal dimension and a second traversal dimension, wherein the input data is stored in the one or more first memory devices sorted first according to a first dimension, and wherein the input data is split along a second dimension.

9. A hardware implementation (600) of a deep neural network comprising a plurality of layers arranged in a plurality of layer groups (LG), the plurality of layer groups comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein, The output data for the first layer group includes input data for the second layer group, the hardware implementation comprising: one or more first memory devices (312) configured to store input data for the first layer group, the input data comprising a multi-dimensional tensor including one or more traversal dimensions, the one or more traversal dimensions being dimensions traversed by a stride in at least one layer of the first layer group; one or more second memory devices (314) configured to store at least the output data for the first layer group; a plurality of kernels (350) including at least a first kernel (350-0) and a second kernel (350-1); and a splitter unit (620), wherein the splitter unit (620) is configured to split (718) the input data for the first layer group into at least a first tile and a second tile along at least one of the traversal dimensions, each tile including a plurality of data elements in each of the one or more traversal dimensions, wherein the splitter unit (620) is configured to assign (719) the first tile to the first kernel (350-0) and the second tile to the second kernel (350-1), wherein the first kernel is configured to evaluate (720) the first layer group based on the first tile to generate a first output tile for the first layer group, wherein the second kernel is configured to evaluate (720) the first layer group based on the second tile to generate a second output tile for the first layer group, each of the first output tile and the second output tile for the first layer group including a plurality of data elements in each of the one or more traversal dimensions, wherein the first kernel and the second kernel are configured to store (722, 724) the first output tile and the second output tile for the first layer group as input data for the second layer group in the one or more second memory devices (314), wherein the first kernel is configured to evaluate (720) the second layer group based on the first output tile for the first layer group to generate a first output tile for the second layer group, and wherein the second kernel is configured to evaluate (720) the second layer group based on the second output tile for the first layer group to generate a second output tile for the second layer group, each of the first output tile and the second output tile for the second layer group including a plurality of data elements in each of the one or more traversal dimensions.

10. The hardware implementation of claim 9, wherein, The first core and the second core are configured to store the first output tile and the second output tile of a final layer group of a defined layer group in the one or more first memory devices (312).

11. The hardware implementation of claim 9 or claim 10, wherein, The splitter unit is further configured to: analyze (714) the input data for the first layer group; and determine (716) that the input data can be split into tiles along the at least one of the traversal dimensions, wherein the splitter unit is configured to split (718) the input data for the first layer group into the first tile and the second tile in response to the determination.

12. The hardware implementation of any of claims 9 to 11, wherein, The splitter unit is configured to split (718) the input data for the first layer group into the first tile, the second tile, and a third tile, and the splitter unit is configured to assign (719) the third tile to the first core; and The first core is configured to evaluate (720) each layer group based on the third tile after having completed evaluating each layer group based on the first tile.

13. The hardware implementation of any of claims 9 to 12, wherein, The one or more second memory devices comprise: a local memory (314a) for the first core, which is only accessible by the first core; a local memory (314a) for the second core, which is only accessible by the second core; and a shared memory (314b), which is accessible by the first core and the second core, wherein the splitter is configured to: identify a one-off portion of the first output tile of the first layer group, which is only needed to evaluate the second layer group based on the first tile; and identify an overlap portion of the first output tile of the first layer group, which is needed to evaluate the second layer group based on the first tile, and also needed to evaluate the second layer group based on the second tile, and the first core is configured to, when storing the first output tile of the first layer group in the one or more second memory devices: write (722) the one-off portion to its local memory; and write (724) the overlap portion to the shared memory.

14. The hardware implementation of any one of claims 9 to 13, wherein, The second core is configured to wait (728) for the first core to complete evaluation of the first layer group based on the first tile, and wait (728) for the second core to complete evaluation of the first layer group based on the second tile, then evaluate (720) the second layer group based on the second tile, and optionally wherein each core is configured to wait (728) for all cores to complete evaluation of the first layer group before evaluating the second layer group.

15. An artificial intelligence accelerator system comprising a hardware implementation according to any of claims 9 to 14 or configured to perform a method according to any of claims 1 to 8.

16. A method of manufacturing an artificial intelligence accelerator system according to claim 15 using an integrated circuit manufacturing system, the method comprising: processing the computer readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description embodying an integrated circuit of the artificial intelligence accelerator system; and fabricating the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system.

17. A computer readable storage medium having stored thereon computer readable code configured such that, on execution of the code, the method of any one of claims 1 to 8 is performed.

18. A computer readable storage medium having stored thereon an integrated circuit definition data set, the integrated circuit definition data set, when processed in an integrated circuit fabrication system, configures the integrated circuit fabrication system to fabricate the artificial intelligence accelerator system of claim 15.

19. A computer readable storage medium having stored thereon a computer readable description of the artificial intelligence accelerator system of claim 15, the computer readable description, when processed in an integrated circuit fabrication system, causes the integrated circuit fabrication system to: process the computer readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description embodying an integrated circuit of the artificial intelligence accelerator system; and fabricate the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system.

20. An integrated circuit fabrication system comprising: a computer readable storage medium having stored thereon a computer readable description of the artificial intelligence accelerator system of claim 15; a layout processing system configured to process the computer readable description to generate a circuit layout description embodying an integrated circuit of the artificial intelligence accelerator system; and an integrated circuit generation system configured to fabricate the artificial intelligence accelerator system from the circuit layout description. ​ ​ ​ ​ ​

Images