Implementation of Neural Networks on Multi-core Hardware

Abstract

The present invention discloses an implementation of a neural network in multi-core hardware. A multi-core hardware implementation of a deep neural network is disclosed. Multiple layers of the network are arranged in multiple layer groups. Input data to the network includes a multidimensional tensor, which includes one or more traversed dimensions and one or more non-traversed dimensions, where the traversed dimensions are dimensions traversed by strides in at least one layer of a first layer group. If the size of the input data in the first dimension is greater than a threshold, the hardware implementation is configured to split the input data for the first layer group into at least a first tile and a second tile along the first dimension. If the size of the input data in the first dimension is not greater than the threshold, the hardware implementation is configured to split the evaluation of the first layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension.

Description

Technical Field

[0001] The present invention relates to an implementation of a neural network in multi-core hardware. Background Art

[0002] A deep neural network (DNN) is a type of artificial neural network that can be used in machine learning applications. In particular, DNNs can be used in signal processing applications, including image processing and computer vision applications.

[0003] DNNs have been implemented in applications where power resources are not a significant factor. Nevertheless, DNNs have applications in many different technology areas where the hardware resources used to implement the DNNs are limited in power consumption, processing power, or silicon area. Therefore, there is a need to implement hardware that is configured to implement a DNN (or at least a portion thereof) in an efficient manner, such as in a manner that requires less silicon area or less processing power to operate. In addition, DNNs can be configured in many different ways for a variety of different applications. Therefore, there is also a need for hardware for implementing DNNs that has the flexibility to support a variety of DNN configurations. As DNNs become more complex in structure and need to process larger amounts of data in shorter time periods (e.g., for inference in real-time applications), there is an increasing need for hardware with greater processing power while maintaining or improving area efficiency and power efficiency. Summary of the Invention

[0004] This summary is provided to introduce a series of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] A multi-core hardware implementation of a deep neural network is disclosed. Multiple layers of the network are arranged in multiple layer groups. Input data to the network comprises a multidimensional tensor comprising one or more traversed dimensions and one or more non-traversed dimensions, the traversed dimensions being dimensions traversed by strides in at least one layer of a first layer group. If the size of the input data in the first dimension is greater than a threshold, the hardware implementation is configured to split the input data for the first layer group into at least a first tile and a second tile along the first dimension. If the size of the input data in the first dimension is not greater than the threshold, the hardware implementation is configured to split evaluation of the first layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension.

[0006] According to one aspect, a method for implementing a deep neural network in multi-core hardware is provided.

[0007] When the size is greater than a threshold, layer groups of a neural network can be evaluated in a depth-first manner by partitioning the data into tiles whose evaluation is distributed across multiple cores. Here, depth-first means that a given core evaluates multiple consecutive layer groups based on a given tile before continuing with any other evaluation. For example, before the first core evaluates any other tiles or layer groups, the first core evaluates the first layer group based on the first tile, and then evaluates the second layer group based on the first output tile of the first layer group. The second core acts accordingly. Each output tile of each layer group serves as the input tile of the next layer group, until the last layer group. Partitioning the data into tiles along one or more of the traversal dimensions enables layer groups to be processed in a depth-first manner through layer fusion, while reducing the size requirements of one or more second memory devices. This size reduction is possible because the one or more second memory devices do not need to store the entire tensor output by each layer group. Instead, they can store a subset of the tensor corresponding to that tile.

[0008] When the size is not greater than a threshold, the layer groups of the neural network can be evaluated by splitting the computation into a propagation process whose evaluation is distributed across multiple cores. Splitting the evaluation into propagation processes along ergodic or non-ergodic dimensions can enable layer groups to be processed in parallel one after another.

[0009] There are synergies between these two approaches. If the tensor is large, it can be split into tiles along at least one traversal dimension and processed depth-first in these tiles. This can reduce the amount of data that needs to be stored in the one or more secondary memory devices, because it may not be necessary to store the entire intermediate tensor in them at the same time. On the other hand, if the tensor is small, it can be split into multiple propagation passes. In this case, even if it is necessary to store the entire intermediate tensor in one or more secondary memory devices, this may not be a limiting factor because the tensor is small to begin with. In other words, whether the input tensor is small or large, the present approach can allow for efficient utilization of parallel hardware.

[0010] Evaluating a layer group means evaluating each layer of the layer group. A layer group can be defined so that all layers of a single layer group can be evaluated in a single hardware propagation process. For example, all layers of a single layer group can be evaluated in the hardware pipeline of a given kernel without writing any data to the one or more second memory devices or the one or more first memory devices. In some cases, a layer group can consist of a single layer of a neural network. In other cases, a layer group can consist of multiple layers. For example, a layer group can include any one or any two or more of the following: a single convolutional layer, a single pooling layer, a single activation layer, a single normalization layer, and a single element-wise operation layer. In some cases, each of the first layer group and the second layer group may include a convolutional layer. Optionally, a layer group may include at most one convolutional layer.

[0011] The cost of accessing the one or more second memory devices may be less than the cost of accessing the one or more first memory devices. The cost may be in terms of time or power consumption. For example, the one or more first memory devices may be provided as off-chip memory, and the one or more second memory devices may be provided as on-chip memory.

[0012] The traversal dimensions may include an X dimension (column) and a Y dimension (row), and an optional batch dimension B. The input data may include one or more additional non-traversal dimensions, such as a channel dimension C. The input data may be processed by a plurality of filters to produce a corresponding number of output channels F.

[0013] Before starting the evaluation of a layer group, the step of splitting the input data (and / or splitting the evaluation) can be pre-planned. In other words, the split can be planned before the training phase or the inference phase of the neural network begins. The split can be pre-planned based on the parameters of the layers in each layer group and the tensor sizes at the input and output of each layer.

[0014] When the size is greater than a threshold, multiple layer groups can form a tile group. A tile group is a set of layer groups that are processed together on a per-tile depth-first basis. Tile groups begin with an initial layer group and end with a final layer group. The output tiles of the final layer group in the tile group can be stored in one or more first memory devices.

[0015] The deep neural network can be a convolutional neural network.

[0016] The size of tiles can vary within and between layer groups. In particular, the size of the header and tail tiles of the input data for a layer group can be different from the other tiles of that layer group. Due to data dependencies, the header tile in an earlier layer group of a tile group may be larger than the header tile in a later layer group of the tile group. For a given layer group, other tiles (i.e., those between the header and tail tiles) can have a consistent size. These combined factors may mean that tail tiles tend to shrink towards earlier layer groups.

[0017] The header tile of the initial layer group may be larger than all other tiles in that layer group. This can be caused by data dependencies causing the header tile to "grow" backwards through the layer group.

[0018] The tile size can be chosen so that the size of the header tile in the final layer group is smaller than some or all other tiles in the final layer group. This can help balance the load between different cores by slowing down the "growth" of header tiles as they progress backward through the layer groups. If the header tile in any given layer group is too large, that tile can delay overall evaluation (see also the note below about synchronization between cores). Reducing the size of the initial tile in the final layer group reduces the size of all layer groups (because of data dependencies), helping to avoid situations where the initial tile is much larger (and takes much longer to process) than other tiles.

[0019] The method may also include: if the size is not greater than the threshold, splitting the evaluation of the second layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension; assigning the first propagation process of the second layer group to the first core, and assigning the second propagation process of the second layer group to the second core; using the first core, evaluating the second layer group for the first propagation process to generate a first part of the output data; and using the second core, evaluating the second layer group for the second propagation process to generate a second part of the output data, each of the first part of the output data and the second part of the output data including multiple data elements.

[0020] In other words, if the tensor is small, the second layer group can be split into multiple propagation passes across multiple cores, just like the first layer group. The first and second layer groups can be split along different dimensions. This approach is flexible because different layer groups do not need to be split along the same dimension. Given the capabilities of the hardware, the split can be chosen based on what is most efficient for the layer group in question.

[0021] Each of the first and second output tiles of the first layer group may include multiple data elements in each of the one or more traversal dimensions. Each of the first and second output tiles of the second layer group may include multiple data elements in each of the one or more traversal dimensions. Each of the first and second portions of output data may include data elements in each of the one or more traversal dimensions.

[0022] The method may also include storing output data of a final layer group in the plurality of layer groups in the one or more first memory devices. The method may also include storing the first output tile and the second output tile of the final layer group in the plurality of layer groups in the one or more first memory devices if the size is greater than a threshold. In this manner, intermediate data generated when evaluating the layer groups is stored in the one or more second memory devices, and output data generated by evaluating the final layer group is stored in the one or more first memory devices.

[0023] If the size is greater than the threshold, segmenting the input data for the first layer group may include segmenting it into a first tile, a second tile, and a third tile. The method may also include: after evaluating each layer group based on the first tile using the first kernel, evaluating each layer group based on the third tile using the first kernel.

[0024] This can be done when the layer group is divided into a number of tiles that is greater than the number of cores in the multi-core hardware. The number of tiles can be selected so that it is an integer multiple of the number of cores (for at least one of the layer groups). The evaluation of all layer groups based on the tile groups of the first and second tiles is called a first multi-core tile propagation process. Then, the evaluation of all layer groups based on the third tile represents (at least a portion of) a second multi-core tile propagation process. It should be noted that if the third tile is the last tile in the plurality of tiles, the second core may be idle during the second multi-core tile propagation process.

[0025] The one or more second memory devices may include: a local memory of the first core, which is accessible only to the first core; a local memory of the second core, which is accessible only to the second core; and a shared memory, which is accessible to the first core and the second core. The method may also include, if the size is greater than a threshold, identifying a disposable portion of the first output tile of the first layer group, the disposable portion being required only for evaluating the second layer group based on the first tile; and identifying an overlapping portion of the first output tile of the first layer group, the overlapping portion being required for evaluating the second layer group based on the first tile and also required for evaluating the second layer group based on the second tile. The method may include, when storing the first output tile of the first layer group in the one or more second memory devices: writing the disposable portion of the first output tile to the local memory of the first core; and writing the overlapping portion of the first output tile to the shared memory.

[0026] As discussed above, the third tile may be assigned to the first core. The method may further include: identifying a disposable portion of the second output tile of the first layer group that is only required for evaluating the second layer group based on the second tile; and identifying an overlapping portion of the second output tile of the first layer group that is required for evaluating the second layer group based on the second tile and is also required for evaluating the second layer group based on the third tile. The method may include, when storing the second output tile of the first layer group in the one or more second memory devices: writing the disposable portion of the second output tile to the local memory of the first core; and writing the overlapping portion of the second output tile to the shared memory.

[0027] The overlapping portion of the second output tile can be retained in the shared memory longer than the overlapping portion of the first output tile. In particular, the overlapping portions can be retained in the shared memory for as long as (and only if) they are needed. The overlapping portion of the first output tile can be retained until it is used by the second core when evaluating the second layer group based on the second output tile. The overlapping portion of the second output tile can be retained until it is used by the first core when evaluating the second layer group based on the third output tile.

[0028] The first layer group may include a convolutional layer, and the method may further include storing weight values ​​and / or bias values ​​of the convolutional layer in a shared memory. This may allow each of the kernels to access coefficient (weight and / or bias) data to evaluate the convolutional layer of its corresponding one or more tiles.

[0029] The method may include, if the size is greater than a threshold, waiting for the first core to complete evaluation of the first layer group for the first tile and waiting for the second core to complete evaluation of the first layer group for the second tile, and then evaluating the second layer group for the second tile using the second core.

[0030] This can represent a minimum synchronization condition to satisfy data dependency requirements, in particular when the first output tile includes overlapping data. It reflects the fact that the second kernel will need overlapping data for (i) the first output tile of the first layer group, and (ii) the second output tile of the first layer group in order to evaluate the second layer group based on the second tile. The presence of overlapping data will depend on the structure of the network. For example, if the kernel size is larger than the stride (ignoring the dilation size), the convolution layer will generate overlapping data; if the stride is larger than 1, the convolution transpose will generate overlapping data; and if the kernel size or stride is larger than 1, the pooling layer will generate overlapping data.

[0031] Optionally, the method may include waiting for all cores to complete evaluation of the first tier group before any core begins evaluating the second tier group.

[0032] In principle, the minimum synchronization condition allows each core to continue its depth-wise processing of a tile as long as the necessary overlapping data from the previous tile is available. Allowing cores to "free-run" in this way can result in some cores significantly pulling ahead of others in their processing. While this may, in principle, be a good way to fully exploit the cores' potential, in practice, the inventors have recognized that this often does not result in an overall gain in performance. After a certain number of layer groups ("tile groups") have been processed depth-wise, the output data can be written to one or more first memory devices. A tile group can include a subset of the layers of a neural network, or it can include all layers of a neural network. A tile group is not complete until all constituent tiles have completed. Therefore, the hardware is inevitably still limited by the slowest path for evaluating operations in parallel across the tile. At the same time, there can be significant management overhead to support free-running. Therefore, the hardware implementation can be simplified by synchronizing all cores after each layer group without slowing down processing. On the other hand, in some applications (e.g., speech processing), it may be advantageous to have some output data available as early as possible. In such cases, the reduced latency in producing the first output that can be achieved through free-running may justify the increased management overhead.

[0033] The one or more traversal dimensions may further include a second dimension, wherein the input data is stored in the one or more first memory devices sorted first according to the second dimension. For example, the input data may be stored row-first, in other words, sorted according to the X dimension. In this case, the input data may be partitioned into tiles along the Y dimension. Alternatively, the input data may be stored column-first (sorted first according to the Y dimension), and it may be partitioned into tiles along the X dimension.

[0034] Splitting in this way can be advantageous when the input data elements are tightly packed into memory and there is no one-to-one correspondence between the data elements and the burst sizes used to read and write memory. It allows the data to be split by reading a set of contiguous bursts from memory for each tile. Similarly, when reassembling the output tiles of the final layer group, it is possible to write each output tile with a single set of contiguous bursts.

[0035] In some embodiments, the input data may be split along only the first dimension, whereby the tiles have the actual size of the original tensor in the second dimension.

[0036] If the size of the input data in the first dimension is not greater than a threshold size, the evaluation may be split along a second or another dimension.

[0037] If the size is greater than the threshold, partitioning the evaluation of the first layer group may include partitioning it into a first propagation process, a second propagation process, and a third propagation process. The method may also include, after evaluating the first layer group for the first propagation process using the first core, evaluating the first layer group for the third propagation process using the first core. This may be done when the layer group is partitioned into a greater number of propagation processes than the number of cores in the multi-core hardware.

[0038] The one or more second memory devices may include: a local memory of the first core that is accessible only to the first core; a local memory of the second core that is accessible only to the second core; and a shared memory that is accessible to both the first core and the second core. The method may include: if the size is not greater than the threshold, storing the first portion of the output data and the second portion of the output data in the shared memory.

[0039] The first layer group may include a convolutional layer, and the method may further include: storing weight values ​​and / or bias values ​​of the convolutional layer in a shared memory if evaluated along one of the splits in the traversal dimension; or splitting the weight values ​​and / or bias values ​​and storing them in a local memory if evaluated along one of the splits in the non-traversal dimension.

[0040] When the evaluation is split along one of the traversal dimensions, the same weights / biases need to be applied to each part of the data, and therefore each core needs to have access to all the same weights / biases. On the other hand, if the evaluation is split based on the input channels or output channels of the layer group, each core may process one or more input channels or output channels, while other cores process one or more other input channels or output channels. Therefore, the weights / biases associated with a given input channel or output channel can be stored locally to the core that needs to use them.

[0041] The evaluation may be split according to input channels, wherein the first layer group does not include a convolutional layer, and wherein the method includes splitting parameters of the first layer group and storing them in a local memory.

[0042] Parameters may include parameters that control pooling, normalization, or element-wise operations. For example, parameters may include stride and / or window size for corresponding input channels.

[0043] Hardware implementation of deep neural networks is also provided.

[0044] Each core can include multiple processing elements configured to process input data to evaluate a layer of a neural network. The cores can be configured to operate in parallel.

[0045] The one or more second memory devices may optionally include: a local memory of the first core, which is accessible only to the first core; a local memory of the second core, which is accessible only to the second core; and a shared memory, which is accessible to the first core and the second core. The partitioner may be configured to, if the size in the first dimension is greater than a threshold, then: identify a disposable portion of the first output tile of the first layer group, which disposable portion is only required to evaluate the second layer group based on the first tile; and identify an overlapping portion of the first output tile of the first layer group, which overlapping portion is required to evaluate the second layer group based on the first tile and is also required to evaluate the second layer group based on the second tile. The first core may be configured to, when storing the first output tile of the first layer group in the one or more second memory devices: write the disposable portion to its local memory; and write the overlapping portion to the shared memory.

[0046] When evaluating the second layer group based on the second tile, the second core may be configured to retrieve the overlapping portion from the shared memory.

[0047] Storing the overlap data so that it is accessible to the second core may avoid the need to recompute the overlap data when evaluating the second layer group based on the second tile.

[0048] Alternatively, the need to store overlapping data can be avoided by computing the overlapping data in both the first kernel and the second kernel. In other words, all data required by a kernel for subsequent layer groups can be computed by that kernel. In this case, the first output tile of the first layer group and the second output tile of the first layer group will contain a common portion (corresponding to the overlapping portion). This approach results in some duplication of computations between kernels operating in parallel; however, it has the advantage that no kernel needs to wait for any other kernel at the end of the layer group - that is, each kernel has complete freedom to operate freely, independent of the computations being performed by any other kernel. The first output tile of the first layer group can be stored entirely in the local memory of the first kernel; and the second output tile of the first layer group can be stored entirely in the local memory of the second kernel.

[0049] The second core can be configured to, if the size in the first dimension is greater than a threshold, wait for the first core to complete evaluation of the first layer group based on the first tile, and wait for the second core to complete evaluation of the first layer group based on the second tile, and then evaluate the second layer group based on the second tile. Alternatively or in addition, each core can be configured to wait for all cores to complete evaluation of the first layer group before evaluating the second layer group.

[0050] The splitter unit may be configured to split the evaluation of the first layer group into a first propagation process, a second propagation process, and a third propagation process if the size in the first dimension is not greater than a threshold, and assign the third propagation process to the third core. The first core may be configured to evaluate the first layer group for the third propagation process after it has completed evaluating the first layer group for the first propagation process.

[0051] Also provided is an artificial intelligence accelerator system comprising a hardware implementation as outlined above. Also provided is an artificial intelligence accelerator system configured to perform the method as outlined above. Such an artificial intelligence accelerator system may be embodied in hardware on an integrated circuit.

[0052] A method for manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system is also provided.

[0053] A method for manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacturing the artificial intelligence accelerator system based on the circuit layout description using the integrated circuit generation system.

[0054] A computer readable code is also provided, the computer readable code being configured to cause the method to be performed when the code is executed. A computer readable storage medium having the computer readable code encoded thereon is also provided.

[0055] An integrated circuit definition dataset is also provided that, when processed in the integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture an artificial intelligence accelerator system.

[0056] Also provided is a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an artificial intelligence accelerator system, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the artificial intelligence accelerator system.

[0057] Also provided is a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an artificial intelligence accelerator system, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process the computer-readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture the artificial intelligence accelerator system based on the circuit layout description using an integrated circuit generation system.

[0058] An integrated circuit manufacturing system configured to manufacture an artificial intelligence accelerator system is also provided.

[0059] Also provided is an integrated circuit manufacturing system comprising: a non-transitory computer-readable storage medium having stored thereon a computer-readable description of an artificial intelligence accelerator system; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and an integrated circuit generation system configured to manufacture the artificial intelligence accelerator system based on the circuit layout description.

[0060] The layout processing system may be configured to determine location information for logic components of a circuit derived from the integrated circuit description in order to generate a circuit layout description of an integrated circuit embodying an artificial intelligence accelerator system.

[0061] A computer program code for executing any of the methods described herein may be provided. A non-transitory computer readable storage medium may be provided having computer readable instructions stored thereon, which, when executed at a computer system, cause the computer system to execute any of the methods described herein.

[0062] As will be apparent to those skilled in the art, the features described above may be combined as appropriate and with any aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0064] Figure 1 is a schematic diagram of an exemplary deep neural network (DNN);

[0065] Figure 2 is a schematic diagram of exemplary data in DNN;

[0066] Figure 3 is a block diagram of an exemplary hardware implementation of a convolutional layer of a DNN;

[0067] Figure 4 yes Figure 3 A block diagram of an exemplary convolution engine;

[0068] Figure 5 is a schematic diagram illustrating data flow according to an exemplary hardware implementation;

[0069] Figure 6 shows the data dependencies within multiple layers of the DNN for the first chunk of data;

[0070] Figure 7 shows for the second block of data, in Figure 6Data dependencies within multiple layers;

[0071] Figure 8 It is a conceptual map of overlapping data concepts;

[0072] Figure 9 schematically illustrates an exemplary allocation of memory associated with a hardware implementation of a DNN;

[0073] Figure 10 A DNN with four layers and no branches is shown;

[0074] Figure 11A An exemplary grouping of layers of a DNN with four layers having branches is shown;

[0075] Figure 11B Shown for Figure 11A An exemplary grouping of alternative DNNs;

[0076] Figure 12 is a block diagram of a multi-core hardware implementation according to an example;

[0077] Figure 13 According to the first aspect, when input data is divided into tiles and these tiles are processed depth-first, Figure 12 An example of execution order in multi-core hardware;

[0078] FIG. 14A to FIG. 14B shows synchronization between cores in another example where input data is split into tiles and processed depth-first;

[0079] Figure 15 is a flow chart illustrating a method for implementing a convolutional neural network in multi-core hardware when input data is segmented into tiles;

[0080] Figure 16 An example of the execution order when splitting the evaluation of a layer into a propagation process according to the second aspect is shown;

[0081] Figure 17 is a flow chart illustrating a method for implementing a convolutional neural network in multi-core hardware when splitting the evaluation of a layer into a propagation process;

[0082] Figure 18 A computer system is shown in which a graphics processing system is implemented; and

[0083] Figure 19 A system for fabricating an integrated circuit embodying a graphics processing system is shown.

[0084] The accompanying drawings show various examples. It will be understood by those skilled in the art that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the accompanying drawings represent one example of boundaries. In some examples, it may be that an element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0085] The following description is given by way of example only to enable those skilled in the art to make and use the present invention. The present invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art. Embodiments will now be described by way of example only.

[0086] A deep neural network (DNN) is an artificial neural network that includes multiple interconnected layers, which enable the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An exemplary DNN 100 is shown that includes a plurality of layers 102-1, 102-2, 102-3. Each layer 102-1, 102-2, 102-3 receives input data and processes the input data according to the layer to produce output data. The output data is either provided to the layer as input data for another layer or output as the final output data of the DNN. For example, in Figure 1 In the DNN 100 of FIG. 1 , the first layer 102-1 receives raw input data 104 of the DNN 100 and processes the input data according to the first layer 102-1 to generate output data. The output data of the first layer 102-1 becomes the input data of the second layer 102-2, which processes the input data according to the second layer 102-2 to generate output data. The output data of the second layer 102-2 becomes the input data of the third layer 102-3, which processes the input data according to the third layer 102-3 to generate output data. The output data of the third layer 102-3 is output as the output data 106 of the DNN.

[0087] The processing performed on the input data of a layer depends on the type of layer. For example, each layer of a DNN can be one of a variety of different types. Exemplary DNN layer types include, but are not limited to, convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It will be apparent to those skilled in the art that these are exemplary DNN layer types and that this is not an exhaustive list, and that other DNN layer types may exist.

[0088] A convolutional layer is configured to convolve the input data using the weights associated with the layer. Specifically, each convolutional layer is associated with a number of weights w1…w iIn association, the plurality of weights may also be referred to as filter weights or coefficients. The weights may be grouped to form or define one or more filters or kernels.

[0089] refer to Figure 2 , which shows an example of the format of data 200 used in DNN. Figure 2 As can be seen in , the data 200 used in the DNN can be arranged into p data planes, where each plane has a size of x×y. The DNN may include one or more convolutional layers, where each convolutional layer is associated with a plurality of filters, each filter including a plurality of weights. Each filter has a size of m×n×p (i.e., each filter includes a set of m×n×p weights w) and is applied to the input data according to a convolution operation across a number of steps (called strides) s and t, as shown in FIG. Figure 2 As shown in Figure 1. The group of data elements to which the filter is applied at each stride is called a window. Each filter produces an output plane. The number of filters and the number of weights per filter can vary from one convolutional layer to another. Convolutional neural networks (CNNs) are an effective method for image recognition and classification. They are a special type of DNN and typically include multiple convolutional layers.

[0090] Before describing the multi-core hardware implementation of the subject matter of the present invention, a single-core example will be described. A (single-core) hardware implementation of a convolutional layer may include a hardware module or block (which may be referred to herein as a "processing element") that is configured to calculate the sum of products between the weights forming all or a portion of a filter and the input data values ​​forming all or a portion of a window (which may be referred to as a filter window calculation). Since a large number of such filter window calculations are typically performed, some hardware implementations may include multiple such processing elements so that more than one filter window calculation can be performed in parallel. Preparing each processing element to perform a filter window calculation involves reading the appropriate input data and weights for each filter window calculation from one or more first memory devices and providing them to one of the processing elements. Typically, a large amount of data needs to be transferred from the memory to the processing element. If this operation cannot be performed efficiently, high memory bandwidth requirements and high power consumption for providing the input data and weights to the processing element may result. This is particularly true when the memory is "off-chip" memory, that is, when it is implemented in an integrated circuit or semiconductor die different from the processing element.

[0091] Thus, described herein are hardware implementations of neural networks and methods for processing data in such hardware implementations, wherein input data and weights are provided to processing elements in an efficient manner to reduce the memory bandwidth required and the power consumed to provide the input data and weights to the processing elements. In particular, blocks of data (also referred to as "tiles") can be processed in a depthwise manner, passing through multiple layers of a neural network. Intermediate outputs of layers within the multiple layers can be stored in one or more second memory devices. The size of the one or more second memory devices can be smaller than the size of the one or more first memory devices used to store data in bulk, but can be accessed in a faster and / or more energy efficient manner. In one example, at least one of the one or more second memory devices is provided on-chip (i.e., in the same integrated circuit or semiconductor die as the processing element).

[0092] Now refer to Figure 3 , which shows an exemplary single-core hardware implementation 300 of a convolutional layer of a DNN. The hardware implementation 300 includes multiple convolution engines 302, multiple accumulators 304, an accumulation buffer 306, a coefficient buffer 308, and an input buffer 310. Each convolution engine 302, along with its corresponding accumulator 304 and its share of the resources of the accumulation buffer 306, represents a processing element 318. The hardware implementation also includes double data rate (DDR) SDRAM 312 as off-chip memory; on-chip memory (OCM) 314; and an output buffer 316.

[0093] Each convolution engine 302 includes hardware logic configured to receive a set of weights (e.g., {w1 ... w8}) representing all or a portion of a filter and a set of input data values ​​(e.g., {d1 ... d8}) representing all or a portion of a window and perform a multiply-accumulate calculation on the received weights and input data values, such as Figure 4 In some examples, such as Figure 4 As shown, each convolution engine 302 may include a plurality of multipliers 402, each of which is configured to convert the weight (w i ) and the corresponding input data value (d i ) are multiplied to produce a multiplication output value. The multiplier 402 is followed by a plurality of adders 404 forming an adder tree to calculate the sum of the multiplication outputs. Figure 4 In the example of FIG, convolution engine 302 includes eight multipliers 402, but in other examples, there may be more or fewer multipliers. For example, in some cases, there may be 128 multipliers. In general, if there are Z multipliers 402, then the adder tree includes Z-1 adders 404.

[0094] In some cases, the multiply-accumulate calculation is pipelined. For example, the multiplier 402 and the adder 404 can be divided into multiple pipeline stages, with register stages (not shown) before the first pipeline stage and between each pair of pipeline stages. For example, the multiplier can form the first pipeline stage, and the adder can be divided into layers, where subsequent pipeline stages include one or more adder layers.

[0095] Figure 3 The exemplary hardware implementation 300 of includes four convolution engines 302; however, it will be apparent to those skilled in the art that the methods and principles described herein are applicable to hardware implementations having any number of convolution engines.

[0096] Because the convolution engine may go through more than one hardware propagation process to generate a complete filter result (e.g., because the convolution engine may only receive and process a portion of the filter weights and / or a portion of the input data values ​​for the window in a loop), the hardware implementation may include multiple accumulators 304. Each accumulator 304 receives the output of one convolution engine 302 and adds that output to the previous convolution engine output associated with the same filter. Because the convolution engine may not generate or produce output associated with the same filter in consecutive loops, partial results for one or more filters may be stored in an accumulation buffer 306, which may then provide the appropriate partial results to the accumulator in each loop. In some examples, the accumulation buffer 306 may be capable of storing partial results associated with 128 different filters.

[0097] The coefficient buffer 308 includes: a memory (not shown) for storing a plurality of weights associated with a convolutional layer (or fully connected layer) of the DNN; and hardware logic (not shown) for providing the weights to the convolution engine 302 for processing in a predetermined order over multiple cycles. The plurality of weights may include all weights associated with the convolutional layer, or only a portion of the weights associated with the convolutional layer. Although the coefficient buffer 308 is shown as a single module, the coefficient buffer 308 may be implemented, for example, by a plurality of coefficient buffers each forming a memory bank.

[0098] The input buffer 310 includes: a memory (not shown) for storing a plurality of input data values ​​associated with a convolutional layer (or fully connected layer) of the DNN; and hardware logic (not shown) for providing the input data values ​​to the convolution engine 302 for processing in a predetermined order in a plurality of cycles. The plurality of input data values ​​may include all input data values ​​associated with the convolutional layer, or only a portion of the input data values ​​associated with the convolutional layer. Although in Figure 3The input buffer 310 is shown as a single module in FIG. 3 , but the input buffer 310 may be implemented by a plurality of input buffers each forming a memory bank, for example.

[0099] DDR 312 is coupled to on-chip memory 314 for providing weight data to on-chip memory 314. DDR 312 is also coupled to input buffer 310 for providing blocks of input data to input buffer 310. On-chip memory 314 is coupled to coefficient buffer 308 for providing weight data to coefficient buffer 308. On-chip memory 314 is also coupled to input buffer 310 for providing intermediate output data (including input data for subsequent layers) to input buffer 310. On-chip memory 314 is also coupled to DDR 312 for providing blocks of output data for the final layer of the plurality of layers to DDR 312. Accumulation buffer 306 is coupled to output buffer 316 to allow the output buffer to receive intermediate output data for layers within the plurality of layers, as well as output data for the final layer. Output buffer 316 is coupled to on-chip memory 314 for providing intermediate output data and output data for the final layer to on-chip memory 314.

[0100] exist Figure 3 The various connections are shown separately in the examples; however, in some embodiments, some or all of them may be provided by one or more shared bus connections. It should also be understood that other connections may be provided as Figure 3 For example, the output buffer 316 may be coupled to the DDR 312 for providing the output data directly to the DDR 312. As a further example, the DDR 312 may be coupled to the coefficient buffer 308 for providing the weight data directly to the coefficient buffer 308. Similarly, in some cases, instead of Figure 3 For example, the DDR 312 need not always be coupled to the input buffer 310 , which may instead obtain input data from the DDR 312 via the on-chip memory 314 .

[0101] Figure 5 is a simplified schematic diagram to better illustrate the Figure 3 The data flow of an example of a hardware implementation is shown in FIG. Figure 5A plurality of layers of a neural network (composed of two layers 502-1 and 502-2) are shown. These can be any two consecutive layers of the network. Note also that the plurality of layers is not limited to two layers. The figure shows input data 504 to the first layer 502-1 of the plurality of layers. Intermediate output data 506 of the first layer 502-1 forms the input data to the second layer 502-2 (the final layer in this two-layer example). The figure also shows output data 508 of the second layer 502-2 (the final layer). DDR 312 is a first memory device with a relatively high read / write cost. On-chip memory 314 is a second memory device with a relatively low read / write cost. For example, reading / writing a given amount of data from / to DDR may consume more time and energy than reading / writing it from / to on-chip memory 314. It should be understood that DDR memory is merely one example of a large-capacity memory storage device, and other memory / storage technologies, such as flash memory, may also be used.

[0102] according to Figure 3 and Figure 5 In the example of , DDR 312 stores input data for a first layer 502-1 of a plurality of layers 502-1, 502-2. DDR 312 also stores weight data representing weights for the plurality of layers 502-1, 502-2. In an example of the present invention, the weight data is transferred from DDR 312 to on-chip memory 314. Coefficient buffer 308 reads the weight data from on-chip memory 314 and provides the weight data to processing element 318. Input buffer 310 reads a first subset of input data representing a first block of input data 504 from DDR 312. This first subset of input data 504 (also referred to as a "header block") is stored in Figure 5 504 and may include one or more data planes. Input buffer 310 provides a first subset of input data 504 to processing element 318. Processing element 318 processes weight data and the first subset of input data 504 to evaluate each of the plurality of layers to compute a first block 508 of output data for the final layer. The first block 508 of output data is Figure 5 314. The output buffer 316 may write the first block of output data 508 to the on-chip memory 314. The first block of output data 508 may be transferred from the on-chip memory to the DDR 312. Alternatively, the output buffer may write the output data directly to the DDR memory 312 (e.g., Figure 3 and Figure 5 (as indicated by the dotted arrow in ).

[0103] like Figure 5As shown, the data input to and output from each layer are arranged in one or more planes (also referred to herein as input datasets and output datasets). Figure 5 In the example shown, the input data 504 includes p in = 3 data planes. The intermediate output data 506 of the first layer 502-1 includes p1 = 3 data planes. The output data 508 of the second layer 502-2 includes p = 2 data planes. The number of planes depends on the weight data. In particular, the weights constituting each filter take a predetermined number of input planes as input data, and each filter outputs a plane of output data. Therefore, the number of data planes generated by a given layer is equal to the number of filters applied by that layer. Each data plane can represent a feature map. In Figure 5 In the example of , the three planes of input data 504 may represent three feature maps generated by an earlier layer of the neural network (if the first layer 502-1 in the figure is not the first layer of the neural network). Alternatively (if the first layer 502-1 is the first layer of the neural network), the three planes of input data 504 may represent three planes of input data. For example, input data with multiple planes often occurs in image processing tasks. Such input planes may include, but are not limited to: red, green, and blue (RGB) planes; or hue, saturation, and value (HSV) planes. The input planes may also include depth data as one of the input planes.

[0104] To compute the first block of output data, processing element 318 must evaluate each of the multiple layers (as described above). In particular, coefficient buffer 308 reads the weight data for first layer 502-1 from on-chip memory 314. Processing element 318 processes the weight data for first layer 502-1 and a first subset of input data 504 to evaluate first layer 502-1 of the multiple layers, thereby computing output data 506 for the first layer. This is a block of output data for first layer 502-1 corresponding to the first block of input data 504. In other words, it is a subset of the complete set of output data for the neural network of first layer 502-1. This block / subset is in Figure 5, shown in gray shading. The output buffer 316 writes blocks of the output data of the first layer 502-1 to the on-chip memory 314. This represents intermediate output data that will be needed as input data to the next layer (the second layer 502-2). In order to evaluate the next layer (the second layer 502-2), the input buffer 310 reads the blocks of output data 506 of the first layer 502-1 stored therein by the output buffer 316 from the on-chip memory 314. The coefficient buffer 308 reads the weight data of the second layer 502-2 from the on-chip memory 314. The processing element 318 processes the weight data of the second layer 502-2 and the blocks of output data 506 of the previous (first) layer to evaluate the second layer 502-2, thereby calculating the output data of the second layer 502-2. This process continues for any subsequent inner layer of the plurality of layers—each time the intermediate output data is saved in the on-chip memory 314 and read to provide the input data to the next layer. This continues until the last layer is reached. Figure 5 In the example shown in FIG. 5 , second layer 502 - 2 is the final layer. Therefore, the output of second layer 502 - 2 is the first block of output data 508. As explained above, output buffer 316 can write the first block of output data 508 to on-chip memory 314. From the on-chip memory, first block of output data 508 can be transferred to DDR 312. Alternatively, output buffer 316 can write the output data directly to DDR 312 (dashed arrow).

[0105] In this way, processing subsets (tiles) of input data through multiple layers of a neural network can efficiently utilize the processing resources of processing element 318. Writing intermediate output data to on-chip memory 314 and reading it from on-chip memory 314 to process the next layer reduces the need to read and write data from off-chip DDR 312, thereby helping to reduce power consumption. It is important to note that on-chip memory 314 can be much smaller than off-chip DDR 312 because, during each propagation process, on-chip memory 314 only needs to store data associated with the current block of input data, the current block of output data, and the associated block of intermediate output data for the internal layer. This contrasts sharply with conventional approaches to evaluating neural network layers, where each layer is processed independently in its entirety and the memory needs to be able to store all input data and all output data for the layer at any given time. This requires much larger memory, which is generally impractical to implement as on-chip memory. Therefore, conventional approaches in practice require the use of off-chip memory to store intermediate output data. Since off-chip read and write operations typically consume more energy than on-chip read and write operations, reducing the amount of data written to off-chip memory can significantly save power.

[0106] The size of the block to be processed during each pass can be selected based on the available storage capacity of the on-chip memory 314, the number of layers in the plurality of layers, and the structure of the neural network. An exemplary method of memory management in the on-chip memory 314 is described below. First, it is useful to consider how blocks of output data of multiple layers in a neural network depend on the intermediate output data of previous layers and the input data of the first layer. Figure 6 This can be understood. The figure shows the input data of three layers of the neural network (not explicitly shown); the output data of the first layer (labeled here as layer 0); the output data of the second layer (labeled as layer 1); and the output data of the ending layer. For simplicity, it is convenient to think of the output data of the ending layer as a single feature map created by a single filter. Similarly, the other layers can be thought of as convolutional layers with a single filter, thereby creating a single output feature map, as shown in simplified form in the figure. Of course, the scope of the present disclosure is not limited in this way, and these are merely simplifications made for ease of understanding. At each layer, a single filter can be applied to multiple input channels (feature maps), or multiple filters can be applied to one input channel or multiple input channels, thereby processing and / or generating multiple data planes. The methods and hardware implementations according to the present disclosure can be used equally in such scenarios.

[0107] Starting from the first block of output data in the final layer (tile 1), it's possible to trace back through the network to find the intermediate and input data that this output block depends on. There's a block of data (tile 1) in the layer 1 output that corresponds to tile 1 in the final layer; in this sense, tile 1 in layer 1 is required to compute tile 1 in the final layer. Assume the kernel size of the filter is greater than 1. Because of this kernel size, tile 1 in the layer 1 output is larger than tile 1 in the final layer output. In other words, tile 1 in the final layer output depends on some additional data in the layer 1 output. For example, with a 3x3 kernel and a stride of 1, an extra row of output data is required in the layer 1 output. Figure 6 The slanted dashed line in indicates this expansion of the required data volume. In this figure, each tile is separated by a solid line.

[0108] Next, consider which data in the layer 0 output is necessary to compute tile 1 in the layer 1 output. For the sake of diversity, we will assume that the stride of layer 1 is greater than 1. This means that the layer 1 output is smaller than the layer 0 output. There is a block in layer 0, tile 1, that corresponds to tile 1 in layer 1. Since the kernel size is also greater than 1, the amount of data required at the boundary increases further. Therefore, tile 1 in the layer 0 output is larger than tile 1 in the layer 1 output.

[0109] The same situation occurs when tracing back from layer 0 to the input data. There is a block in the input data, tile 1, that corresponds to tile 1 in layer 0. Assume that the stride in layer 0 is 1; therefore, the size of the input data is approximately the same as the size of the layer 0 output (depending on padding). Again, because the kernel size is greater than 1, additional data is required at the boundaries. Therefore, tile 1 in the input data is also larger than tile 1 in the layer 0 output. Tile 1 in the input data constitutes the first subset of the input data (block 1). Tile 1 in the output data of the final layer constitutes the output block.

[0110] From the above, we can infer that the total amount of data required for the first tile increases from the end tile to the input tile, depending on the parameters of all intermediate layers (kernel size, stride, dilation). Figure 6 Also shown in is 'overlap'. This is data that can be reused during subsequent propagation, i.e., data that is reused when computing subsequent tiles. Overlap data is indicated by horizontal dashed lines in the data for each layer. Overlap 1 in the output of layer 1 is the portion of tile 1 that can be used (repeatedly) to compute the subsequent tile (tile 2) of the output of the end layer. Overlap 1 in the output of layer 0 is the portion of tile 1 that can be used (repeatedly) to compute tile 2 of the output of layer 1. The size of the overlap depends only on the parameters of the next layer (kernel size, dilation). For example, if stride = 1 and dilation = 1, the size of the overlap (number of rows overlapped) is the kernel size minus 1.

[0111] Immediately afterwards Figure 6 , Figure 7 The data dependencies for the second tile are shown. The second tile (Tile 2) of the output data of the ending layer is adjacent to the first tile (Tile 1). Like Tile 1, Tile 2 depends on the amount of intermediate data (ultimately the input data) that is added as the layers are traversed backward. However, some of these data dependencies are the same as the dependencies of Tile 1 in the output data of the ending layer. In particular, when evaluating Layer 1 for Tile 1 (Tile 1), the overlap 1 in the output of Layer 1 has already been calculated. The only additional data that needs to be calculated is the tile in Layer 1, Tile 2. Similarly, moving backward to the Layer 0 output, the only additional data that needs to be calculated is the tile in Layer 0, Tile 2. Finally, moving backward to the input data, the only additional input data that needs to be read from memory is Tile 2 (Tile 2). As Figure 6 As shown, Figure 7Each tile in is separated by a solid line. Overlapping data is separated by dashed lines. Dashed lines indicate the highest limit of data dependencies (i.e., the first row of data required to calculate the tile in the next layer). Slanted dashed lines indicate the lowest limit of data dependencies (i.e., the last row of data required to calculate the tile in the next layer). So, for example, tile 2 in the output of layer 1 depends on overlap 1 and tile 2 in the output of layer 0. Tile 2 contains overlap 2, which will be reused in the next propagation to calculate tile 3 in the subsequent layer, and so on. Note that if the overlapping data of the first propagation process is retained, the amount of "fresh" data required for tile 2 will not increase when traversing these layers backward (except where indicated by a stride greater than 1). The same is true for all subsequent propagation processes (tiles) after tile 2. In other words, for all propagation processes after the first propagation process (operating on the head tile), a smaller amount of input data needs to be retrieved from memory, and a smaller amount of intermediate data needs to be calculated at each layer.

[0112] This leads to an important observation: the overlapping data constitutes the non-disposable portion of the data. The rest of each tile constitutes the disposable portion of the data. This means that the overlapping data is not only required to compute the output data of the end layer of the current tile, but is also needed later to compute the output data of the end layer of subsequent tiles (subsequent tiles). In contrast, the rest of the data in each tile is necessary to compute the output data of the end layer of the current tile, but can be deleted / overwritten thereafter.

[0113] For a detailed consideration of this using a simplified one-dimensional example, refer to Figure 8 . Figure 8 The diagram in Figure 1 shows the individual data elements in the input and output data for each of the three layers of a neural network (layer 0, layer 1, and the final layer). Each circle represents a data element (which could be, for example, a pixel in an image or feature map). Each element depends on a subset of elements in the previous layer. Figure 8The connecting lines in indicate the elements of the previous layer on which a given element depends. So, for example, the 3rd element in the output of the ending layer depends on the 2nd, 3rd, and 4th elements in the output of layer 1. The 4th element in the output of the ending layer depends on the 3rd, 4th, and 5th elements in the output of layer 1. In other words, in this example, the kernel size of the ending layer is 3 and the stride is 1. The kernel size of layer 1 is 3 and the stride is 2. The kernel size of layer 0 is 3 and the stride is 1. Since the stride of both layer 0 and the ending layer is 1, the output of layer 0 is approximately the same size as the input data, and the output of the ending layer is approximately the same size as the output of layer 1 (subject to padding requirements). At the same time, since the stride of layer 1 is 2, the output of layer 1 is approximately half the output of layer 0. The dotted lines represent the boundaries between blocks (tiles). That is, the 3rd data element shown in the ending layer is the final data element of the first block (tile 1). Taking the fourth element in the final layer output as an example, we can see that each element in the final layer output depends on: 3 elements in the output of layer 1; 7 elements in the output of layer 0; and 9 elements in the input data. Elements in tile 1 (except for the Overlap 1 element) are shown with diagonal shading. Three such elements (1-3) are shown in the final layer output by way of example.

[0114] Due to data dependencies, some elements may be discarded after the final layer output of the current block (tile 1) has been calculated. These elements are called "one-time" elements. Some other elements will also be used to calculate the final layer output of the next block (tile 2). These elements are called "non-one-time" elements. In particular, the overlap 1 element ( Figure 8 ) are non-disposable elements. Elements 7 and 8 in the input data are used to calculate elements 6 and 7 in the output of layer 0 during the first pass through multiple layers to calculate the ending layer output data for the current block (tile 1). However, these elements are also needed to calculate elements 8 and 9 in the output of layer 0. These elements 8 and 9 are not needed to calculate the first block during the first propagation, but they will be needed to calculate the second block during the second propagation. Therefore, in order to minimize memory access overhead, it is desirable to retain elements 7 and 8 of the input data in the on-chip memory 314 to avoid the need to read these elements again from DDR 312 during the second propagation. Element 7 in the output of layer 0 and elements 3 and 4 in the output of layer 1 are also overlap 1 elements because these elements will also be needed when performing calculations in the second propagation (element 7 in the output of layer 0 will be needed to calculate element 5 in the output of layer 1; and elements 3 and 4 in the output of layer 1 will be needed to calculate elements 4 and 5 in the ending layer output).

[0115] Now consider Figure 8The remaining elements of tile 1 are shown with diagonal shading in the figure. Element 6 of the input data is used to compute elements 5, 6, and 7 of the layer 0 output. However, each of these three elements must be computed during the first propagation to compute the first output block (tile 1). Element 6 of the input data is not needed in any additional computation during the second propagation; therefore, this element is "one-time" data. Similarly, element 6 in the layer 0 output is used to compute element 4 in the layer 1 output, but is no longer needed during the second propagation (assuming that element 4 in the layer 1 output is retained). The elements of tile 2 (except for the Overlap 2 elements) are shown as hollow circles. The Overlap 2 elements are shown in gray shading. These elements will be reused in the computation of tile 3 and are therefore non-one-time data. Some tile 3 elements are shown with horizontal shading at the bottom of the figure.

[0116] The inventors of the present invention have recognized that disposable data is only used to calculate the next layer in the current propagation process through multiple layers. Once the next layer in the current propagation process is calculated, the disposable data from the previous layer can be deleted / overwritten. For all layers of the current propagation process, non-disposable data (overlap) must be retained so that it can be used to calculate the second block in the second propagation process. This recognition can reduce unnecessary read / write operations from / to DDR 314 and can also reduce unnecessary recalculation of intermediate output data.

[0117] Now refer to Figure 9 The memory management strategy according to an example of hardware implementation is described. The inventors have realized that one-time data is relatively large but only valid for a short time. In contrast, non-one-time data (overlapping) is relatively small but valid for the entire propagation process. Figure 9 An exemplary memory allocation for on-chip memory 314 that exploits these characteristics is shown. In this example, assume that six layers of a neural network (numbered 0, 1, 2, 3, 4, and 5) are processed together. In one branch, each layer is connected to its previous layer. In the second branch, layer 5 is connected to layer 1. This means that the output data of layer 1 is part of the input data of layer 5.

[0118] The memory management strategy divides the on-chip memory 314 into three sections. The first section 802 (which may be referred to as "swap" memory) is used to store one-time data. The second section 804 (which may be referred to as "heap" memory) is used to store non-one-time data. The third section 806 (coefficient memory) is used to store weight data for each layer. Within the first section 802, multiple slots A, B, and C are defined. During a given propagation process (for computing a given block of output data), the method proceeds as follows. The one-time output of layer 0 is written to slot A. The non-one-time output of layer 0 is written to the beginning of the second section 804. Before evaluating layer 1, data is read from these two locations. During the evaluation of layer 1, the one-time output of layer 1 is written to slot B, and the non-one-time output of layer 1 is written to the second section 804, e.g., appended to the non-one-time output of layer 0. At this point, the one-time data from layer 0 is no longer needed. Therefore, during the evaluation of layer 2, the one-time output of layer 2 can overwrite the contents of slot A. The non-disposable outputs of layer 2 are again appended to the data in the second section 804. However, when evaluating layer 3, neither slot B nor slot A should be overwritten. Due to the branch, the contents of slot B (the one-time outputs of layer 1) are again needed to evaluate layer 5. Of course, the contents of slot A are immediately needed to evaluate the current layer (layer 3). Therefore, the one-time outputs of layer 3 are written to slot C. The one-time outputs of layer 4 are written to slot A, again overwriting the contents of that slot that are no longer needed for further calculations. To evaluate layer 5, the one-time output data of layer 1 is read from slot B, and the one-time output data from layer 4 is read from slot A. (The non-disposable output data from layers 1 and 4 are read from the second section 804.)

[0119] In this way, slots used to store one-time data are frequently reused during a given propagation process, thereby helping to reduce the total amount of on-chip memory 314 occupied at any given time. The size of each slot can be selected based on the maximum size of one-time data to be stored in the slot during the propagation process. Note that in this example, the second section 804 is double-buffered. The non-one-time output data (overlap) for all layers of the current block are stored in a set of memory locations. During the next propagation process (for computing the next block), the overlap is retrieved from this set of memory locations as needed. The overlap for the next block is stored in the second set of memory locations. For subsequent propagation processes (subsequent blocks), the hardware implementation alternates between using the first set of memory locations and the second set of memory locations, i.e., reading from one set while writing to the other.

[0120] The one-time and non-one-time portions of the data can be identified by analyzing the structure of each of the multiple layers. Similarly, assigning the one-time data for each layer to a specific slot in the first section 802 of the on-chip memory 314 can be accomplished by analyzing the overall layer structure of the multiple layers. This analysis can be performed dynamically by software at runtime, or when mapping a given neural network to a given hardware implementation. Alternatively, this can be manually defined by the designer of the hardware implementation. However, the use of software facilitates mapping a variety of different neural networks to a given hardware implementation.

[0121] As described above, the multiple layers of the neural network evaluated together during each propagation (also referred to herein as "tile groups") can include all layers of the neural network, but this is not required. The multiple layers can be composed of a subset of the layers of the entire neural network. Depending on the structure of the neural network, different groupings of layers may be found to be advantageous. Figure 10 A simple neural network is shown, consisting of four layers with no branches, where each layer depends only on the output of the previous layer. This allows a high degree of freedom in choosing how to group layers together. For example, the layers can be grouped into one or more groups according to any of the following grouping strategies:

[0122] 1-4

[0123] 1-2, then 3-4

[0124] 1-2, then 3, then 4

[0125] 1-3, then 4

[0126] 1, then 2-4

[0127] 1, then 2-3, then 4

[0128] 1, then 2, then 3-4

[0129] Generally speaking, subject to the size of on-chip memory 314, it may be preferable to group as many layers together as possible.

[0130] Figure 11A shows a slightly more complex network structure. Again, there are four layers, each layer depending on the previous layer. However, there are additional branches from layer 2 to layer 4. Figure 11AIn Figure 3, the network is divided into two groups, each with two layers (Layer 1 to Layer 2 and Layer 3 to Layer 4), indicated by the dashed boxes in the figure. However, it should be noted that this may not be optimal in terms of memory access requirements. The output of Layer 1 to Layer 2 will be written to DDR 312. When Layer 3 (the first layer of Layer 3 to Layer 4) is evaluated, this output will be read from DDR 312; however, the evaluation of Layer 4 also requires the output of Layer 2. Therefore, the same data must be read again from DDR 312 in order to evaluate Layer 4. Instead, Figure 11B An alternative grouping of layers is shown that may be more efficient in terms of memory access requirements. Here, layer 1 is handled by itself, and layers 2 through 4 are grouped into multiple layers. This way, both branches are captured within the multiple layers. The output data of layer 1 is read once from DDR 312 as input data for the first layer (layer 2) of the multiple layers. Using the above reference Figure 9 The memory management strategy can process the output data of layer 2 in multiple layers.

[0131] First, refer to the above Figure 2 , an example of a convolutional layer of a neural network is given, operating on two-dimensional (2D) image data in multiple p-planes or feature maps. However, it should be understood that the present disclosure is not limited to this approach. The above principles can be applied to data of 1D, 2D or higher dimensions. When applied to 2D (or higher dimensional) data, blocks (tiles) of input data and output data can be constructed in a variety of ways. Each block can be defined in one dimension, two dimensions, or more. For example, for 2D image data, blocks defined in one dimension can form 2D stripes (in the horizontal or vertical direction). Overlapping data can be defined above and below the stripe (for horizontal stripes) or to the left and right of the stripe (for vertical stripes). With this in mind, the following is drawn. Figure 5 、 Figure 6 and Figure 7 (These figures also apply to 1D blocks defined in 1D data.) It should be understood that although Figures 5 to 8 This indicates that the data is divided into blocks in the Y (height) dimension, but this is not intended to be limiting. As mentioned above, the same approach can be applied to the X (width) dimension. However, in some cases, it may be more preferable to split in another dimension than in another dimension. For example, if the data is stored in memory in rows, and the individual data elements do not align neatly with the burst size used to read / write the memory, it may be preferable to split the tiles in the Y (height) dimension.

[0132] Alternatively, for 2D image data, blocks can be defined in two dimensions, forming a 2D rectangle of data. Overlapping data can then be defined above, below, to the left, and to the right of this rectangle. Overlapping data from a current block may be useful for computing multiple subsequent blocks of output data, such as the block to the right of the current block, the block below the current block, and the block to the right below the current block (assuming blocks are processed in raster scan order). The same principle can be extended to blocks in more dimensions, with overlapping data in multiple directions in multidimensional space.

[0133] In the above example, the depth-first processing of the neural network in tile units is discussed in terms of the layers of the network. This applies to Figure 3 , which is primarily designed to process a single convolutional layer. However, it is also possible to provide a more sophisticated hardware implementation that adds additional processing modules to the implementation 300, thereby enabling multiple layers to be processed in a single pass through the hardware implementation. For example, a pooling module, an element-wise operation module, a normalization module, and an activation module may be provided (see, for example, GB 2566702 A). For example, this may allow a convolutional layer to be executed, followed by an activation function, followed by a pooling operation, in a single pass through the hardware implementation. In some cases, data may be propagated from one layer / operation to the next in a pipelined manner without having to write it to on-chip memory. When the hardware implementation is capable of performing multiple sequential operations like this in a single hardware pass, it makes sense to group together layers that are processed together by the hardware. To accommodate this, the concept of a "layer group" is introduced. A layer group is a group of layers that can be processed together by the hardware implementation in a single pass through the hardware. A layer group may consist of a single layer (as described above with reference to Figure 3 however, a layer group may consist of more than one layer if the hardware has the necessary functionality.

[0134] As mentioned above, partitioning the data into tiles can provide a way to facilitate parallel multi-core implementations of convolutional neural networks. Figures 12 to 15 An example of a multi-core implementation according to the first aspect is described. Figure 12 A block diagram of a hardware implementation 600 is shown. It includes three cores 350-0, 350-1, and 350-2. Each core has a corresponding local on-chip memory (OCM) 314a, which is accessible only to that core. The hardware implementation 600 also includes a shared on-chip memory 314b. All cores can access (i.e., read and write) the shared OCM 314b via a high-bandwidth interconnect 610. A splitter unit 620 is provided, which is configured to manage the distribution of work between the cores.

[0135] Figure 13An example of processing tiles in a depth-first manner using different cores 350 of a hardware implementation 600 is schematically shown. The individual tiles are similar to those described previously. In this example, the tiles are divided along the Y dimension of the input data tensor. This is convenient when the data is stored in memory row-first (i.e., X-dimension first). As shown, the neural network includes five layer groups (labeled LG0 to LG4). Each layer group is indicated as including three layers represented by circles in the figure. (It should be understood that this is not limiting - different layer groups can have different numbers of layers.) The layer groups will be evaluated by the hardware implementation in two "tile groups" (labeled Tile Group 0 and Tile Group 1). A tile group consists of a set of layer groups that will be evaluated together depth-first after the data is divided into tiles. The input data for the first layer group in the tile group is read from DDR 312, and the output data from the final layer group in the tile group is written to DDR 312. Within a tile group, intermediate data (in units of tiles) is written to the OCM 314 by each layer group and read from the OCM 314 by the next layer group. The intermediate data may be partially stored in the local OCM 314a and partially stored in the shared OCM 314b. This will be explained in more detail below.

[0136] As shown in the figure, in this example, the input data stored in DDR 312 is divided into six tiles (labeled as tile 0 to tile 5) for evaluating tile group 0. There are only three cores 350; therefore, in this tile group, not all tiles can be processed in parallel. Instead, the first three tiles are assigned to the three cores respectively for the first multi-core tile propagation process (multi-core tile propagation process 0). Here, the multi-core tile propagation process refers to the evaluation of all layer groups of a tile group of N consecutive tiles, where N is the number of cores. Tile group 0 consists of the first three layer groups LG0-LG2. In multi-core tile propagation process 0, core 0 evaluates layer groups LG0-LG2 based on tile 0; core 1 evaluates these layer groups based on tile 1; and core 2 evaluates these layer groups based on tile 2. This is in Figure 13 Next, as indicated by the dashed arrows, the hardware implementation 600 proceeds to the multi-core tile propagation process 1, where core 0 evaluates the layer group LG0-LG2 based on tile 3; core 1 evaluates the layer group based on tile 4; and core 2 evaluates the layer group based on tile 5. This completes the evaluation of the first tile group. Note that Figure 13 The main purpose is to show the execution order of the check blocks. This figure does not provide a detailed description of which parts of the data are stored in which memory. Figure 14A and Figure 14B Further details are provided to explain how data is arranged in memories 312 and 314 .

[0137] During each multicore tile propagation, the output tiles of LG0 are written to OCM 314 and read from OCM 314 as input tiles for LG1. Similarly, the output tiles of LG1 are written to OCM 314 and read from OCM 314 as input tiles for LG2. Before the hardware implementation 600 continues evaluating tile group 1, the output data of the first tile group (i.e., the output tiles of LG2) are written to DDR 312.

[0138] Tile group 1 consists of two layer groups LG3-LG4. As shown, for this tile group, the data is partitioned into only three tiles. The number of tiles can be selected based on the size of the input data for the tile group, which is determined by the size on OCM 314. The number of tiles may also be affected by the number and configuration of layers within the tile group, as these will affect the amount of data that needs to be stored in OCM 314 during the evaluation of the tile group for each tile. Tile group 1 can be evaluated in a single multi-core tile propagation process because the number of tiles is not greater than the number of cores. Core 0 evaluates layer groups LG3-LG4 based on tile 0; core 1 evaluates layer groups LG3-LG4 based on tile 1; and core 2 evaluates layer groups LG3-LG4 based on tile 2. The output tiles of LG3 are written to OCM 314 and read from OCM 314 as input tiles for LG4. The output tiles of LG4 are written to DDR 312.

[0139] FIG. 14A to FIG. 14B It is shown (for different examples) how tiles can be assigned to and processed by multiple cores. Figure 14A The output data of three layer groups LG0-LG2 divided into tiles is shown. Here, the notation "LGxTn" means "layer group x, tile n". As mentioned above, Figure 5 As explained, the size of the tiles in each layer is defined by the data dependencies in the neural network layers. Starting from the final layer group, the size of the head tile is expanded from the head output tile LG2T0 in the final layer group to the head output tile LG1T0 in the intermediate layer group, and again to the head output tile LG0T0 in the initial layer group LG0. In each layer group, the subsequent tiles after the head tile LGxT0 have a consistent size in that layer group, with the exception of the tail tile. When moving backward through the layer groups from the final layer group to the initial layer group, the tail tile may shrink. This can be seen by comparing the tail tile LG2T7 of the final layer group and the tail tile LG1T7 of the intermediate layer group. In addition, the tail tile LG1T7 completely "disappears" in the initial layer group, that is, there are only seven tiles in this layer group instead of eight.

[0140] Tiles are processed in a multi-core architecture with four cores (C0-C3). Figure 14A and Figure 14BIn both, tiles are colored / shaded depending on which core processes them. Figure 14B shows how layer groups are processed by cores over time, with arrows indicating data dependencies between tiles. During the first multi-core tile propagation, the first four tiles are processed by four cores C0-C3 respectively. In the first layer group, the head tile LG0T0 is larger than the other tiles, so the time it takes for core C0 to evaluate the layer group based on this tile is longer than the time it takes for the other cores C1-C3 to evaluate the layer group based on the other tiles. In the example of the present invention, each core waits until all cores have completed the evaluation of the layer group. Therefore, cores C1-C3 wait for C0 to complete the head tile LG0T0. This provides a simple and efficient way to achieve synchronization between cores. After the evaluation of the first layer group, all four cores continue with the evaluation of the second layer group, and then the evaluation of the third layer group. From Figure 14B As can be seen in Figure 2, in the evaluation of the final layer group during this multi-core tile propagation process, the head tile LG2T0 is evaluated faster than the other tiles. This is because the head tile LG2T0 has been chosen to be smaller than the subsequent tiles LG2T1-LG2T7. This is done to prevent the head tile from becoming too large in the initial layers. The goal is to balance the division of labor between the cores and to prevent one core from completing its work significantly earlier or later than any other core.

[0141] As referenced above Figure 7 and Figure 8 As explained, the evaluation of a layer group for a given tile (except the header tile) depends on overlapping data from previous output tiles in previous layer groups. For example, the evaluation of LG1T1 depends on both LG0T1 and LG0T0. Similarly, the evaluation of LG1T2 depends on LG0T2 and LG0T1, and so on. This dependency between tiles during multi-core tile propagation is given by Figure 14B 314b. The dependency relationship means that each core needs to access overlapping data generated by another core. To this end, the overlapping (non-one-time) data of each tile is stored in the shared OCM 314b. The remaining data (i.e., the one-time data) can be stored in the local OCM 314a of each core because no other core needs to access it. This avoids the need for each core to recalculate the overlapping data, while also avoiding the expensive overhead that may be involved in writing the overlapping data to the DDR 312. At the same time, the access cost of the local OCM 314a may be lower than that of the shared OCM 314b, providing improved efficiency compared to the implementation of storing all data in the shared OCM.

[0142] Note that, in principle, it is not necessary for each core to wait for all cores to complete a layer group. For example, in principle, the evaluation of LG1T2 could begin immediately after LG0T1 and LG0T2 complete. Similarly, the evaluation of LG1T0 could begin at any time after the evaluation of LG0T0 completes (i.e., an indeterminate amount of time). This "free-running" approach could indeed be adopted in some other implementations. However, this implementation has the advantage of simplicity because it avoids the additional complexity required to track free-running.

[0143] The first multi-core tile propagation process ends with Figure 14B , as indicated by the thick black vertical line in . Next, the core performs a second multi-core tile propagation process to evaluate the layer groups based on the remaining tiles. There are a few things to note about the second multi-core tile propagation process. For the second and third layer groups LG1-LG2, it should be understood that the fifth tile is dependent on overlapping data generated when the fourth tile was evaluated in the previous multi-core tile propagation process. In other words, although the overlapping data from LG0T0, LG0T1 and LG0T2 are used "immediately" by the core in the evaluation of LG1T1, LG1T2 and LG1T3 respectively, the overlapping data from LG0T3 needs to be retained in the shared OCM 314b until the second multi-core tile propagation process, at which time it is used by core C0 in the evaluation of LG1T4. Similarly, the overlapping data from LG1T3 is retained in the shared OCM 314b until it is used by core C0 in the evaluation of LG2T4. These dependencies between tiles processed in different multi-core tile propagation processes are indicated by Figure 14B Another point to note is that the evaluation of the second layer group based on the last tile LG1T7 is completed faster than the other tiles in this layer group. This is because the size of the tile is smaller, and it is reduced compared to the last tile of the final layer group LG2T7 (see Figure 14A ). Additionally, kernel C3 is idle during the evaluation of LG0 during the second multi-core tile propagation process, since that layer group has only seven tiles. It will be appreciated that due to the overlap at the boundaries between tiles, dependencies between tiles and the corresponding need for synchronization between kernels arise. In cases where there is no overlap (e.g., a series of convolutions with a 1x1x1 kernel), no synchronization is required, and the kernels can be allowed to run freely (without any complexity penalty). Generally speaking, for convolutions, the formula for determining the size of any overlap in the Y dimension is: overlap = max(0, (kernel height - 1) * dilation height - (stride height - 1)).

[0144] Will refer to Figure 15 The method is described in more detail. In step 710, the splitter unit 620 defines a plurality of layer groups and arranges them in tile groups. The arrangement in the tile groups can be as described above with reference to Figures 10 to 11BThe process is completed as described above. In step 712, input data for the first layer group is provided in DDR 312. In step 714, the splitter unit 620 analyzes the input data for the first layer group. In particular, it checks whether the size of the input data in the Y dimension is above a threshold. If so, it determines (step 716) that the input data can be split into tiles along the Y dimension. In other words, if the input data has a sufficiently large size in the Y dimension, it will be split into tiles. In response to the determination in step 716, the splitter unit splits the data into tiles (step 718) and assigns each tile to one of the available cores (step 719). If there are more tiles than cores, the splitter unit 620 controls the cores to perform several multi-core tile propagation processes. Generally speaking, the splitter unit attempts to split the input data into tiles so that all cores are occupied. Therefore, it may try (where possible) to split the input data into a number of tiles that is an integer multiple of the number of cores. The number of tiles (and the number of multi-core tile spreading processes) will be affected by the capacity of the OCM 314 .

[0145] In step 720, at least two cores evaluate the first layer group based on at least two corresponding tiles. That is, each core evaluates the first layer group based on one tile. The splitter unit is configured to identify the disposable portion and the non-disposable (overlapping) portion of each output tile. It instructs the cores to store these different portions in different places. Therefore, the first core 350-0 writes the disposable portion of the first output tile to its local OCM 314a (step 722) and writes the overlapping portion of the first output tile to the shared OCM 314b (step 724). Simultaneously, the second core 350-1 writes the disposable portion of the second output tile to its local OCM 314a (step 722) and writes the overlapping portion of the second output tile to the shared OCM 314b (step 724). The flowchart only shows two instances of steps 720-724, which means that there are two cores; however, it should be understood that there can be as many parallel instances of these steps as there are cores involved in the evaluation. It should be noted that steps 722 and 724 can also be performed in the opposite order or in parallel.

[0146] In step 728, any core that completes its evaluation before the other cores waits for all cores to complete their evaluation of the first layer group before continuing. The method proceeds to step 730. If there are additional layer groups in the current tile group, each core proceeds to evaluate the next layer group based on its tile. When evaluating the second and subsequent layer groups, each core reads the necessary one-time data from the previous layer group from its own local OCM 314a and reads the necessary overlapping data from the previous layer group from the shared OCM 314b.

[0147] This loop continues until all layer groups for the current tile group have been evaluated by each core based on the corresponding tiles. When (at step 730) there are no additional layer groups in the current tile group, the method proceeds to step 732 to check whether there are additional tiles to be processed. This represents the end of one multi-core tile propagation process. If there are additional tiles to be processed, the core will continue with a second multi-core tile propagation process to evaluate all layer groups for the current tile group for the next group of tiles. The method continues to loop within the multi-core tile propagation process until all tiles have been processed. Then, in step 732, when there are no more tiles to be processed, the evaluation of the tile group is complete and the method ends. If necessary, the method can be repeated for subsequent tile groups.

[0148] It should be noted that in steps 714 and 716, the splitter unit 620 determines that the input data can be split into tiles based on the size of the input data (particularly in the Y dimension). The splitting into tiles depends on whether the tensor size is large enough. However, even for relatively small tensors, it is advantageous to be able to split the processing among multiple cores. According to a second aspect, a method for doing so is provided. Reference will now be made to Figure 12 、 Figure 16 and Figure 17 An example of this approach is described. According to this example, the evaluation may be split between the cores along one of the other dimensions. This may include splitting the evaluation of the layer group along the X dimension, which is one of the traversed dimensions. Likewise, it may include splitting the evaluation of the layer group across one of the non-traversed dimensions, in particular across input channels C, or across output channels F. Typically, when the evaluation is split into propagation processes, the input data for the layer group is not split between the cores, but is stored in the shared OCM 314b and accessed from there by each of the cores 350. This is different from the first aspect (e.g., Figures 13 to 15 Unlike the use of tiles in the example (illustrated in FIG), in the first aspect, both the data and the evaluation are partitioned between the kernels. When partitioning along the C dimension, each kernel evaluates a layer group of the corresponding channel (or channel group) of the input data. When partitioning along the F dimension, each kernel evaluates a layer group of the corresponding filter (or filter group), producing a corresponding output channel or output channel group. When partitioning along the X dimension, each kernel evaluates a layer group of the corresponding column (or group of columns) of the input data.

[0149] Figure 16An example of splitting the evaluation of multiple layer groups into multiple propagation processes, split across multiple cores, and evaluating one layer group at a time is schematically shown. There are five layer groups LG0-LG4. Layer group LG2 includes two layers; the other layer groups each include three layers. The input data for the first layer group is stored in DDR 312. In this example, the first layer group LG0 is split into three propagation processes to be evaluated by different cores. The division is along the X dimension. Each of the three cores C0-C2 evaluates a corresponding one of the three propagation processes. In each propagation process, the corresponding core processes a portion of the input data to produce a portion of the output data. In this example, it is assumed that there is at most one convolutional layer in each layer group. Therefore, the problem of overlapping data does not arise. Depending on the kernel size and stride, different cores can access overlapping parts of the input data, but they generate different parts of the output data. Figure 13 Unlike the case of , in this example, the data is not split into separate tiles. (In the current example, this would be difficult to do in the X dimension because the data is arranged row-major and the data elements in the X dimension do not align with the bursts of data in memory.) Instead, each core accesses the portion of data it needs from the shared OCM 314b. As already noted, each core generates a corresponding portion of the output data. When all cores have evaluated the first-level group, the portions of the output data are combined in OCM 314.

[0150] The next two layer groups, LG1-LG2, are not split; therefore, each layer group is evaluated by a core in a single pass. The subsequent layer group, LG3, is split into six passes across the output channels (the F dimension). Since there are more passes than cores here, cores C0-C2 first evaluate the layer group for the first three passes in the F dimension (labeled 0, 1, and 2 in the figure). Cores C0-C2 then evaluate the layer group for the second set of three passes (labeled 3, 4, and 5 in the figure) to complete the layer group. The final layer group, LG4, is again processed by core C0 in a single pass. The outputs of the first four layer groups, LG0-LG3, are combined in OCM 314; the output of the final layer group, LG4, is written to DDR 312.

[0151] As shown by this example, the segmentation is flexible in that different layer groups can be segmented along different dimensions (e.g., compare LG0 and LG3). This is achieved by combining the complete output of each layer group in OCM 314, which makes the segmentation of one layer group independent of the segmentation of other layer groups. This is in contrast to Figure 13 Another difference between the patch-based segmentation and Figure 13 In

[15] , each kernel operates on a specific tile, and the correspondence between tiles is preserved when the layer groups of tile groups are evaluated depth-first.

[0152] Will refer to Figure 17The method is described in more detail. Figure 15 Similar steps to those in will be given the same reference numerals. In step 710', the splitter unit 620 defines a plurality of layer groups. In step 712', input data for a first layer group is provided in the DDR 312. In step 714', the splitter unit 620 analyzes the input data to check whether it can be split into tiles. In particular, the splitter unit checks whether the size of the input data in the Y dimension is above a threshold. In step 746, the splitter unit 620 determines that the size is not above the threshold - therefore, the data is not suitable for tiling. In response to this determination, the splitter unit 620 splits the evaluation of the layer groups into propagation processes (step 748) and assigns each propagation process to the cores (step 749). In step 750, the cores evaluate the layer groups of their corresponding propagation processes to generate corresponding portions of the output data. If it is determined in step 752 that there are additional propagation processes to be completed in the current layer group, the cores evaluate the layer group for the necessary additional propagation processes (as in Figure 16 In the example of layer group LG3 in FIG. 7 , if it is determined in step 752 that there are no additional propagation processes to evaluate for the current layer group, the method proceeds to step 754, and portions of the output data from each propagation process are combined to generate output data for the layer group. In step 756, this output data is written to OCM 314. In step 758, a check is performed to determine whether there is another layer group to evaluate. If so, the method proceeds again to step 748 to split the evaluation of the next layer group into propagation processes. Note that although only two instances of step 750 are shown in the flowchart, implying the presence of two cores, it should be understood that as many parallel instances of this step as there are cores occupied by the evaluation are possible.

[0153] The output data of each layer group is written to the shared OCM 314b so that all cores can access it to evaluate the next layer group. The manner in which the evaluation is partitioned in step 748 and the manner in which the output data portions are combined in step 754 will depend on the characteristics of the layers in the layer group and the dimension along which the evaluation is partitioned. In some cases, the evaluation may be partitioned by input channel C, so that each core evaluates the layer group for a corresponding channel (or group of channels) of the input data. For example, if the layer group includes a convolutional layer with a single filter, and each core processes a different input channel, this would mean that each core applies one channel of the filter kernel to the corresponding channel of the input data. The corresponding output portions produced by the different cores would then need to be combined via element-wise summation to produce the convolved output. This can be accomplished by an element-wise operation module in one of the cores 350 (if available) or by another processing module separate from the core 350. The weights (and biases) for the different channels of the kernel can be stored in the local OCM 314a of each corresponding core, as each core only needs to access the weights for the channel (or channels) it is responsible for evaluating.

[0154] If the layer group consists of element-wise operations and each core processes a different input channel, then the corresponding output portions produced by different cores can be combined by concatenating them in the input channel dimension (P dimension). Note that because the step of combining the corresponding output portions consists of concatenating them, this can be achieved by each core simply writing the output portion to the appropriate location in the shared OCM 314b. In other words, the "combine" of step 754 and the "write" of step 756 can be performed together and can be performed by each core immediately after the "evaluate" step 750.

[0155] Parameters controlling the element-wise operation of each input channel may be split and stored in corresponding different local OCMs 314a. Similarly, for other operations that may be split by input channel, parameters controlling the processing of each channel may be split into corresponding different local OCMs 314a.

[0156] If the layer group includes a convolutional layer with multiple output channels (filters), and the evaluation is split across the cores according to the output channels (F dimension), each core will be responsible for evaluating a given filter or filter group. The weights (and biases) of the relevant filters can be stored in the local OCM 314a of the corresponding core.

[0157] On the other hand, if the layer group includes convolutional layers, and the evaluation is split across cores along one of the traversal dimensions (e.g., the X dimension), the weights and biases are stored in the shared OCM 314b where they are accessible to all cores.

[0158] Some layers may be difficult, inefficient, or impossible to split across multiple cores. For example, it should be noted that Figure 16 Layers LG1, LG2, and LG4 in

[15] are not split across cores at all—they are all executed by a single core (possibly leaving other cores idle if no other work occupies them). This approach can be used, for example, to process a fully connected layer consisting of one filter.

[0159] Now it will be understood that Figure 15 and Figure 17 The methods can be advantageously combined. Figure 15 The method can be used to process large input data tensors (ie, if the size is above a threshold in step 714 of analyzing the size of the input data tensor). Figure 17 The method can be used to process small input data tensors (if the size is below the threshold in step 714'). In other words, the threshold used in step 714 and the threshold used in step 714' can be the same, so that the hardware implementation 600 follows Figure 15 or follow the Figure 17 method.

[0160] The processing system may include a hardware implementation as described above, or may be configured to perform the method as described above. The processing system may be an artificial intelligence accelerator system, such as a neural network accelerator (NNA), or a graphics processing system / graphics processing unit (GPU).

[0161] Figure 18 9. A computer system in which a processing system according to the present disclosure can be implemented is shown. The computer system includes a CPU 902, an NNA 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. A processing block 910 (corresponding to hardware implementation 600) is implemented on the NNA 904. In other examples, the processing block 910 can be implemented on the CPU 902. The components of the computer system can communicate with each other via a communication bus 920. A memory bank 912 (corresponding to DDR 312) is implemented as part of the memory 906.

[0162] Although Figure 18 One implementation of an artificial intelligence accelerator system is shown, but it should be understood that a similar block diagram can be drawn for a graphics processing system, for example, by replacing CPU 902 or NNA 904 with a GPU, or by adding a GPU as an additional unit. In this case, processing block 910 can be implemented in the GPU.

[0163] Figure 12 The hardware implementation 600 is shown as including a number of functional blocks. This is illustrative only and is not intended to define a strict division between the different logical elements of such an entity. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by the hardware implementation need not be physically generated by the hardware implementation at any point in time and may simply represent logical values ​​that conveniently describe the processing performed by the hardware implementation between its input and output terminals.

[0164] The hardware implementation described herein can be embodied in hardware on an integrated circuit. The hardware implementation described herein can be configured to perform any of the methods described herein. Generally speaking, any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry) or any combination thereof. The terms "module," "function," "component," "element," "unit," "block," and "logic" can be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of software implementation, a module, function, component, element, unit, block, or logic represents a program code that performs a specified task when executed on a processor. The algorithms and methods described herein can be performed by one or more processors of an execution code that causes the processor to execute the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disks, flash memory, hard disk storage, and other memory devices that can store instructions or other data using magnetic, optical, and other technologies and can be accessed by a machine.

[0165] As used herein, the terms computer program code and computer readable instructions refer to any type of executable code for execution by a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code that defines an integrated circuit (such as a hardware description language or netlist), and code written in languages ​​such as C, Executable code can be, for example, any type of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0166] A processor, computer, or computer system may be any kind of device, machine, or specialized circuit, or a collection or portion thereof, that has processing capabilities to execute instructions. A processor may be any kind of general-purpose or specialized processor, such as a CPU, GPU, NNA, system on a chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0167] The present invention is also intended to encompass software that defines the configuration of hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which is encoded computer-readable program code in the form of an integrated circuit definition data set that, when processed (i.e., executed) in an integrated circuit manufacturing system, configures the system to manufacture hardware implementations configured to perform any of the methods described herein, or to manufacture hardware implementations including any of the devices described herein. The integrated circuit definition data set may be, for example, an integrated circuit description.

[0168] Thus, a method of manufacturing a hardware implementation as described herein at an integrated circuit manufacturing system may be provided. Furthermore, an integrated circuit definition data set may be provided that, when processed in an integrated circuit manufacturing system, causes the method of manufacturing a hardware implementation to be performed.

[0169] The integrated circuit definition data set may be in the form of computer code, for example, as a netlist, code for configuring a programmable chip, as a hardware description language that defines hardware suitable for fabrication at any level in the integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation (such as Verilog or VHDL), and as a low-level circuit representation (such as OASIS (RTM) and GDSII). Higher-level representations that logically define hardware suitable for fabrication in the integrated circuit (such as RTL) may be processed on a computer system configured to generate a manufacturing definition of the integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate a manufacturing definition of the integrated circuit defined by the representation. As is typically the case when software is executed at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate a manufacturing definition of the integrated circuit in order to execute the code that defines the integrated circuit in order to generate the manufacturing definition of the integrated circuit.

[0170] Now refer to Figure 19 An example of processing an integrated circuit definition dataset at an integrated circuit fabrication system to configure the system to fabricate a hardware implementation is described.

[0171] Figure 19An example of an integrated circuit (IC) manufacturing system 1002 is shown that is configured to manufacture hardware implementations as described in any of the examples herein. In particular, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining a hardware implementation as described in any of the examples herein), process the IC definition dataset, and generate an IC based on the IC definition dataset (e.g., embodying a hardware implementation as described in any of the examples herein). Processing the IC definition dataset configures the IC manufacturing system 1002 to manufacture integrated circuits that embody the hardware implementations as described in any of the examples herein.

[0172] The layout processing system 1004 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout based on an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of the circuit to be generated, such as for logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user input to optimize the circuit layout. When the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition can be, for example, a circuit layout description.

[0173] As is known in the art, IC generation system 1006 generates an IC based on a circuit layout definition. For example, IC generation system 1006 may implement a semiconductor device manufacturing process to generate the IC, which may involve a multi-step sequence of photolithography and chemical processing steps during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask that can be used in a photolithography process to generate the IC based on the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code that IC generation system 1006 can use to generate a suitable mask for generating the IC.

[0174] The various processes performed by IC manufacturing system 1002 may all be performed at a single location, such as by one party. Alternatively, IC manufacturing system 1002 may be a distributed system, such that some processes may be performed at different locations and by different parties. For example, some of the following stages may be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) generating a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0175] In other examples, processing of an integrated circuit definition data set at an integrated circuit manufacturing system may configure the system to manufacture a hardware implementation without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set may define a configuration of a reconfigurable processor, such as an FPGA, and processing the data set may configure the IC manufacturing system to generate a reconfigurable processor having the defined configuration (e.g., by loading the configuration data into the FPGA).

[0176] In some embodiments, the integrated circuit manufacturing definition data set, when processed in the integrated circuit manufacturing system, can cause the integrated circuit manufacturing system to generate the device as described herein. Figure 19 Configuration of an integrated circuit fabrication system in the manner described may result in the fabrication of the apparatus described herein.

[0177] In some examples, the integrated circuit definition data set may include software that runs on, or in combination with, hardware defined at the data set. Figure 19 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset when manufacturing the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0178] Compared to known implementations, the implementation of the concepts set forth in this application in devices, equipment, modules and / or systems (and in the methods implemented herein) can result in performance improvements. Performance improvements can include one or more of improved computing performance, reduced latency, increased throughput and / or reduced power consumption. During the manufacture of such devices, devices, modules and systems (e.g., in integrated circuits), a trade-off can be made between performance improvements and physical implementations, thereby improving manufacturing methods. For example, a trade-off can be made between performance improvements and layout area, thereby matching the performance of known implementations, but using less silicon. For example, this can be accomplished by reusing functional blocks in a serial manner or sharing functional blocks between elements of a device, device, module and / or system. In contrast, the concepts of improvements (such as reduced silicon area) that result in physical implementations of devices, devices, modules and systems set forth in this application can be compromised for performance improvements. For example, this can be accomplished by manufacturing multiple instances of a module within a predefined area budget.

[0179] The applicant hereby independently discloses each individual feature described herein, as well as any combination of two or more such features, to the extent that such feature or combination can be implemented according to the common general knowledge of a person skilled in the art based on the present specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein. In view of the foregoing description, it will be clear to a person skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A method for implementing a deep neural network in multi-core hardware (600), the deep neural network comprising a plurality of layers, the multi-core hardware comprising at least a first core (350-0) and a second core (350-1), the method comprising: defining (710) a plurality of layer groups comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein output data of the first layer group comprises input data for the second layer group; providing (712) in one or more first memory devices (312) input data for the first layer group, the input data comprising a multidimensional tensor, the multidimensional tensor comprising one or more traversed dimensions and one or more non-traversed dimensions, the one or more traversed dimensions being dimensions traversed by strides in at least one layer of the first layer group; determining (714) a size of the input data in a first dimension among one or more traversal dimensions, If (716) the size in the first dimension is greater than a threshold, evaluating the first and second layer groups by depth-first tiling, including: partitioning (718) the input data for the first layer group along the first dimension into at least a first tile and a second tile, each tile including a plurality of data elements in each of the one or more traversal dimensions; evaluating ( 720 ) the first layer group based on the first tile using the first kernel to generate a first output tile of the first layer group; evaluating ( 720 ) the first layer group based on the second tile using the second kernel to generate a second output tile of the first layer group; storing (722, 724) the first output tile and the second output tile of the first layer group in one or more second memory devices (314) as input data for the second layer group; evaluating ( 720 ) the second layer group based on the first output tile of the first layer group using the first kernel to generate a first output tile of the second layer group; evaluating (720) the second layer group based on the second output tile of the first layer group using the second kernel to generate a second output tile of the second layer group, and If (746) the magnitude in the first dimension is not greater than the threshold, evaluating the first and second layer groups, one layer group at a time, in a multi-pass process comprising: splitting (748) the evaluation of the first layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension; evaluating (750) the first set of layers for the first propagation process using the first kernel to generate a first portion of output data; evaluating (750) the first set of layers for the second propagation process using the second kernel to generate a second portion of output data; generating the output data of the first layer group, comprising combining (754) the first portion of output data and the second portion of output data; and The output data of the first layer group is stored (756) in the one or more second memory devices as input data for the second layer group.

2. The method according to claim 1, further comprising: if the size is not greater than the threshold, splitting (748) the evaluation of the second layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension; assigning (749) the first propagation process of the second layer group to the first core, and assigning the second propagation process of the second layer group to the second core; evaluating (750) the second set of layers for the first propagation process using the first kernel to generate a first portion of output data; and The second set of layers is evaluated (750) for the second propagation process using the second kernel to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements.

3. The method according to claim 1 or 2, wherein: The plurality of layer groups include a third layer group including one or more consecutive layers of the neural network, the method further comprising storing (722, 724) the first output tile and the second output tile of the second layer group in the one or more second memory devices as input data for the third layer group if the size is greater than the threshold.

4. The method according to claim 1 or 2, wherein: If the size is greater than the threshold, then one or both of the following are true: (A) The method further includes storing the first output tile and the second output tile of a final layer group in the plurality of layer groups in the one or more first memory devices (312); and (B)1] Splitting (718) the input data for the first layer group includes splitting it into the first tile, the second tile and the third tile, the method further comprising evaluating (720) each layer group based on the third tile using the first kernel after evaluating (720) each layer group based on the first tile using the first kernel.

5. The method according to claim 1 or 2, wherein: The one or more second memory devices include: a local memory (314a) for the first core, accessible only by the first core; local memory (314a) for the second core, accessible only by the second core; and a shared memory (314b) accessible by the first core and the second core, The method further comprises, if the size is greater than the threshold, then: identifying a one-time portion of the first output tile of the first layer group that is only needed to evaluate the second layer group based on the first tile; and identifying overlapping portions of the first output tiles of the first layer group that are required to evaluate the second layer group based on the first tiles and that are also required to evaluate the second layer group based on the second tiles, The method includes, when storing the first output tile of the first layer group in the one or more second memory devices: writing (722) the one-time portion of the first output tile to the local memory of the first core; and writing (724) the overlapping portion of the first output tile to the shared memory, and optionally The first layer group includes a convolutional layer, and the method further includes storing weight values ​​and / or bias values ​​of the convolutional layer in the shared memory (314b).

6. The method according to claim 1 or 2, comprising, if the size is greater than the threshold, then: Waiting (728) for the first core to complete evaluation of the first layer group for the first tile, and Waiting (728) for the second core to complete evaluation of the first layer group for the second tile, The second layer group of the second tile is then evaluated (720) using the second kernel.

7. The method according to claim 1 or 2, wherein: The one or more traversal dimensions further include a second dimension, Wherein, the input data is stored in the one or more first memory devices first sorted according to the second dimension.

8. The method according to claim 1 or 2, wherein: If the size is not greater than the threshold, splitting (748) the evaluation of the first layer group includes splitting the evaluation into the first propagation process, the second propagation process, and a third propagation process, The method further comprises: assigning (749) the third propagation process to the first core; and After evaluating (750) the first set of layers for the first propagation process using the first kernel, the first set of layers is evaluated (750) for the third propagation process using the first kernel.

9. The method according to claim 1 or 2, wherein: The one or more second memory devices include: a local memory (314a) for the first core, accessible only by the first core; local memory (314a) for the second core, accessible only by the second core; and a shared memory (314b) accessible by the first core and the second core, The method includes storing the first portion of output data and the second portion of output data in the shared memory (314b) if the size is not greater than the threshold.

10. The method according to claim 9, wherein: or: (A) The first layer group includes a convolutional layer, and the method further includes: storing weight values ​​and / or bias values ​​of the convolutional layer in the shared memory (314b) if the evaluation is split along one of the traversal dimensions; and If the evaluation is split along one of the non-ergodic dimensions, splitting the weight values ​​and / or bias values ​​and storing the weight values ​​and / or bias values ​​in the local memory (314a), or (B) The evaluation is split according to input channels, wherein the first layer group does not include a convolutional layer, and wherein the method includes splitting parameters of the first layer group and storing the parameters in the local memory (314a).

11. A hardware implementation (600) of a deep neural network, comprising a plurality of layers arranged in a plurality of layer groups (LGs), the plurality of layer groups comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein: The output data of the first layer group includes input data for the second layer group, and the hardware implementation includes: one or more first memory devices (312) configured to store input data for the first layer group, the input data comprising a multidimensional tensor, the multidimensional tensor comprising one or more traversed dimensions and one or more non-traversed dimensions, the one or more traversed dimensions being dimensions traversed by strides in at least one layer of the first layer group; one or more second memory devices (314) configured to store at least the output data of the first layer group; a plurality of cores (350), including at least a first core (350-0) and a second core (350-1); and a splitter unit (620), wherein the splitter unit is configured to determine (714) a size of the input data in a first dimension among the one or more traversal dimensions, and If (716) said size in said first dimension is greater than a threshold, then The partitioner unit (620) is configured to partition (718) the input data for the first layer group along the first dimension into at least a first tile and a second tile, each tile comprising a plurality of data elements in each of the one or more traversal dimensions, The first kernel is configured to evaluate (720) the first layer group based on the first tile to generate a first output tile of the first layer group, The second kernel is configured to evaluate (720) the first layer group based on the second tile to generate a second output tile of the first layer group, The first core and the second core are configured to store the first output tile and the second output tile of the first layer group as input data for the second layer group in the one or more second memory devices (314). The first kernel is configured to evaluate (720) the second layer group based on the first output tile of the first layer group to generate a first output tile of the second layer group, and The second kernel is configured to evaluate (720) the second layer group based on the second output tile of the first layer group to generate a second output tile of the second layer group, And, if (746) said size in said first dimension is not greater than said threshold, then The partitioner unit is configured to partition (748) the evaluation of the first layer group into at least a first propagation process and a second propagation process along a dimension other than the first dimension, The first kernel is configured to evaluate (750) the first set of layers for the first propagation process to generate a first portion of output data, The second core is configured to evaluate (750) the first set of layers for the second propagation process to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements, The plurality of cores are configured to generate the output data of the first layer group, including combining (754) the first portion of output data and the second portion of output data, and The plurality of cores are configured to store (756) the output data of the first tier group in the one or more second memory devices as input data for the second tier group.

12. The hardware implementation according to claim 11, wherein: If (716) the size in the first dimension is greater than the threshold, the first core and the second core are configured to store the first output tile and the second output tile of the final layer group in the defined layer group in the one or more first memory devices (312).

13. The hardware implementation according to claim 11 or 12, wherein: If (716) the size in the first dimension is greater than the threshold, the splitter unit is configured to split (718) the input data for the first layer group into the first tile, the second tile and a third tile, and The first core is configured to evaluate ( 720 ) each layer group based on the third tile after having completed evaluating each layer group based on the first tile.

14. The hardware implementation according to claim 11 or 12, wherein: The one or more second memory devices include: a local memory (314a) for the first core, accessible only by the first core; local memory (314a) for the second core, accessible only by the second core; and a shared memory (314b) accessible by the first core and the second core, wherein the segmenter unit is configured to, if (716) the size in the first dimension is greater than the threshold, then: identifying a one-time portion of the first output tile of the first layer group that is only needed to evaluate the second layer group based on the first tile; and identifying overlapping portions of the first output tiles of the first layer group that are required to evaluate the second layer group based on the first tiles and that are also required to evaluate the second layer group based on the second tiles, and the first core is configured to, when storing the first output tile of the first layer group in the one or more second memory devices: writing (722) the one-time portion to its local memory; and The overlapping portion is written (724) to the shared memory.

15. The hardware implementation according to claim 11 or 12, wherein: The second core is configured to, if (716) the size in the first dimension is greater than the threshold, Wait (728) for the first core to complete evaluation of the first layer group based on the first tile, and Waiting (728) for the second core to complete evaluation of the first layer group based on the second tile, The second layer group is then evaluated (720) based on the second tile.

16. The hardware implementation according to claim 11 or 12, wherein: The one or more second memory devices include: a local memory (314a) for the first core, accessible only by the first core; local memory (314a) for the second core, accessible only by the second core; and a shared memory (314b) accessible by the first core and the second core, wherein if (746) the size in the first dimension is not greater than the threshold, the plurality of cores are configured to store (756) the first portion of output data and the second portion of output data in the shared memory (314b).

17. The hardware implementation according to claim 16, wherein: or: (A) The first layer group includes convolutional layers, and the segmenter unit (620) is configured to: If the evaluation is split along one of the traversal dimensions, storing the weight values ​​and / or bias values ​​of the convolutional layer in the shared memory (314b); and If the evaluation is split along one of the non-ergodic dimensions, splitting the weight values ​​and / or bias values ​​and storing the weight values ​​and / or bias values ​​in the local memory (314a), or (B) If the evaluation is segmented according to input channels (748) and the first layer group does not include a convolutional layer, the segmenter unit (620) is configured to segment the parameters of the first layer group and store them in the local memory (314a).

18. An artificial intelligence accelerator system, comprising a hardware implementation according to any one of claims 11 to 17, and / or configured to perform a method according to any one of claims 1 to 10.

19. A computer-readable storage medium having computer-readable code stored thereon, wherein the computer-readable code is configured to cause the method according to any one of claims 1 to 10 to be executed when the code is executed.

20. A computer-readable storage medium having stored thereon an integrated circuit definition dataset, which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the artificial intelligence accelerator system of claim 18.

Images