Implementation of neural networks in multi-core hardware

Abstract

This invention discloses an implementation of a neural network in multi-core hardware. Specifically, it discloses a multi-core hardware implementation of a deep neural network. Multiple layers of the network are arranged in multiple layer groups. The network's input data includes a multidimensional tensor comprising one or more traversal dimensions and one or more non-traversal dimensions, these traversal dimensions being dimensions traversed by the stride of at least one layer in a first layer group. The hardware implementation is configured to partition the evaluation of the first layer group into at least a first propagation process and a second propagation process along one of the traversal dimensions or one of the non-traversal dimensions. A first kernel is configured to evaluate the first layer group against the first propagation process to generate a first portion of output data. A second kernel is configured to evaluate the first layer group against the second propagation process to generate a second portion of output data. The hardware implementation is configured to combine the first portion and the second portion of the output data to produce the output data of the first layer group.

Description

Technical Field

[0001] This invention relates to the implementation of neural networks in multi-core hardware. Background Technology

[0002] Deep neural networks (DNNs) are artificial neural networks that can be used in machine learning applications. In particular, DNNs can be used in signal processing applications, including image processing and computer vision applications.

[0003] DNNs have been implemented in applications where power resources are not a critical factor. Nevertheless, DNNs are used in many different technical fields where the hardware resources used to implement them are limited by power consumption, processing power, or silicon area. Therefore, there is a need for hardware configured to implement DNNs (or at least a portion thereof) in an efficient manner, such as requiring less silicon area or less processing power during operation. Furthermore, DNNs can be configured in many different ways for various applications. Therefore, there is also a need for hardware that offers the flexibility to support diverse DNN configurations. As DNNs become more structurally complex and require processing larger volumes of data in shorter time intervals (e.g., for inference in real-time applications), there is an increasing demand for hardware with greater processing power while maintaining or improving area and power efficiency. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, a series of concepts further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] A multi-core hardware implementation of a deep neural network is disclosed. Multiple layers of the network are arranged in multiple layer groups. The network input data includes a multidimensional tensor comprising one or more traversal dimensions and one or more non-traversal dimensions, the traversal dimensions being dimensions traversed by the stride of at least one layer in the first layer group. The hardware implementation is configured to partition the evaluation of the first layer group into at least a first propagation process and a second propagation process along one of the traversal dimensions or one of the non-traversal dimensions. A first kernel is configured to evaluate the first layer group against the first propagation process to generate a first portion of output data. A second kernel is configured to evaluate the first layer group against the second propagation process to generate a second portion of output data. The hardware implementation is configured to combine the first portion and the second portion of the output data to produce the output data of the first layer group.

[0006] According to one aspect, a method for implementing deep neural networks in multi-core hardware is provided.

[0007] According to this method, layers of a neural network are evaluated by dividing the computation into a propagation process whose evaluation is distributed across multiple kernels. Dividing the evaluation into a propagation process along ergonomic or non-ergonomic dimensions allows layers to be processed in parallel, one layer at a time.

[0008] Evaluating a layer group means evaluating each layer of the group. A layer group can be defined such that all layers of a single layer group can be evaluated in a single hardware pipeline. For example, all layers of a single layer group can be evaluated in a hardware pipeline of a given core without writing any data to one or more second memory devices or one or more first memory devices. In some cases, a layer group can consist of a single layer of a neural network. In other cases, a layer group can consist of multiple layers. For example, a layer group can include any one or any two or more of the following: a single convolutional layer, a single pooling layer, a single activation layer, a single normalization layer, and a single element-wise operation layer. Optionally, a layer group can include at most one convolutional layer.

[0009] Accessing one or more second memory devices can be less expensive than accessing one or more first memory devices. The cost may be in terms of time or power consumption. For example, one or more first memory devices can be provided as off-chip memory; one or more second memory devices can be provided as on-chip memory.

[0010] The traversal dimension can include an X dimension (columns) and a Y dimension (rows), as well as an optional batching dimension B. The input data includes one or more additional non-traversal dimensions, such as the channel dimension C. The input data can be processed by multiple filters to produce a corresponding number of output channels F.

[0011] After multiple parallel evaluations of the first layer group, the tensor (the output data of the first layer group) is reassembled in one or more second memory devices. The second layer group and any subsequent layer groups may be partitioned differently from the first layer group, or may not be partitioned at all.

[0012] Segmentation can be planned before the training or inference phase of a neural network begins. Segmentation can be pre-planned based on the parameters of the layers in each layer group and the tensor sizes at the input and output of each layer.

[0013] Deep neural networks can be convolutional neural networks.

[0014] The method may further include: partitioning the evaluation of the second layer group into at least a first propagation process and a second propagation process along one of the traversed dimensions or one of the non-traversed dimensions; assigning the first propagation process of the second layer group to a first kernel, and assigning the second propagation process of the second layer group to a second kernel; evaluating the second layer group against the first propagation process using the first kernel to generate a first portion of output data; and evaluating the second layer group against the second propagation process using the second kernel to generate a second portion of output data, each of the first portion and the second portion of the output data comprising a plurality of data elements. The evaluation of the first layer group may be partitioned along a first dimension, and the evaluation of the second layer group may be partitioned along a second dimension. The first dimension and the second dimension may be different dimensions.

[0015] This approach is flexible because different layers do not need to be partitioned along the same dimensions. Given hardware capabilities, the partitioning can be chosen based on what is most efficient for the layer group in question.

[0016] The first layer group can be segmented according to the input channel, thereby the first propagation process includes evaluating the first layer group for the first input channel, and the second propagation process includes evaluating the first layer group for the second input channel.

[0017] The input channel dimension is an example of a non-traversal dimension. In some cases, the first propagation process includes evaluating a first-level group for a first set of input channels, and the second propagation process includes evaluating a first-level group for a second set of input channels.

[0018] The first layer group can be segmented according to the output channel, thereby the first propagation process includes evaluating the first layer group for the first output channel, and the second propagation process includes evaluating the first layer group for the second output channel.

[0019] Different output channels are sometimes referred to as being generated by different "filters". The output channel dimension is another example of a non-ergodic dimension. In some cases, the first propagation process involves evaluating a first layer group for a first set of output channels, and the second propagation process involves evaluating a first layer group for a second set of output channels.

[0020] The method may further include storing the output data of the final layer group from a plurality of layer groups in one or more first memory devices. In this manner, intermediate data generated during the evaluation of layer groups is stored in one or more second memory devices, and the output data generated by evaluating the final layer group is stored in one or more first memory devices.

[0021] The method may further include: analyzing at least one characteristic of the input data for the first layer group; and determining, based on the analysis, that the input data should be segmented into a propagation process along said one dimension, wherein the segmentation of the input data for the first layer group is performed in response to the determination. The at least one characteristic may include the size of the input data in at least one traversed dimension, and determining that the input data should be segmented into a propagation process may include determining that the size in at least one traversed dimension is not greater than a threshold size.

[0022] If the size in at least one traversal dimension is greater than the threshold size, the evaluation (and input data) of the layer group can be split along at least one traversal dimension.

[0023] On the other hand, if the size in at least one traversal dimension is not greater than a threshold size, the evaluation of the layer group can be partitioned along one of the other dimensions. In other words, while the processing of larger tensors can be partitioned across kernels along one or more traversal dimensions, smaller tensors can be partitioned across kernels along other dimensions such as batch, input channel, or output channel dimensions (which can be non-traversal dimensions).

[0024] There is a synergy between these two approaches. If the tensor is large, it can be partitioned into tiles along at least one traversal dimension and processed depth-first within these tiles. This reduces the amount of data that needs to be stored in one or more second memory devices, as it may not be necessary to store the entire intermediate tensor in them simultaneously. On the other hand, if the tensor is small, it can be partitioned into multiple propagation processes. In this case, even if it is necessary to store the entire intermediate tensor in one or more second memory devices, this may not be a limiting factor, since the tensor starts small. In other words, this approach allows for efficient utilization of parallel hardware regardless of whether the input tensor is small or large.

[0025] The evaluation of segmenting a first-layer group may include dividing it into a first propagation process, a second propagation process, and a third propagation process. The method may also include: assigning the third propagation process to a first core; and evaluating the first-layer group against the third propagation process using the first core after evaluating the first-layer group against the first propagation process. This can be done when the layer group is segmented into multiple propagation processes that are more numerous than the number of cores in a multi-core hardware system.

[0026] The input data for the first layer group may include a first traversal dimension and a second traversal dimension, wherein the input data is stored in one or more first memory devices that are first sorted according to the first dimension, and wherein the evaluation of the first layer group is segmented along the first dimension.

[0027] Evaluations can be segmented in this way, especially if the input data is not large enough in the second dimension to allow for segmentation in that dimension. For example, the input data can be stored row-majorly, in other words, sorted according to dimension X. In this case, the evaluation of the first-level group can be segmented as a propagation process along dimension X. Alternatively, the input data can be stored column-majorly (first sorted according to dimension Y), and the evaluation of the first-level group can be segmented as a propagation process along dimension Y.

[0028] One or more second memory devices may include: local memory of a first core, accessible only by the first core; local memory of a second core, accessible only by the second core; and shared memory accessible by both the first and second cores, the method optionally including storing a first portion and a second portion of output data in the shared memory.

[0029] The first layer group may include convolutional layers, and the method may further include: if the evaluation is performed along a segment in an traversal dimension, storing the weight values ​​and / or bias values ​​of the convolutional layers in shared memory; or if the evaluation is performed along a segment in a non-traversal dimension, segmenting the weight values ​​and / or bias values ​​and storing them in local memory.

[0030] When evaluating a segment along a traversal dimension, the same weights / biases need to be applied to each part of the data, and therefore each core needs access to all the same weights / biases. On the other hand, if the evaluation is based on a segmentation of the layer group's input or output channels, each core may handle one or more input or output channels, while other cores handle one or more other input or output channels. Therefore, the weights / biases associated with a given input or output channel can be stored locally in the cores that need to use them.

[0031] The evaluation can be segmented according to the input channels, wherein the first layer group does not include convolutional layers, and wherein the method includes the parameters for segmenting the first layer group and storing them in local memory.

[0032] Parameters may include controls for pooling, normalization, or element-wise operations. For example, for the corresponding input channel, parameters may include stride and / or window size.

[0033] It also provides hardware implementations of deep neural networks.

[0034] Each core may include multiple processing elements configured to process input data to evaluate a layer of neural network.

[0035] Multiple cores can be configured to store the output data of the final layer group in one or more first memory devices.

[0036] The segmenter unit can be configured to: analyze at least one characteristic of the input data for a first layer group; and determine, based on the analysis, that the input data should be segmented into a propagation process along said one dimension, wherein the segmenter unit is configured to segment the input data for the first layer group along said one dimension in response. The at least one characteristic may include the size of the input data in at least one traversal dimension, wherein the segmenter unit is configured to determine that the input data should be segmented into a propagation process by determining that the size in the at least one traversal dimension is not greater than a threshold size.

[0037] One or more second memory devices may include: local memory of a first core, accessible only by the first core; local memory of a second core, accessible only by the second core; and shared memory accessible by both the first and second cores. Multiple cores may be configured to store a first portion and a second portion of the output data in the shared memory.

[0038] The first layer group may include convolutional layers. The segmenter unit may be configured to: if the evaluation is segmented along one of the traversal dimensions, store the weights and / or biases of the convolutional layers in shared memory; and if the evaluation is segmented along one of the non-traversal dimensions, segment the weights and / or biases and store them in local memory. Alternatively, if the evaluation is segmented according to the input channels, and the first layer group does not include convolutional layers, the segmenter unit may be configured to segment the parameters of the first layer group and store them in local memory.

[0039] An artificial intelligence accelerator system is also provided, comprising the hardware implementation described above. An artificial intelligence accelerator system is also provided, configured to perform the methods described above. Such an artificial intelligence accelerator system can be implemented in hardware on an integrated circuit.

[0040] A method for manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system is also provided.

[0041] A method for manufacturing an artificial intelligence accelerator system using an integrated circuit manufacturing system is also provided, the method comprising: processing a computer-readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacturing the artificial intelligence accelerator system using an integrated circuit generation system based on the circuit layout description.

[0042] A computer-readable code is also provided, configured to cause a method to be executed when the code is run. A computer-readable storage medium is also provided on which the computer-readable code is encoded.

[0043] An integrated circuit definition dataset is also provided, which configures the integrated circuit manufacturing system to manufacture an artificial intelligence accelerator system when it is processed in the integrated circuit manufacturing system.

[0044] A non-transitory computer-readable storage medium is also provided, on which a computer-readable description of an artificial intelligence accelerator system is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the artificial intelligence accelerator system.

[0045] A non-transitory computer-readable storage medium is also provided, on which a computer-readable description of an artificial intelligence accelerator system is stored, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process the computer-readable description of the artificial intelligence accelerator system using a layout processing system to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture the artificial intelligence accelerator system using an integrated circuit generation system based on the circuit layout description.

[0046] An integrated circuit manufacturing system is also provided, which is configured to manufacture an artificial intelligence accelerator system.

[0047] An integrated circuit manufacturing system is also provided, comprising: a non-transitory computer-readable storage medium storing a computer-readable description of an artificial intelligence accelerator system thereon; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and an integrated circuit generation system configured to manufacture the artificial intelligence accelerator system according to the circuit layout description.

[0048] The layout processing system can be configured to determine the location information of the logic components of a circuit derived from an integrated circuit description in order to generate a circuit layout description of an integrated circuit that embodies an artificial intelligence accelerator system.

[0049] Computer program code for performing any of the methods described herein may be provided. A non-transitory computer-readable storage medium may be provided, on which computer-readable instructions are stored, which, when executed at a computer system, cause the computer system to perform any of the methods described herein.

[0050] As will be apparent to those skilled in the art, the above features can be appropriately combined, and can be combined with any aspect of the examples described herein. Attached Figure Description

[0051] The example will now be described in detail with reference to the accompanying drawings, in which:

[0052] Figure 1 This is a schematic diagram of an exemplary deep neural network (DNN);

[0053] Figure 2 This is a schematic diagram of exemplary data in a DNN;

[0054] Figure 3 This is a block diagram of an exemplary hardware implementation of a convolutional layer in a DNN;

[0055] Figure 4 yes Figure 3 A block diagram of an exemplary convolutional engine;

[0056] Figure 5 This is a schematic diagram illustrating the data flow according to an exemplary hardware implementation.

[0057] Figure 6 This shows the data dependencies across multiple layers of the DNN for the first block of data;

[0058] Figure 7 The second block of data is shown. Figure 6 Data dependencies within multiple layers;

[0059] Figure 8 It is a concept map of overlapping data concepts;

[0060] Figure 9 An exemplary allocation of memory associated with the hardware implementation of a DNN is illustrated schematically;

[0061] Figure 10 A DNN with four layers and no branches is shown;

[0062] Figure 11A An exemplary grouping of layers in a DNN with four layers having branches is shown;

[0063] Figure 11B Showing the target Figure 11A An exemplary grouping of alternatives to DNNs;

[0064] Figure 12 This is a block diagram based on the example of a multi-core hardware implementation.

[0065] Figure 13 This illustrates the process according to the first aspect: when the input data is segmented into tiles and these tiles are processed in a depth-first manner... Figure 12 An example of execution order in multi-core hardware;

[0066] Figures 14A to 14B This illustrates synchronization between cores in another example, where the input data is segmented into tiles and processed in a depth-first manner;

[0067] Figure 15 This is a flowchart illustrating a method for implementing a convolutional neural network in multi-core hardware when input data is segmented into tiles;

[0068] Figure 16 An example of the execution order is shown when the evaluation of a layer is divided into a propagation process according to the second aspect;

[0069] Figure 17 This is a flowchart illustrating how to implement a convolutional neural network in multi-core hardware when the evaluation of a layer is divided into a propagation process;

[0070] Figure 18 A computer system in which a graphics processing system is implemented is shown; and

[0071] Figure 19 A manufacturing system for generating integrated circuits that embody a graphics processing system is shown.

[0072] The accompanying drawings illustrate various examples. Those skilled in the art will understand that the element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the drawings represent one example of a boundary. In some examples, it may be that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Implementation

[0073] The following description is given by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be readily apparent to those skilled in the art. Schemes are now described by way of example only.

[0074] A deep neural network (DNN) is an artificial neural network that consists of multiple interconnected layers that enable the DNN to perform signal processing tasks, including but not limited to computer vision tasks. Figure 1 An exemplary DNN 100 is shown, comprising multiple layers 102-1, 102-2, and 102-3. Each layer 102-1, 102-2, and 102-3 receives input data and processes that input data according to the layer to produce output data. The output data is either provided to that layer as input data to another layer or output as the final output data of the DNN. For example, in Figure 1In the DNN 100, the first layer 102-1 receives the raw input data 104 from the DNN 100 and processes the input data to produce output data. The output data of the first layer 102-1 becomes the input data of the second layer 102-2, which processes the input data to produce output data. The output data of the second layer 102-2 becomes the input data of the third layer 102-3, which processes the input data to produce output data. The output data of the third layer 102-3 is output as the output data 106 of the DNN.

[0075] The processing performed on the input data of a layer depends on the type of layer. For example, each layer of a DNN can be one of several different types. Exemplary DNN layer types include, but are not limited to, convolutional layers, activation layers, normalization layers, pooling layers, and fully connected layers. It will be apparent to those skilled in the art that these are exemplary DNN layer types and that this is not an exhaustive list, and that other DNN layer types may exist.

[0076] Convolutional layers are configured to convolve the input data using weights associated with that layer. Specifically, each convolutional layer is associated with multiple weights w1…w i Relatedly, these weights can also be referred to as filter weights or coefficients. Weights can be grouped to form or define one or more filters or kernels.

[0077] refer to Figure 2 This figure illustrates an example of the data format 200 used in a DNN. (As shown in...) Figure 2 As can be seen, the data 200 used in a DNN can be arranged into p data planes, where each plane has a size of x×y. A DNN may include one or more convolutional layers, where each convolutional layer is associated with multiple filters, each filter including multiple weights. Each filter has a size of m×n×p (i.e., each filter includes a set of m×n×p weights w), and is applied to the input data according to a convolution operation spanning several steps (called strides) s and t, such as... Figure 2 As shown. The group of data elements to which filters are applied at each stride is called a window. Each filter produces an output plane. The number of filters and the number of weights for each filter can vary between convolutional layers. Convolutional Neural Networks (CNNs) are an effective method for image recognition and classification; they are a special type of DNN and generally consist of multiple convolutional layers.

[0078] Before describing multi-core hardware implementations of the subject matter of this invention, a single-core example will be described. A (single-core) hardware implementation of a convolutional layer may include a hardware module or block (referred to herein as a "processing element") configured to compute the sum of the products between all or a portion of the weights forming a filter and all or a portion of the input data values ​​forming a window (referred to as a filter window computation). Since a large number of such filter window computations are typically performed, some hardware implementations may include multiple such processing elements, allowing more than one filter window computation to be performed in parallel. Preparing each processing element to perform a filter window computation involves reading the appropriate input data and weights for each filter window computation from one or more first memory devices and providing them to one of the processing elements. Typically, a large amount of data needs to be transferred from memory to the processing element. If this operation is not performed efficiently, it can result in high memory bandwidth requirements and high power consumption for providing input data and weights to the processing element. This is especially true when the memory is "off-chip" memory, i.e., implemented in a different integrated circuit or semiconductor die than the processing element.

[0079] Therefore, this paper describes hardware implementations of neural networks and methods for processing data in such hardware implementations, wherein input data and weights are provided to the processing element in an efficient manner to reduce the memory bandwidth and power consumption required to provide the input data and weights to the processing element. Specifically, data can be processed in blocks (also called “tiles”) along the depth direction, through multiple layers of the neural network. Intermediate outputs of layers within multiple layers can be stored in one or more second memory devices. The size of the one or more second memory devices may be smaller than the size of the one or more first memory devices used for bulk data storage, but can be accessed in a faster and / or more energy-efficient manner. In one example, at least one of the one or more second memory devices is disposed on a chip (i.e., in the same integrated circuit or semiconductor die as the processing element).

[0080] Now for reference Figure 3 This figure illustrates an exemplary single-core hardware implementation 300 of a convolutional layer in a DNN. Hardware implementation 300 includes multiple convolutional engines 302, multiple accumulators 304, an accumulation buffer 306, a coefficient buffer 308, and an input buffer 310. Each convolutional engine 302, along with its corresponding accumulator 304 and its share of resources in the accumulation buffer 306, represents a processing element 318. The hardware implementation also includes double data rate (DDR) SDRAM 312 as off-chip memory; on-chip memory (OCM) 314; and an output buffer 316.

[0081] Each convolutional engine 302 includes hardware logic configured to receive a set of weights (e.g., {w1…w8}) representing all or a portion of a filter and a set of input data values ​​(e.g., {d1…d8}) representing all or a portion of a window, and to perform multiplication-accumulation calculations on the received weights and input data values, such as... Figure 4 As shown. In some examples, such as Figure 4 As shown, each convolutional engine 302 may include multiple multipliers 402, each multiplier being configured to multiply the weights (w... i ) and the corresponding input data value (d) i The multipliers 402 and 404 are multiplied to produce a multiplicative output value. Following multiplier 402 are multiple adders 404 forming an adder tree to calculate the sum of the multiplicative outputs. Figure 4 In the example, the convolution engine 302 includes eight multipliers 402, but in other examples, there may be more or fewer multipliers. For example, in some cases, there may be 128 multipliers. Typically, if there are Z multipliers 402, the adder tree includes Z-1 adders 404.

[0082] In some cases, multiplication-accumulation computations are pipelined. For example, multiplier 402 and adder 404 may be divided into multiple pipeline stages, with a register stage (not shown) preceding the first pipeline stage and between each pair of pipeline stages. For example, the multiplier may form the first pipeline stage, and the adder may be divided into layers, with subsequent pipeline stages comprising one or more adder layers.

[0083] Figure 3 An exemplary hardware implementation 300 includes four convolution engines 302; however, it will be apparent to those skilled in the art that the methods and principles described herein are applicable to hardware implementations with any number of convolution engines.

[0084] Because the convolution engine may require more than one hardware propagation process to generate the complete filter result (e.g., because the convolution engine may only receive and process a portion of the filter weights and / or a portion of the input data values ​​of the window in a loop), the hardware implementation may include multiple accumulators 304. Each accumulator 304 receives the output of a convolution engine 302 and adds that output to a previous convolution engine output associated with the same filter. Since the convolution engine may not generate or produce outputs associated with the same filter in consecutive loops, partial results of one or more filters may be stored in an accumulation buffer 306, which can then provide the appropriate partial results to the accumulator in each loop. In some examples, the accumulation buffer 306 may be able to store partial results associated with 128 different filters.

[0085] The coefficient buffer 308 includes: a memory (not shown) for storing multiple weights associated with convolutional layers (or fully connected layers) of the DNN; and hardware logic (not shown) for providing the weights to the convolutional engine 302 for processing in a predetermined order across multiple loops. The multiple weights may include all weights associated with the convolutional layers, or only a portion of the weights associated with the convolutional layers. Although the coefficient buffer 308 is shown as a single module, it can be implemented, for example, by multiple coefficient buffers, each forming a memory volume.

[0086] The input buffer 310 includes: a memory (not shown) for storing multiple input data values ​​associated with convolutional layers (or fully connected layers) of the DNN; and hardware logic (not shown) for providing the input data values ​​to the convolutional engine 302 for processing in a predetermined order across multiple loops. The multiple input data values ​​may include all input data values ​​associated with the convolutional layers, or only a subset of the input data values ​​associated with the convolutional layers. Although in Figure 3 The input buffer 310 is shown as a single module, but for example, the input buffer 310 may be implemented by multiple input buffers, each forming a memory bank.

[0087] DDR 312 is coupled to on-chip memory 314 for providing weight data to on-chip memory 314. DDR 312 is also coupled to input buffer 310 for providing blocks of input data to input buffer 310. On-chip memory 314 is coupled to coefficient buffer 308 for providing weight data to coefficient buffer 308. On-chip memory 314 is also coupled to input buffer 310 for providing intermediate output data (including input data for subsequent layers) to input buffer 310. On-chip memory 314 is also coupled to DDR 312 for providing blocks of output data from the end layer of multiple layers to DDR 312. Accumulation buffer 306 is coupled to output buffer 316 to allow the output buffer to receive intermediate output data from multiple layers, as well as output data from the end layer. Output buffer 316 is coupled to on-chip memory 314 for providing intermediate output data and output data from the end layer to on-chip memory 314.

[0088] exist Figure 3 The examples illustrate various connections; however, in some implementations, some or all of them may be provided by one or more shared bus connections. It should also be understood that other connections may be provided as... Figure 3Alternatives or supplements to the connections shown. For example, output buffer 316 can be coupled to DDR 312 to provide output data directly to DDR 312. As a further example, DDR 312 can be coupled to coefficient buffer 308 to provide weight data directly to coefficient buffer 308. Similarly, in some cases, not... Figure 3 All the connections shown are necessary. For example, the DDR 312 does not always need to be coupled to the input buffer 310, which can instead obtain input data from the DDR 312 via the on-chip memory 314.

[0089] Figure 5 This is a simplified diagram to better illustrate the situation based on similar... Figure 3 A data stream is an example of a hardware implementation. Figure 5 The diagram illustrates multiple layers of a neural network (consisting of two layers 502-1 and 502-2). These can be any two consecutive layers of the network. It should also be noted that multiple layers are not limited to two layers. The diagram shows input data 504 to the first layer 502-1 of the multiple layers. Intermediate output data 506 of the first layer 502-1 forms the input data to the second layer 502-2 (the ending layer in this example of two layers). The diagram also shows the output data 508 of the second layer 502-2 (the ending layer). DDR 312 is a first memory device with relatively high read / write costs. On-chip memory 314 is a second memory device with relatively low read / write costs. For example, reading / writing a given amount of data from DDR can consume more time and energy compared to reading / writing from on-chip memory 314. It should be understood that DDR memory is merely one example of a high-capacity memory storage device, and other memory / storage technologies, such as flash memory, can also be used.

[0090] according to Figure 3 and Figure 5 For example, DDR 312 stores input data for the first layer 502-1 of multiple layers 502-1, 502-2. DDR 312 also stores weight data representing the weights of the multiple layers 502-1, 502-2. In this example of the invention, the weight data is transferred from DDR 312 to on-chip memory 314. Coefficient buffer 308 reads the weight data from on-chip memory 314 and provides the weight data to processing element 318. Input buffer 310 reads a first subset of input data representing a first block of input data 504 from DDR 312. This first subset of input data 504 (also referred to as a "header block") is... Figure 5The data is shown in gray shading and may include one or more data planes. Input buffer 310 provides a first subset of input data 504 to processing element 318. Processing element 318 processes the weight data and the first subset of input data 504 to evaluate each of the multiple layers, thereby computing a first block 508 of output data for the final layer. The first block 508 of output data is... Figure 5 Shown in gray shades, it may include one or more data planes. Output buffer 316 can write a first block of output data 508 to on-chip memory 314. The first block of output data 508 can be transferred from the on-chip memory to DDR 312. Alternatively, the output buffer can write output data directly to DDR memory 312 (e.g., ...). Figure 3 and Figure 5 (As shown by the dashed arrow in the image).

[0091] like Figure 5 As shown, the data input to and output from each layer is arranged in one or more planes (also referred to in this paper as the input dataset and output dataset). Figure 5 In the example shown, input data 504 includes p in = 3 data planes. The intermediate output data 506 of the first layer 502-1 includes p1 = 3 data planes. The output data 508 of the second layer 502-2 includes p = 2 data planes. The number of planes depends on the weight data. Specifically, the weights constituting each filter take a predetermined number of input planes as input data, and each filter outputs a plane of output data. Therefore, the number of data planes generated by a given layer is equal to the number of filters applied to that layer. Each data plane can represent a feature map. Figure 5 In the example, the three planes of input data 504 can represent three feature maps generated by an earlier layer of the neural network (if the first layer 502-1 in the figure is not the first layer of the neural network). Alternatively (if the first layer 502-1 is the first layer of the neural network), the three planes of input data 504 can represent three input data planes. For example, input data with multiple planes often occurs in image processing tasks. Such input planes can include, but are not limited to: red, green, and blue (RGB) planes; or hue, saturation, and lightness (HSV) planes. Input planes can also include depth data as one of the input planes.

[0092] To compute the first block of output data, processing element 318 must evaluate each of the multiple layers (as described above). Specifically, coefficient buffer 308 reads the weight data of the first layer 502-1 from on-chip memory 314. Processing element 318 processes the weight data of the first layer 502-1 and a first subset of the input data 504 to evaluate the first layer 502-1 among the multiple layers, thereby computing the output data 506 of the first layer. This is a block of output data of the first layer 502-1 corresponding to the first block of input data 504. In other words, it is a subset of the complete set of output data of the neural network of the first layer 502-1. This block / subset is in Figure 5 Shown in gray shading. Output buffer 316 writes a block of output data from layer 502-1 to on-chip memory 314. This represents intermediate output data that will be needed as input data to the next layer (layer 2 502-2). To evaluate the next layer (layer 2 502-2), input buffer 310 reads a block of output data 506 from layer 1 502-1 stored in on-chip memory 314 by output buffer 316. Coefficient buffer 308 reads weight data for layer 2 502-2 from on-chip memory 314. Processing element 318 processes the weight data for layer 2 502-2 and the block of output data 506 from the previous (first) layer to evaluate layer 502-2, thereby calculating the output data for layer 2 502-2. For any subsequent inner layer of multiple layers, this process continues—each time intermediate output data is stored in on-chip memory 314 and read to provide input data to the next layer. This continues until the last layer is reached. Figure 5 In the example, the second layer 502-2 is the end layer. Therefore, the output of the second layer 502-2 is the first block of output data 508. As explained above, the output buffer 316 can write the first block of output data 508 into the on-chip memory 314. The first block of output data 508 can then be transferred from the on-chip memory to DDR 312. Alternatively, the output buffer 316 can write the output data directly into DDR 312 (dashed arrow).

[0093] In this way, processing a subset of input data (tiles) through multiple layers of the neural network can efficiently utilize the processing resources of processing element 318. Writing intermediate output data to and reading it from on-chip memory 314 to process the next layer reduces the need for reading and writing data from off-chip DDR 312, thus helping to reduce power consumption. It should be noted that on-chip memory 314 may be much smaller than off-chip DDR 312 because during each propagation, on-chip memory 314 only needs to store data associated with the current block of input data, the current block of output data, and the associated blocks of intermediate output data for inner layers. This contrasts sharply with conventional methods of evaluating neural network layers, which process each layer as a whole independently, requiring memory capable of storing all input and all output data of a layer at any given time. This requires a much larger memory, which is often impractical to implement as on-chip memory. Therefore, conventional methods in practice require the use of off-chip memory to store intermediate output data. Since off-chip read and write operations typically consume more energy than on-chip read and write operations, reducing the amount of data written to off-chip memory can significantly save power consumption.

[0094] The size of the block to be processed in each propagation can be selected based on the available storage capacity of the on-chip memory 314, the number of layers in multiple layers, and the structure of the neural network. An exemplary method of memory management in the on-chip memory 314 will be described below. First, it is useful to consider how the blocks of output data from multiple layers in a neural network depend on the intermediate output data of previous layers and the input data of the first layer. (Reference) Figure 6 This is understandable. The figure illustrates the input data of three layers (not explicitly shown) of a neural network; the output data of the first layer (labeled layer 0 here); the output data of the second layer (labeled layer 1); and the output data of the final layer. For simplicity, the output data of the final layer can conveniently be viewed as a single feature map created by a single filter. Similarly, the other layers can be viewed as convolutional layers with single filters, thus creating single output feature maps, as shown in a simplified form in the figure. Of course, the scope of this disclosure is not limited to this approach; these are merely simplifications for ease of understanding. At each layer, a single filter can be applied to multiple input channels (feature maps), or multiple filters can be applied to one or more input channels to process and / or generate multiple data planes. The methods and hardware implementations according to this disclosure can be used equivalently in such scenarios.

[0095] Starting with the first block of output data (patch 1) in the final layer, we can trace back through the network to find the intermediate and input data that this output block depends on. There is a block of data (patch 1) in the layer 1 output that corresponds to patch 1 in the final layer; in this sense, patch 1 in layer 1 is needed to compute patch 1 in the final layer. Assume the filter kernel size is greater than 1. Due to this kernel size, patch 1 in the layer 1 output is larger than patch 1 in the final layer output. In other words, patch 1 in the final layer output depends on some additional data in the layer 1 output. For example, for a 3x3 kernel and a stride of 1, an extra row of output data is needed in the layer 1 output. Figure 6 The dashed lines in the diagram indicate this expansion of the required data volume. In this figure, each tile is separated by a solid line.

[0096] Next, consider which data in the layer 0 output is necessary to compute tile 1 in the layer 1 output. For diversity, we will assume that the stride of layer 1 is greater than 1. This means that the layer 1 output is smaller than the layer 0 output. There is a tile in layer 0, namely tile 1, which corresponds to tile 1 in layer 1. Since the kernel size is also greater than 1, the amount of data required at the boundaries increases further. Therefore, tile 1 in the layer 0 output is larger than tile 1 in the layer 1 output.

[0097] The same situation occurs when traversing back from layer 0 to the input data. The input data contains a block, namely tile 1, corresponding to tile 1 in layer 0. Assume the stride in layer 0 is 1; therefore, the size of the input data is approximately the same as the size of the layer 0 output (depending on padding). Similarly, since the kernel size is greater than 1, additional data is needed at the boundaries. Therefore, tile 1 in the input data is also larger than tile 1 in the layer 0 output. Tile 1 in the input data forms the first subset (tile 1) of the input data. Tile 1 in the output data of the final layer forms the output block.

[0098] As can be inferred from the above, the total amount of data required for the first tile increases sequentially from the end tile to the input tile. This depends specifically on the parameters of all intermediate layers (kernel size, stride, and expansion). Figure 6 The diagram also illustrates "overlap." This is data that can be reused in subsequent propagation processes, i.e., data reused when computing subsequent tiles. Overlap data is indicated by horizontal dashed lines in the data of each layer. Overlap 1 in the output of layer 1 is a portion of tile 1 and can be (repeatedly) used to compute the subsequent tile (tile 2) in the output of the final layer. Overlap 1 in the output of layer 0 is a portion of tile 1 and can be (repeatedly) used to compute tile 2 in the output of layer 1. The size of the overlap depends only on the parameters of the next layer (kernel size, dilation). For example, if stride = 1 and dilation = 1, the size of the overlap (the number of overlapping rows) is the kernel size minus 1.

[0099] Immediately afterwards Figure 6 , Figure 7 The data dependencies for the second block are shown. The second block (block 2) of the output data of the final layer is adjacent to the first block (block 1). As with block 1, block 2 depends on the amount of intermediate data (ultimately input data) added as the layers are traversed backward. However, some of these data dependencies are the same as those of block 1 in the output data of the final layer. Specifically, when evaluating block 1 (block 1) in layer 1, overlap 1 in the output of layer 1 has been calculated. The only additional data that needs to be calculated is the block in layer 1, i.e., block 2. Similarly, moving backward to the layer 0 output, the only additional data that needs to be calculated is the block in layer 0, i.e., block 2. Finally, moving backward to the input data, the only additional input data that needs to be read from memory is block 2 (block 2). Figure 6 As shown, Figure 7 Each tile in the graph is separated by a solid line. Overlapping data is separated by dashed lines. Dashed lines indicate the highest limit of a data dependency (i.e., the first row of data required to compute a tile in the next layer). Sloping dashed lines indicate the lowest limit of a data dependency (i.e., the last row of data required to compute a tile in the next layer). Thus, for example, tile 2 in the output of layer 1 depends on overlap 1 and tile 2 in the output of layer 0. Tile 2 contains overlap 2, which will be reused in the next propagation to compute tile 3 in subsequent layers, and so on. Note that if the overlap data from the first propagation is preserved, the amount of “fresh” data required for tile 2 does not increase when traversing these layers backward (except where a stride greater than 1 indicates). The same applies to all subsequent propagations (tiles) after tile 2. In other words, for all propagations after the first propagation (operating on the head tile), a smaller amount of input data needs to be retrieved from memory, and a smaller amount of intermediate data needs to be computed at each layer.

[0100] This leads to an important observation: overlapping data constitutes the non-disposable portion of the data. The remainder of each tile constitutes the disposable portion of the data. This means that overlapping data is needed not only to compute the output data of the ending layer of the current tile, but also subsequently to compute the output data of the ending layer of subsequent tiles. Conversely, the remaining data in each tile is necessary for computed to the output data of the ending layer of the current tile, but can be deleted / rewritten afterward.

[0101] To consider this in detail using a simplified one-dimensional example, refer to... Figure 8 . Figure 8 The diagram illustrates the individual data elements in the input and output data of each of the three layers (layer 0, layer 1, and the final layer) of a neural network. Each circle represents a data element (which could be a pixel in, for example, an image or feature map). Each element depends on elements from a subgroup in a previous layer. Figure 8The connecting lines in the diagram indicate the elements of the previous layers that a given element depends on. Therefore, for example, the 3rd element in the output of the ending layer depends on the 2nd, 3rd, and 4th elements in the output of layer 1. The 4th element in the output of the ending layer depends on the 3rd, 4th, and 5th elements in the output of layer 1. In other words, in this example, the kernel size of the ending layer is 3, and the stride is 1. The kernel size of layer 1 is 3, and the stride is 2. The kernel size of layer 0 is 3, and the stride is 1. Since the stride of both layer 0 and the ending layer is 1, the output of layer 0 is approximately the same size as the input data, and the output of the ending layer is approximately the same size as the output of layer 1 (depending on padding needs). Meanwhile, since the stride of layer 1 is 2, the output of layer 1 is approximately half the output of layer 0. The dashed lines represent the boundaries between blocks (tiles). That is, the 3rd data element shown in the ending layer is the final data element of the first block (tile 1). Taking the 4th element in the final layer output as an example, it can be seen that each element in the final layer output depends on: the 3 elements in the output of layer 1; the 7 elements in the output of layer 0; and the 9 elements in the input data. The elements in tile 1 (except for the overlapping element 1) are shown with diagonal shading. Three such elements (1-3) are shown in the final layer output as examples.

[0102] Due to data dependencies, some elements can be discarded after the final layer output of the current block (tile 1) has been computed. These elements are called "disposable" elements. Some other elements will also be used to compute the final layer output of the next block (tile 2). These elements are called "non-disposable" elements. Specifically, overlap 1 elements ( Figure 8 Elements 7 and 8 in the input data (colored as solid black) are non-one-time elements. Elements 7 and 8 in the input data are used to calculate elements 6 and 7 in the output of layer 0 during the first pass through multiple layers, to calculate the final layer output data of the current block (block 1). However, these elements are also needed to calculate elements 8 and 9 in the output of layer 0. Elements 8 and 9 are not needed when calculating the first block during the first propagation, but will be needed when calculating the second block during the second propagation. Therefore, to minimize memory access overhead, it is desirable to retain elements 7 and 8 of the input data in the on-chip memory 314 to avoid needing to read these elements again from DDR 312 during the second propagation. Elements 7 in the output of layer 0 and elements 3 and 4 in the output of layer 1 are also overlapping elements, because these elements will also be needed during the calculation in the second propagation (elements 7 in the output of layer 0 will be needed to calculate elements 5 in the output of layer 1; and elements 3 and 4 in the output of layer 1 will be needed to calculate elements 4 and 5 in the final layer output).

[0103] Now consider Figure 8The remaining elements of tile 1 are shown in diagonal shading. Element 6 of the input data is used to calculate elements 5, 6, and 7 of the layer 0 output. However, each of these three elements must be calculated during the first propagation for the calculation of the first output block (tile 1). Element 6 of the input data is not needed in any additional calculations during the second propagation; therefore, this element is "disposable" data. Similarly, element 6 in the layer 0 output is used to calculate element 4 in the layer 1 output, but this element is no longer needed during the second propagation (provided that element 4 in the layer 1 output is retained). Elements of tile 2 (except for overlapping 2 elements) are shown as hollow circles. Overlapping 2 elements are shown in gray shading. These elements will be reused in the calculation of tile 3, and are therefore not disposable data. Some tile 3 elements are shown in horizontal shading at the bottom of the figure.

[0104] The inventors of this invention have recognized that one-time data is only used to compute the next layer in the current propagation process, which involves multiple layers. Once the next layer in the current propagation process has been computed, one-time data from previous layers can be deleted / rewritten. For all layers in the current propagation process, non-one-time data (overlapping) must be retained so that it can be used for computation of the second block in the second propagation process. This recognition reduces unnecessary read / write operations to / from DDR 314 and also reduces unnecessary recomputation of intermediate output data.

[0105] Now refer to Figure 9 This describes a memory management strategy based on an example of a hardware implementation. The inventors have recognized that one-time data is relatively large but only valid for a short period of time. Conversely, non-one-time data (overlapping) is relatively small but valid for the entire propagation process. Figure 9 An exemplary memory allocation of on-chip memory 314 utilizing these characteristics is shown. In this example, it is assumed that six layers (numbered 0, 1, 2, 3, 4, and 5) of a neural network will be processed together. In one branch, each layer is connected to its preceding layer. In a second branch, layer 5 is connected to layer 1. This means that the output data of layer 1 is a portion of the input data of layer 5.

[0106] The memory management strategy divides the on-chip memory 314 into three segments. The first segment 802 (which may be referred to as "swap" memory) stores one-time data. The second segment 804 (which may be referred to as "heap" memory) stores non-one-time data. The third segment 806 (coefficient memory) stores the weight data for each layer. Within the first segment 802, multiple slots A, B, and C are defined. During a given propagation process (for a given block of output data), the method is as follows: The one-time output of layer 0 is written to slot A. The non-one-time output of layer 0 is written to the beginning of the second segment 804. Data is read from these two locations before evaluating layer 1. During the evaluation of layer 1, the one-time output of layer 1 is written to slot B, and the non-one-time output of layer 1 is written to the second segment 804, for example, appended to the non-one-time output of layer 0. At this point, one-time data from layer 0 is no longer needed. Therefore, during the evaluation of layer 2, the contents of slot A can be rewritten using the one-time output of layer 2. The non-one-time output of layer 2 is appended to the data in the second section 804. However, neither slot B nor slot A should be overwritten when evaluating layer 3. Due to the branching, the contents of slot B (the one-time output of layer 1) are needed again to evaluate layer 5. Of course, the contents of slot A are immediately needed to evaluate the current layer (layer 3). Therefore, the one-time output of layer 3 is written to slot C. The one-time output of layer 4 is written to slot A, thus overwriting the contents of that slot that are no longer needed for further computation. To evaluate layer 5, the one-time output data of layer 1 is read from slot B, and the one-time output data from layer 4 is read from slot A. (Non-one-time output data from layers 1 and 4 is read from the second section 804.)

[0107] In this way, slots used for storing one-off data are frequently reused during a given propagation process, thus helping to reduce the total amount of on-chip memory 314 occupied at any given time. The size of each slot can be selected based on the maximum size of one-off data to be stored in the slot during the propagation process. Note that in this example, the second segment 804 is double-buffered. The non-one-off output data (overlap) of all layers of the current block is stored in one set of memory locations. During the next propagation process (for computing the next block), the overlap is retrieved from this set of memory locations as needed. The overlap for the next block is stored in a second set of memory locations. For subsequent propagation processes (subsequent blocks), the hardware implementation alternates between using the first and second sets of memory locations, i.e., reading from one set while writing to the other.

[0108] One-time and non-one-time portions of data can be identified by analyzing the structure of each of the multiple layers. Similarly, the allocation of one-time data from each layer to a specific slot in the first segment 802 of on-chip memory 314 can be accomplished by analyzing the overall layer structure of the multiple layers. This analysis can be performed dynamically by software at runtime or when mapping a given neural network to a given hardware implementation. Alternatively, this can be manually defined by the designer of the hardware implementation. However, the use of software facilitates mapping a wide variety of different neural networks to a given hardware implementation.

[0109] As mentioned above, the multiple layers of the neural network evaluated together in each propagation (also referred to as "tile groups" in this paper) can include all the layers of the neural network, but this is not required. Multiple layers can consist of a subset of the layers of the entire neural network. Depending on the structure of the neural network, different advantageous layer groupings can be found. Figure 10 A simple neural network is shown, consisting of four layers with no branches, each layer depending solely on the output of the previous layer. This allows for a high degree of freedom in choosing how to group the layers together. For example, layers can be grouped into one or more groups according to any of the following grouping strategies:

[0110] 1-4

[0111] 1-2, then 3-4

[0112] 1-2, then 3, then 4

[0113] 1-3, then 4

[0114] 1, then 2-4

[0115] 1, then 2-3, then 4

[0116] 1, then 2, then 3-4

[0117] Generally speaking, given the size constraints of the on-chip memory 314, it may be preferable to group as many layers as possible together.

[0118] Figure 11A A slightly more complex network structure is shown. Again, there are four layers, each depending on the previous layer. However, there is an additional branch from layer 2 to layer 4. Figure 11AIn the diagram, the network is divided into two groups, each with two layers (layers 1-2 and 3-4), indicated by the dashed boxes. However, it should be noted that this may not be optimal in terms of memory access requirements. The outputs of layers 1-2 are written to DDR 312. When evaluating layer 3 (the first layer of layers 3-4), this output is read from DDR 312; however, evaluating layer 4 also requires the output of layer 2. Therefore, the same data must be read from DDR 312 again to evaluate layer 4. Conversely, Figure 11B An alternative grouping of layers is shown, which may be more efficient in terms of memory access requirements. Here, layer 1 is processed by itself, and layers 2 through 4 are grouped into multiple layers. In this way, both branches are captured within multiple layers. The output data of layer 1 from DDR 312 is read once as the input data of the first layer (layer 2) of the multiple layers. Using the above reference... Figure 9 The memory management strategy described above can process the output data of layer 2 in multiple layers.

[0119] First, refer to the above Figure 2 An example of a convolutional layer in a neural network is given, operating on two-dimensional (2D) image data across multiple p-planes or feature maps. However, it should be understood that this disclosure is not limited to this approach. The principles described above can be applied to 1D, 2D, or higher-dimensional data. When applied to 2D (or higher-dimensional) data, various methods can be used to construct blocks (tiles) of input and output data. Each block can be defined in one, two, or more dimensions. For example, for 2D image data, blocks defined in one dimension can form 2D stripes (along a horizontal or vertical direction). Overlapping data can be defined above and below the stripes (for horizontal stripes) or to the left and right of the stripes (for vertical stripes). With this in mind, a diagram is drawn... Figure 5 , Figure 6 and Figure 7 The diagrams. (These diagrams also apply to 1D blocks defined in 1D data.) It should be understood that, although... Figures 5 to 8 The instruction indicates that the data should be divided into blocks along the Y (height) dimension, but this is not limiting. As mentioned above, the same approach can be applied along the X (width) dimension. However, in some cases, it may be preferable to split along another dimension than to split along one dimension. For example, if the data is stored in memory row by row and the individual data elements are not neatly aligned with the burst size used for reading / writing to memory, then splitting the blocks along the Y (height) dimension may be preferred.

[0120] Alternatively, for 2D image data, blocks can be defined in two dimensions to form a 2D rectangle of data. Overlapping data can then be defined above, below, to the left, and to the right of this rectangle. The overlapping data from the current block may be useful for computing multiple subsequent blocks of output data, such as the block to the right of the current block, the block below the current block, and the block to the lower right of the current block (assuming the blocks are processed in raster scan order). The same principle can be extended to blocks in more dimensions that have overlapping data in multiple directions in multidimensional space.

[0121] In the example above, the depth-first processing of the neural network on a tile-by-tile basis is discussed layer by layer. This applies to... Figure 3 Implementation 300 is primarily designed to process a single convolutional layer. However, it is possible to provide a more refined hardware implementation that adds additional processing modules to implementation 300, enabling the processing of multiple layers in a single pass through the hardware implementation. For example, pooling modules, element-wise operation modules, normalization modules, and activation modules could be provided (see, for example, GB 2566702 A). This could, for instance, allow the execution of a convolutional layer, followed by an activation function, and then a pooling operation in a single pass through the hardware implementation. In some cases, data can be piped from one layer / operation to the next without being written to on-chip memory. When the hardware implementation is capable of performing multiple sequential operations in a single hardware pass, it makes sense to group layers processed together by the hardware. To accommodate this, the concept of a "layer group" is introduced. A layer group is a set of layers that can be processed together by the hardware implementation in a single pass through the hardware. A layer group can consist of a single layer (as referenced above). Figure 3 (as described in the example); however, a layer group can consist of more than one layer if the hardware has the necessary functionality.

[0122] As mentioned above, segmenting data into tiles can provide a way to facilitate parallel multi-core implementations of convolutional neural networks. Now, reference will be made to... Figures 12 to 15 Describe an example of a multi-core implementation based on the first aspect. Figure 12 A block diagram of hardware implementation 600 is shown. It includes three cores 350-0, 350-1, and 350-2. Each core has a corresponding local on-chip memory (OCM) 314a, which can only be accessed by that core. Hardware implementation 600 also includes a shared on-chip memory 314b. All cores are able to access (i.e., read and write) the shared OCM 314b via a high-bandwidth interconnect 610. A partitioning unit 620 is provided, which is configured to manage the allocation of work among the cores.

[0123] Figure 13An example of depth-first processing of tiles using different cores 350 of hardware implementation 600 is illustrated. The individual tiles are similar to those described previously. In this example, the tiles are partitioned along the Y-axis of the input data tensor. This is convenient when the data is stored in memory in row-first order (i.e., X-axis-first order). As shown, the neural network comprises five layer groups (labeled LG0 to LG4). Each layer group is indicated as including three layers represented by circles in the diagram. (It should be understood that this is not limiting—different layer groups can have different numbers of layers.) The layer groups will be evaluated by the hardware implementation in two “tile groups” (labeled tile group 0 and tile group 1). A tile group consists of a set of layer groups that will be depth-first evaluated together after the data has been partitioned into tiles. Input data for the first layer group is read from the tile group from DDR 312, and output data from the final layer group in the tile group is written to DDR 312. Within a tile group, intermediate data (at the tile level) is written to the OCM 314 by each layer group and read from the OCM 314 by the next layer group. Intermediate data can be stored partly in the local OCM 314a and partly in the shared OCM 314b. This will be explained in more detail below.

[0124] As shown in the figure, in this example, the input data stored in DDR 312 is divided into six tiles (labeled tiles 0 to 5) for evaluating tile group 0. Only three cores 350 exist; therefore, not all tiles in this tile group can be processed in parallel. Instead, the first three tiles are assigned to three cores respectively for the first multi-core tile propagation process (multi-core tile propagation process 0). Here, the multi-core tile propagation process refers to the evaluation of all layer groups of a tile group of N consecutive tiles, where N is the number of cores. Tile group 0 consists of the first three layer groups LG0-LG2. In multi-core tile propagation process 0, core 0 evaluates layer groups LG0-LG2 based on tile 0; core 1 evaluates these layer groups based on tile 1; and core 2 evaluates these layer groups based on tile 2. This is in... Figure 13 The solid arrow indicates the next step. Next, as indicated by the dashed arrow, hardware implementation 600 proceeds to the multi-core tile propagation process 1, where core 0 evaluates layer group LG0-LG2 based on tile 3; core 1 evaluates layer group 4 based on tile 4; and core 2 evaluates layer group 5 based on tile 5. This completes the evaluation of the first tile group. Note that... Figure 13 The primary purpose is to illustrate the execution order of the checksum tiles. This diagram does not provide a detailed description of which parts of the data are stored in which memory. The following will refer to... Figure 14A and Figure 14B Further details are provided to explain how the data is arranged in memories 312 and 314.

[0125] During each multi-core tile propagation process, the output tile of LG0 is written to OCM 314 and read from OCM 314 as the input tile of LG1. Similarly, the output tile of LG1 is written to OCM 314 and read from OCM as the input tile of LG2. Before hardware implementation 600 continues to evaluate tile group 1, the output data of the first tile group (i.e., the output tile of LG2) is written to DDR 312.

[0126] Tile group 1 consists of two layer groups LG3-LG4. As shown, for this tile group, the data is divided into only three tiles. The number of tiles can be chosen based on the size of the input data for the tile group, which depends on the size of the OCM314. The number of tiles can also be affected by the number and configuration of layers within the tile group, as these will affect the amount of data that needs to be stored in the OCM 314 during the evaluation of the tile group for each tile. Tile group 1 can be evaluated in a single multi-core tile propagation process because the number of tiles is no greater than the number of cores. Core 0 evaluates layer group LG3-LG4 based on tile 0; Core 1 evaluates layer group LG3-LG4 based on tile 1; and Core 2 evaluates layer group LG3-LG4 based on tile 2. The output tiles of LG3 are written to the OCM314 and read from the OCM314 as input tiles for LG4. The output tiles of LG4 are written to DDR 312.

[0127] Figures 14A to 14B This demonstrates (for different examples) how tiles can be assigned to multiple cores and processed by them. Figure 14A The output data for the three layer groups LG0-LG2, divided into tiles, is shown. Here, the symbol "LGxTn" represents "layer group x, tile n". (See the reference above.) Figure 5 The size of the tiles in each layer is defined by the data dependencies within the neural network layers. Starting with the final layer group, the size of the head tiles expands from the head output tile LG2T0 in the final layer group to the head output tile LG1T0 in the intermediate layer group, and then back to the head output tile LG0T0 in the initial layer group LG0. Within each layer group, subsequent tiles after the head tile LGxT0 have a consistent size, except for the tail tile. The tail tile may shrink as you move backward through the layer groups from the final layer group to the initial layer group. This can be seen by comparing the tail tile LG2T7 of the final layer group with the tail tile LG1T7 of the intermediate layer group. Furthermore, the tail tile LG1T7 completely "disappears" in the initial layer group, meaning there are only seven tiles in this layer group instead of eight.

[0128] The tiles are processed in a multi-core architecture with four cores (C0-C3). Figure 14A and Figure 14BIn either case, the tiles are colored / shaded depending on which kernel processes them. Figure 14B This illustrates how layer groups are processed by the cores over time, with arrows indicating data dependencies between tiles. During the first multi-core tile propagation, the first four tiles are processed by the four cores C0-C3 respectively. In the first layer group, the header tile LG0T0 is larger than the other tiles; therefore, core C0 takes longer to evaluate the layer group based on this tile than cores C1-C3 take to evaluate the layer group based on the other tiles. In this example, each core waits until all cores have completed their evaluation of the layer group. Thus, cores C1-C3 wait for C0 to complete the header tile LG0T0. This provides a simple and efficient method for achieving synchronization between cores. After the evaluation of the first layer group, all four cores proceed to the evaluation of the second layer group, and then the evaluation of the third layer group. Figure 14B As can be seen, in the evaluation of the final layer group during this multi-core tile propagation process, the evaluation of the header tile LG2T0 is faster than that of other tiles. This is because the header tile LG2T0 has been selected to be smaller than the subsequent tiles LG2T1-LG2T7. This is done to avoid the header tile becoming too large in the initial layer. The aim is to attempt to balance the division of labor among the kernels and to avoid one kernel completing its work significantly earlier or later than any other kernel.

[0129] As referenced above Figure 7 and Figure 8 As explained, the evaluation of a layer group for a given tile (excluding the head tile) depends on the overlapping data of previously output tiles in the previous layer group. For example, the evaluation of LG1T1 depends on both LG0T1 and LG0T0. Similarly, the evaluation of LG1T2 depends on LG0T2 and LG0T1, and so on. This dependency between tiles during multi-kernel tile propagation is explained by... Figure 14B The solid arrows in the diagram illustrate this. The dependency means that each core needs to access the overlapping data generated by the other core. For this purpose, the overlapping (non-one-time) data for each tile is stored in a shared OCM 314b. The remaining data (i.e., one-time data) can be stored in each core's local OCM 314a, since no other core needs to access it. This avoids the need for each core to recalculate the overlapping data, and also avoids the potentially expensive overhead of writing overlapping data to DDR 312. Furthermore, the access cost of the local OCM 314a is likely lower than that of the shared OCM 314b, providing improved efficiency compared to implementations that store all data in a shared OCM.

[0130] It should be noted that, in principle, it is not necessary for each core to wait for all cores to complete a layer group. For example, in principle, the evaluation of LG1T2 can begin immediately after LG0T1 and LG0T2 have completed. Similarly, the evaluation of LG1T0 can begin at any time after the evaluation of LG0T0 has completed (i.e., an indeterminate amount of time). This “free-running” approach can indeed be adopted in some other implementations. However, this implementation has the advantage of simplicity because it avoids the additional complexity necessary to track free runs.

[0131] The first multi-core tile propagation process ends by Figure 14B The thick black vertical line in the diagram indicates this. Next, the kernel performs a second multi-kernel tile propagation process to evaluate the layer group based on the remaining tiles. Several points need to be noted regarding the second multi-kernel tile propagation process. For the second and third layer groups LG1-LG2, it should be understood that the fifth tile depends on the overlapping data generated during the evaluation of the fourth tile in the previous multi-kernel tile propagation process. In other words, although the overlapping data from LG0T0, LG0T1, and LG0T2 are used "immediately" by the kernel in the evaluations of LG1T1, LG1T2, and LG1T3 respectively, the overlapping data from LG0T3 needs to be retained in the shared OCM 314b until the second multi-kernel tile propagation process, at which point it is used by kernel C0 in the evaluation of LG1T4. Similarly, the overlapping data from LG1T3 is retained in the shared OCM 314b until it is used by kernel C0 in the evaluation of LG2T4. These dependencies between tiles processed in different multi-kernel tile propagation processes are defined by… Figure 14B The dashed arrows in the diagram illustrate this. Another point to note is that the evaluation of the second layer group based on tail tile LG1T7 is completed faster than other tiles in that layer group. This is because the tile size is smaller, scaled down compared to the tail tile of the final layer group LG2T7 (see [link]). Figure 14A Furthermore, kernel C3 is idle during the evaluation of LG0 in the second multi-kernel tile propagation process because this layer group only has seven tiles. It should be understood that due to the overlap at the boundaries between tiles, the dependencies between tiles and the corresponding need for synchronization between kernels arise. In the absence of overlap (e.g., a series of convolutions with 1x1x1 kernels), synchronization is not required, and the kernels can run freely (without any complexity penalty). Generally, for convolutions, the formula used to determine the size of any overlap in the Y dimension is: overlap = max(0, (kernel height - 1) * dilation height - (stride height - 1)).

[0132] Reference Figure 15 The method is described in more detail below. In step 710, the segmenter unit 620 defines multiple layer groups and arranges them in a tile group. The arrangement in the tile group can be referenced above. Figures 10 to 11BThe process is completed as described above. In step 712, input data for the first layer group is provided in DDR 312. In step 714, the segmenter unit 620 analyzes the input data for the first layer group. Specifically, it checks whether the size of the input data in the Y dimension is higher than a threshold. If so, it determines (step 716) that the input data can be segmented into tiles along the Y dimension. In other words, if the input data has a sufficiently large size in the Y dimension, it will be segmented into tiles. In response to the determination in step 716, the segmenter unit segments the data into tiles (step 718) and assigns each tile to one of the available cores (step 719). If there are more tiles than cores, the segmenter unit 620 will control the cores to perform several multi-core tile propagation processes. Generally, the segmenter unit attempts to segment the input data into tiles so that all cores are occupied. Therefore, it may attempt (where possible) to segment the input data into a number of tiles that is an integer multiple of the number of cores. The number of tiles (and the number of multi-core tile propagation processes) will be affected by the capacity of the OCM 314.

[0133] In step 720, at least two cores evaluate the first layer group based on at least two corresponding tiles. That is, each core evaluates the first layer group based on one tile. The segmenter unit is configured to identify the one-time and non-one-time (overlapping) portions of each output tile. It instructs the cores to store these different portions in different locations. Thus, the first core 350-0 writes the one-time portion of the first output tile to its local OCM 314a (step 722) and writes the overlapping portion of the first output tile to the shared OCM 314b (step 724). Simultaneously, the second core 350-1 writes the one-time portion of the second output tile to its local OCM 314a (step 722) and writes the overlapping portion of the second output tile to the shared OCM 314b (step 724). Only two instances of steps 720-724 are shown in the flowchart, implying the presence of two cores; however, it should be understood that parallel instances of these steps are as numerous as the number of cores occupied by the evaluation. Note that steps 722 and 724 can also be executed in reverse order or in parallel.

[0134] In step 728, any kernel that completes its evaluation before the others waits for all kernels to complete their evaluation of the first layer group before proceeding. The method proceeds to step 730. If there are other layer groups in the current tile group, each kernel continues to evaluate the next layer group based on its tiles. When evaluating the second and subsequent layer groups, each kernel reads the necessary one-time data from the previous layer group from its own local OCM 314a and the necessary overlap data from the previous layer group from the shared OCM 314b.

[0135] The loop continues until all layer groups in the current tile group have been evaluated by each kernel based on the corresponding tiles. When (in step 730) there are no other layer groups in the current tile group, the method proceeds to step 732 to check if there are any other tiles to process. This represents the end of a multi-kernel tile propagation process. If there are other tiles to process, the kernel continues a second multi-kernel tile propagation process to evaluate all layer groups in the current tile group for the next set of tiles. The method continues to loop within the multi-kernel tile propagation process until all tiles have been processed. Then, in step 732, when there are no more tiles to process, the evaluation of the tile group is complete, and the method ends. If necessary, the method can be repeated for subsequent tile groups.

[0136] It should be noted that in steps 714 and 716, the segmenter unit 620 determines that the input data can be segmented into patches based on the size of the input data (particularly in the Y dimension). Segmentation into patches depends on whether the tensor size is large enough. However, even for relatively small tensors, it is advantageous to be able to segment the data across multiple kernels. According to the second aspect, a method for doing so is provided. Reference will now be made to... Figure 12 , Figure 16 and Figure 17 An example describing this method is provided. According to this example, the evaluation can be split between kernels along one of the other dimensions. This could include splitting the evaluation of the layer group along the X dimension, which is one of the traversal dimensions. Similarly, it could include splitting the evaluation of the layer group across one of the non-traversal dimensions, specifically across input channel C, or across output channel F. Typically, when the evaluation is split into a propagation process, the input data for the layer group is not split between kernels, but is stored in a shared OCM 314b and accessed from there by each of the kernels 350. This is related to the first aspect (as...). Figures 13 to 15 The use of tiles differs in the example shown. In the first aspect, both data and evaluation are partitioned between kernels. When partitioning along the C dimension, each kernel evaluates the layer group of the corresponding channel (or channel group) of the input data. When partitioning along the F dimension, each kernel evaluates the layer group of the corresponding filter (or filter group), producing the corresponding output channel or output channel group. When partitioning along the X dimension, each kernel evaluates the layer group of the corresponding column (or set of columns) of the input data.

[0137] Figure 16This illustration schematically demonstrates an example of partitioning the evaluation of multiple layer groups into multiple propagation processes, splitting across multiple kernels, and evaluating one layer group at a time. There are five layer groups LG0-LG4. Layer group LG2 consists of two layers; the other layer groups each consist of three layers. The input data for the first layer group is stored in DDR 312. In this example, the first layer group LG0 is partitioned into three propagation processes that will be evaluated by different kernels. The partitioning is along the X dimension. Each of the three kernels C0-C2 evaluates the corresponding one of the three propagation processes. In each propagation process, the corresponding kernel processes a portion of the input data to produce a portion of the output data. In this example, it is assumed that there is at most one convolutional layer in each layer group. Therefore, there is no issue of overlapping data. Depending on the kernel size and stride, different kernels can access overlapping portions of the input data, but they generate different portions of the output data. Figure 13 Unlike the previous example, in this case, the data is not segmented into individual tiles. (This would be difficult to do in the X dimension in the current example because the data is row-majorly arranged, and the data elements in the X dimension are not aligned with bursts of data in memory.) Instead, each core accesses the portion of data it needs from the shared OCM 314b. As already noted, each core generates the corresponding portion of the output data. Once all cores have evaluated the first layer group, the portions of the output data are combined in the OCM 314.

[0138] The next two layer groups, LG1-LG2, are not split; therefore, each layer group is evaluated by a core in a single propagation. The subsequent layer group, LG3, is split across the output channel (F dimension) into six propagation processes. Since there are more propagation processes here than cores, cores C0-C2 first evaluate the layer group for the first three propagation processes in the F dimension (labeled 0, 1, and 2 in the diagram). Then, cores C0-C2 evaluate the layer group for the three propagation processes of the second group (labeled 3, 4, and 5 in the diagram) to complete the layer group. The final layer group, LG4, is again processed by core C0 in a single propagation. The outputs of the first four layer groups, LG0-LG3, are combined in the OCM 314; the output of the final layer group, LG4, is written to the DDR 312.

[0139] As this example illustrates, the splitting is flexible because different layer groups can be split along different dimensions (e.g., comparing LG0 and LG3). This is achieved by combining the complete output of each layer group in the OCM 314, making the splitting of one layer group independent of the splitting of other layer groups. This is related to... Figure 13 Another difference in tile-based segmentation is that... Figure 13 In this process, each kernel operates on a specific tile, and the correspondence between tiles is preserved when the layer groups of tile groups are evaluated in a depth-first manner.

[0140] Reference Figure 17Describe the method in more detail. (and) Figure 15 Steps similar to those in the figure will be given the same reference numerals. In step 710', the segmenter unit 620 defines multiple layer groups. In step 712', input data for the first layer group is provided in DDR 312. In step 714', the segmenter unit 620 analyzes the input data to check if it can be segmented into tiles. Specifically, the segmenter unit checks whether the size of the input data in the Y dimension is higher than a threshold. In step 746, the segmenter unit 620 determines that the size is not higher than the threshold—therefore, the data is not suitable for tileization. In response to this determination, the segmenter unit 620 divides the evaluation of the layer group into propagation processes (step 748) and assigns each propagation process to a kernel (step 749). In step 750, the kernels evaluate the layer groups of their respective propagation processes to generate the corresponding portions of the output data. If it is determined in step 752 that there is another propagation process to be completed in the current layer group, the kernel evaluates the layer group for the necessary additional propagation processes (such as in...). Figure 16 (In the example of layer group LG3). When it is determined in step 752 that there are no additional propagation processes to evaluate for the current layer group, the method proceeds to step 754, and portions of the output data from each propagation process are combined to produce the output data for the layer group. In step 756, this output data is written to OCM 314. In step 758, it is checked whether there is another layer group to evaluate. If so, the method proceeds again to step 748 to split the evaluation of the next layer group into propagation processes. Note that although only two instances of step 750 are shown in the flowchart, implying the presence of two cores, it should be understood that the number of parallel instances of this step is as many as the number of cores used for the evaluation.

[0141] The output data of each layer group is written to a shared OCM 314b so that all kernels can access it to evaluate the next layer group. The way the evaluation is segmented in step 748 and the way the output data portions are combined in step 754 will depend on the characteristics of the layers in the layer group and along the dimensions along which the evaluation is segmented. In some cases, the evaluation may be segmented by the input channel C, so that each kernel evaluates the layer group for the corresponding channel (or channel group) of the input data. For example, if the layer group includes a convolutional layer with a filter, and each kernel processes a different input channel, this would mean that each kernel applies one channel of the filter kernel to the corresponding channel of the input data. The corresponding output portions produced by the different kernels will then need to be combined by element-wise summation to produce the output of the convolution. This can be done by an element-wise operation module in one of the kernels 350 (if available) or by another processing module separate from kernel 350. The weights (and biases) of the different channels of the kernel can be stored in the local OCM 314a of each corresponding kernel, since each kernel only needs to access the weights of the channel (or channels) it is responsible for evaluating.

[0142] If the layer group consists of element-wise operations, and each core processes a different input channel, then the corresponding output portions produced by different cores can be combined by concatenating them in the input channel dimension (P dimension). Note that because the step of combining the corresponding output portions consists of concatenating them, this can be achieved by each core simply writing its output portion to the appropriate location in the shared OCM 314b. In other words, the "combination" in step 754 and the "writing" in step 756 can be performed together, and can be performed by each core immediately following the "evaluation" step 750.

[0143] The parameters controlling the element-wise operation of each input channel can be segmented and stored in a separate local OCM 314a. Similarly, for other operations that can be segmented by input channels, the parameters controlling the processing of each channel can be segmented into separate local OCM 314a.

[0144] If a layer group comprises convolutional layers with multiple output channels (filters), and the evaluation is split across kernels according to the output channels (F dimension), then each kernel will be responsible for evaluating a given filter or filter group. The weights (and biases) of the relevant filters can be stored in the local OCM 314a of the corresponding kernel.

[0145] On the other hand, if the layer group includes convolutional layers and the evaluation is performed across the kernel along one of the traversal dimensions (e.g., the X dimension), the weights and biases are stored in a shared OCM 314b, where they are accessible to all kernels.

[0146] Some layer partitioning across multiple kernels may be difficult, inefficient, or impossible. For example, it should be noted that... Figure 16 Layers LG1, LG2, and LG4 in the code are not partitioned across cores at all—they are all executed by a single core (which may leave other cores idle if no other work is occupying them). For example, this approach can be used to process fully connected layers that include a filter.

[0147] Now we will understand, Figure 15 and Figure 17 The methods can be combined advantageously. Figure 15 This method can be used to handle large input data tensors (i.e., if the size exceeds a threshold in step 714 when analyzing the size of the input data tensor). And... Figure 17 This method can be used to process small input data tensors (if the size is below a threshold in step 714'). In other words, the threshold used in step 714 and the threshold used in step 714' can be the same, allowing the hardware implementation 600 to follow... Figure 15 The method or following Figure 17 The method.

[0148] The processing system may include the hardware implementation described above, or may be configured to perform the methods described above. The processing system may be an artificial intelligence accelerator system, such as a neural network accelerator (NNA), or a graphics processing system / graphics processing unit (GPU).

[0149] Figure 18 A computer system in which the processing system according to this disclosure can be implemented is shown. The computer system includes a CPU 902, an NNA 904, a memory 906, and other devices 914, such as a display 916, a speaker 918, and a camera 919. A processing block 910 (corresponding to hardware implementation 600) is implemented on the NNA 904. In other examples, the processing block 910 may be implemented on the CPU 902. Components of the computer system can communicate with each other via a communication bus 920. A memory bank 912 (corresponding to DDR 312) is implemented as part of the memory 906.

[0150] Although Figure 18 One implementation of an AI accelerator system is shown, but it should be understood that similar block diagrams can be drawn for graphics processing systems, for example, by replacing CPU 902 or NNA904 with a GPU, or by adding a GPU as an additional unit. In this case, processing block 910 can be implemented in the GPU.

[0151] Figure 12 The hardware implementation 600 is shown as comprising a number of functional blocks. This is merely illustrative and not intended to define a strict division between different logical elements of such an entity. Each functional block may be provided in any suitable manner. It should be understood that intermediate values ​​described herein as being formed by the hardware implementation do not need to be physically generated by the hardware implementation at any point in time, and may only represent logical values ​​that conveniently describe the processing performed by the hardware implementation between its inputs and outputs.

[0152] The hardware implementations described herein can be embodied in hardware on an integrated circuit. The hardware implementations described herein can be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above can be implemented in software, firmware, hardware (e.g., a fixed logic circuit system), or any combination thereof. The terms “module,” “function,” “component,” “element,” “cell,” “block,” and “logic” are used herein to generally denote software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, cell, block, or logic represents program code that, when executed on a processor, performs a specified task. The algorithms and methods described herein can be executed by one or more processors executing code that causes the processor to execute the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical disk, flash memory, hard disk storage, and other memory devices that can use magnetic, optical, and other techniques to store instructions or other data and can be accessed by a machine.

[0153] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for processor execution, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, bytecode, code defining integrated circuits (e.g., hardware description languages ​​or netlists), and code expressed in, for example, C++. Executable code can be expressed in programming languages ​​such as OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes the processor of a computer system that supports the executable code to perform the tasks specified by that code.

[0154] A processor, computer, or computer system can be any kind of device, machine, or special-purpose circuit, or a collection or part thereof, that has the processing power to execute instructions. A processor can be any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-a-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system may include one or more processors.

[0155] This invention also intends to cover software, such as hardware description language (HDL) software, that defines the configuration of hardware as described herein for designing integrated circuits or for configuring programmable chips to perform desired functions. That is, a computer-readable storage medium may be provided on which computer-readable program code in the form of an integrated circuit definition dataset is encoded, configuring the system, when processed (i.e., run) in an integrated circuit manufacturing system, to manufacture a hardware implementation configured to perform any of the methods described herein, or a hardware implementation including any of the devices described herein. The integrated circuit definition dataset may, for example, be an integrated circuit description.

[0156] Therefore, a method can be provided for manufacturing the hardware implementation as described herein at an integrated circuit manufacturing system. Furthermore, an integrated circuit definition dataset can be provided, which, when processed in the integrated circuit manufacturing system, enables the method for manufacturing the hardware implementation to be executed.

[0157] Integrated circuit definition datasets can be in the form of computer code, such as netlists, code for configuring programmable chips, or hardware description languages ​​suitable for manufacturing at any level in integrated circuits, including register-transfer level (RTL) code, high-level circuit representations (such as Verilog or VHDL), and low-level circuit representations (such as OASIS(RTM) and GDSII). Higher-level representations (such as RTL) that logically define hardware suitable for manufacturing in integrated circuits can be processed on a computer system configured to generate manufacturing definitions of integrated circuits within the context of a software environment that includes definitions of circuit elements and rules for combining these elements to generate the manufacturing definitions of the integrated circuits defined by that representation. As is typically the case where software executes at a computer system to define a machine, one or more intermediate user steps (e.g., providing commands, variables, etc.) may be required to configure the computer system to generate the manufacturing definitions of the integrated circuits, executing code that defines the integrated circuits to generate the manufacturing definitions of said integrated circuits.

[0158] Now refer to Figure 19 Describe an example of processing integrated circuit definition datasets at an integrated circuit manufacturing system in order to configure the system for manufacturing hardware implementations.

[0159] Figure 19An example of an integrated circuit (IC) manufacturing system 1002 is shown, configured to manufacture hardware implementations as described in any of the examples herein. Specifically, the IC manufacturing system 1002 includes a layout processing system 1004 and an integrated circuit generation system 1006. The IC manufacturing system 1002 is configured to receive an IC definition dataset (e.g., defining hardware implementations as described in any of the examples herein), process the IC definition dataset, and generate ICs based on the IC definition dataset (e.g., embodying hardware implementations as described in any of the examples herein). The processing of the IC definition dataset configures the IC manufacturing system 1002 to manufacture integrated circuits embodying hardware implementations as described in any of the examples herein.

[0160] The layout processing system 1004 is configured to receive and process an IC definition dataset to determine a circuit layout. Methods for determining a circuit layout based on an IC definition dataset are known in the art and may involve, for example, synthesizing RTL code to determine the gate-level representation of the circuit to be generated, for example, in relation to logic components (e.g., NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined based on the gate-level representation of the circuit. This can be done automatically or with user intervention to optimize the circuit layout. Once the layout processing system 1004 has determined the circuit layout, it can output the circuit layout definition to the IC generation system 1006. The circuit layout definition may be, for example, a circuit layout description.

[0161] As is known in the art, IC generation system 1006 generates ICs according to a circuit layout definition. For example, IC generation system 1006 may implement a semiconductor device manufacturing process for generating ICs, which may involve a multi-step sequence of photolithography and chemical processing steps, during which electronic circuits are gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which can be used in the photolithography process to generate ICs according to the circuit definition. Alternatively, the circuit layout definition provided to IC generation system 1006 may be in the form of computer-readable code, which IC generation system 1006 can use to form a suitable mask for generating ICs.

[0162] The various processes performed by the IC manufacturing system 1002 can all be implemented in one location, for example, by one party. Alternatively, the IC manufacturing system 1002 can be a distributed system, allowing some processes to be performed at different locations and by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition dataset to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask based on the circuit layout; and (iv) using the mask to manufacture the integrated circuit.

[0163] In other examples, processing of an integrated circuit definition dataset at an integrated circuit manufacturing system can configure the system to manufacture a hardware implementation without processing the IC definition dataset to determine circuit layout. For example, an IC definition dataset can define the configuration of a reconfigurable processor, such as an FPGA, and processing of said dataset can configure the IC manufacturing system (e.g., by loading configuration data into the FPGA) to generate a reconfigurable processor with said defined configuration.

[0164] In some embodiments, when the integrated circuit manufacturing definition dataset is processed in an integrated circuit manufacturing system, it can enable the integrated circuit manufacturing system to generate apparatuses as described herein. For example, the integrated circuit manufacturing definition dataset referenced above... Figure 19 The manner in which the integrated circuit manufacturing system is configured can result in the manufacture of devices as described herein.

[0165] In some examples, an integrated circuit definition dataset may include software running on hardware defined at the dataset, or software running in combination with hardware defined at the dataset. Figure 19 In the example shown, the IC generation system can be further configured by the integrated circuit definition dataset to load firmware onto the integrated circuit according to the program code defined in the integrated circuit definition dataset during the manufacturing of the integrated circuit, or otherwise provide the integrated circuit with program code for use with the integrated circuit.

[0166] Compared to known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can lead to performance improvements. Performance improvements may include one or more of increased computational performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), trade-offs may be made between performance improvements and physical implementation methods, thereby improving the manufacturing process. For example, a trade-off may be made between performance improvements and layout area, matching the performance of known implementations but using less silicon. This can be accomplished, for example, by reusing functional blocks serially or sharing functional blocks among elements of the device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that lead to improvements in the physical implementation of devices, apparatuses, modules, and systems (such as reduced silicon area) can be traded off for performance improvements. This can be accomplished, for example, by manufacturing multiple instances of the module within a predefined area budget.

[0167] The applicant has independently disclosed each individual feature described herein, as well as any combination of two or more such features, to the extent that such features or combinations can be implemented based on the specification as a whole, in accordance with the common knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein. In view of the foregoing description, those skilled in the art will understand that various modifications can be made within the scope of this invention.

Claims

1. A method for implementing a deep neural network in a multi-core hardware (600), the deep neural network comprising a plurality of layers, the multi-core hardware comprising at least a first core and a second core, the method comprising: defining (710’) a plurality of layer groups (LG) comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein output data of the first layer group comprises input data for the second layer group; providing (712’) input data for the first layer group in one or more first memory devices, the input data comprising a multi-dimensional tensor comprising one or more traversed dimensions and one or more non-traversed dimensions, the one or more traversed dimensions being dimensions traversed by a stride in at least one layer of the first layer group; splitting (748) the evaluation of the first layer group along one of the non-traversed dimensions into at least a first propagation process and a second propagation process; assigning (749) the first propagation process to the first core and the second propagation process to the second core; using the first core, evaluating (750) the first layer group for the first propagation process to generate a first portion of output data; using the second core, evaluating (750) the first layer group for the second propagation process to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements; producing the output data of the first layer group comprising combining (754) the first portion of output data and the second portion of output data; and storing (756) the output data of the first layer group as input data for the second layer group in one or more second memory devices.

2. The method of claim 1, further comprising: splitting (748) the evaluation of the second layer group along one of the traversed dimensions or one of the non-traversed dimensions into at least a first propagation process and a second propagation process; assigning (749) the first propagation process of the second layer group to the first core and the second propagation process of the second layer group to the second core; using the first core, evaluating (750) the second layer group for the first propagation process to generate a first portion of output data; and using the second core, evaluating (750) the second layer group for the second propagation process to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements. the evaluation of the first layer group is split along a first dimension and the evaluation of the second layer group is split along a second dimension, wherein the first and second dimensions are different dimensions.

3. The method of claim 2, wherein, the first layer group is split according to input channels, whereby the first propagation process comprises evaluating (750) the first layer group for a first input channel and the second propagation process comprises evaluating (750) the first layer group for a second input channel.

4. The method of any one of claims 1 to 3, wherein, or:

5. The method of claim 4, wherein, ​ The layer group comprises convolution layers, and wherein combining (754) the first portion of output data and the second portion of output data comprises element-wise summation; or The layer group comprises element-wise operations, and wherein combining (754) the first portion of output data and the second portion of output data comprises concatenating them.

6. The method of claim 1, wherein, The first layer group is split (748) according to output channels, whereby the first propagation process comprises evaluating (750) the first layer group for a first output channel, and the second propagation process comprises evaluating (750) the first layer group for a second output channel.

7. The method of claim 1, further comprising storing output data of a final layer group of the plurality of layer groups in the one or more first memory devices (312).

8. The method of claim 1, further comprising: analyzing (714’) at least one characteristic of the input data for the first layer group; and based on the analysis, determining (746) that the input data should be split along the one of the dimensions into propagation processes, wherein the splitting (748) of the input data for the first layer group is performed in response to the determination, and optionally wherein the at least one characteristic comprises a size of the input data in at least one traversal dimension, and wherein determining (746) that the input data should be split into propagation processes comprises determining that the size in the at least one traversal dimension is not greater than a threshold size.

9. The method of claim 1, wherein, The evaluation of the first layer group is split (748) into the first propagation process, the second propagation process, and a third propagation process, The method further comprises: allocating (749) the third propagation process to the first core; and after evaluating (750) the first layer group for the first propagation process using the first core, evaluating (750) the first layer group for the third propagation process using the first core.

10. The method of claim 1, wherein, The input data for the first layer group comprises a first traversal dimension and a second traversal dimension, wherein the input data is stored in the one or more first memory devices first ordered according to the first traversal dimension, and wherein the evaluation of the first layer group is split (748) along the first traversal dimension.

11. The method of claim 1, wherein, The one or more second memory devices comprise: a local memory (314a) for the first core, which is only accessible by the first core; a local memory (314a) for the second core, which is only accessible by the second core; and a shared memory (314b), which is accessible by the first core and the second core, The method comprises storing the first portion of output data and the second portion of output data in the shared memory (314b).

12. The method of claim 11, wherein, or: (A) The first layer group comprises convolution layers, the method further comprising splitting weight values and / or bias values and storing the weight values and / or bias values in the local memory (314a), or (B) The first layer group comprises element-wise operations, the method further comprising storing weight values and / or bias values in the local memory (314a). (B) the evaluation is split according to input channels, wherein the first group of layers does not comprise convolutional layers, and wherein the method comprises splitting parameters of the first group of layers and storing the parameters in the local memory (314a).

13. A hardware apparatus for implementing a deep neural network comprising a plurality of layers arranged in a plurality of layer groups (LGs), the plurality of layer groups comprising at least a first layer group and a second layer group, each layer group comprising one or more consecutive layers of the neural network, wherein, The output data of the first group of layers comprises input data for the second group of layers, the hardware apparatus comprises: one or more first memory devices (312) configured to store input data for the first group of layers, the input data comprising a multi-dimensional tensor, the multi-dimensional tensor comprising one or more traversed dimensions and one or more non-traversed dimensions, the one or more traversed dimensions being dimensions traversed by a stride in at least one layer of the first group of layers; one or more second memory devices (314) configured to store the output data of the first group of layers; a plurality of cores (350) comprising at least a first core (350-0) and a second core (350-1); and a splitter unit (620), wherein the splitter unit is configured to split (748) the evaluation of the first group of layers along one of the non-traversed dimensions into at least a first propagation process and a second propagation process, wherein the splitter unit (620) is configured to assign (749) the first propagation process to the first core and the second propagation process to the second core, wherein the first core is configured to evaluate (750) the first group of layers for the first propagation process to generate a first portion of output data; wherein the second core is configured to evaluate (750) the first group of layers for the second propagation process to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements, wherein the plurality of cores is configured to produce the output data of the first group of layers, comprising combining (754) the first portion of output data and the second portion of output data, and storing (756) the output data of the first group of layers as input data for the second group of layers in the one or more second memory devices.

14. The hardware apparatus of claim 13, wherein, the splitter unit (620) is configured to split (748) the evaluation of the second group of layers along one of the traversed dimensions or one of the non-traversed dimensions into at least a first propagation process and a second propagation process, wherein the splitter unit (620) is configured to assign (749) the first propagation process to the first core and the second propagation process to the second core, wherein the first core is configured to evaluate (750) the second group of layers for the first propagation process to generate a first portion of output data, wherein the second core is configured to evaluate (750) the second group of layers for the second propagation process to generate a second portion of output data, each of the first portion of output data and the second portion of output data comprising a plurality of data elements.

15. The hardware apparatus of claim 14, wherein, The splitter unit (620) is configured to split (748) the evaluations of the first layer group along a first dimension and to split (748) the evaluations of the second layer group along a second dimension, wherein the first and second dimensions are different dimensions.

16. The hardware apparatus of any of claims 13 to 15, wherein, The splitter unit (620) is configured to split (748) the evaluations of the first layer group into the first propagation process, the second propagation process, and a third propagation process and to assign (749) the third propagation process to the first core; and wherein the first core is configured to evaluate (750) the first layer group for the third propagation process after having completed evaluating the first layer group for the first propagation process.

17. An artificial intelligence accelerator system comprising the hardware apparatus of any one of claims 13 to 16, and / or configured to perform the method of any one of claims 1 to 12.

18. A computer readable storage medium having stored thereon computer readable code which, when executed by a computer, causes the method of any one of claims 1 to 12 to be performed, optionally encoded on a computer readable storage medium.

19. A computer readable storage medium having stored thereon an integrated circuit definition data set which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the artificial intelligence accelerator system of claim 17.

20. A computer readable storage medium having stored thereon a computer readable description of the artificial intelligence accelerator system of claim 17, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process the computer readable description of the artificial intelligence accelerator system using a layout processing system so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system.

21. A computer readable storage medium having stored thereon a computer readable description of the artificial intelligence accelerator system of claim 17, which, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to: process the computer readable description of the artificial intelligence accelerator system using a layout processing system so as to generate a circuit layout description of an integrated circuit embodying the artificial intelligence accelerator system; and manufacture the artificial intelligence accelerator system from the circuit layout description using an integrated circuit generation system. ​

Images