Method and device for parallel and adaptive processing of data streams from deep neural networks

Abstract

The invention relates to the execution of a neural network comprising at least N successive layers C_i, i = 1 to N, in an accelerator (1) comprising: - a matrix (11) of computing nodes (14) each comprising a router; - one or more rows of local buffer memories (13) each connected to a router of an associated computing node; the routers being adapted to route the data, through the matrix, according to a control word associated with each data; during the execution of layer C_i by the accelerator: - before said ofmap output data of layer C_i is stored in said local buffer memories (13), a packing block (16) determines, according to the configuration word associated with layer C_i+1, the control words, for the ifmap input data of layer C_i+1 corresponding to the ofmap output data of layer C_i;- then said ofmap output data are inserted into data packets stored in said local buffers (13) with the control words thus determined for layer C_i+1.;

Description

Title of the invention: Method and device for parallel and adaptive processing of data streams from deep neural networks. Technical field

[0001] The invention lies in the field of artificial intelligence and deep neural networks and more particularly in the field of accelerating inference computation by neural networks, for example convolutional ones. Previous technique

[0002] Artificial intelligence (AI) algorithms currently constitute a vast field of research, as they are destined to become essential components of next-generation applications, relying on intelligent processes to make decisions based on knowledge of their environment, for example, the detection of objects such as pedestrians for a self-driving car or activity recognition for a health-tracking smartwatch. This knowledge is gathered by sensors combined with high-performance detection and / or recognition algorithms.

[0003] In particular, deep neural networks (DNNs), and among them, convolutional neural networks (CNNs – see, for example, Y. LeCun et al. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278–2324), are good candidates for integration into such systems because of their excellent performance in detection and recognition tasks. They are based on filter layers that perform feature extraction and then classification. These operations require significant computation and memory, and the integration of such algorithms into systems necessitates the use of accelerators.

[0004] These accelerators are electronic devices that primarily perform numerous operations in parallel, for example, multiply-accumulate (MAC) operations in CNN algorithms. The objective of these accelerators is to improve the execution performance of DNN algorithms to meet application constraints and improve the system's energy efficiency. They are mainly based on a large number of processing elements involving operators optimized for executing operations and a memory hierarchy enabling efficient data storage.

[0005] In the application to CNNs, most hardware accelerators are based on a network of elementary processors (Processing Elements - PEs) implementing MACs use local buffers to store frequently reused data, such as filter parameters or intermediate data. Communication between the PEs themselves and between the PEs and memory is a crucial aspect to consider when designing a CNN accelerator. Indeed, CNN algorithms exhibit high inherent parallelism and data reuse capabilities. Therefore, the on-chip communication infrastructure must be carefully designed to leverage the large number of PEs and the specific characteristics of CNN algorithms, which improve both performance and energy efficiency. For example, multicasting or broadcasting specific data within the communication network allows target PEs to simultaneously process different data with the same filter using a single memory read.

[0006] Several factors have contributed to limiting or complicating the scalability and flexibility of existing CNN accelerators on the market. These factors manifest themselves through: (i) limited bandwidth due to the lack of an efficient broadcast medium, (ii) excessive energy consumption due to the size of the memory (e.g., 40% of the energy consumption in some architectures is induced by the memory) and the problem of memory capacity walls, (iii) as well as limited data reuse and a need for efficient support for processing the different communication modes.

[0007] There is therefore a need to increase the efficiency of processing in the neural accelerators of DNN architectures, taking into account the high number of PEs and in the case of CNNs, the particularities of CNN algorithms. Summary of the invention

[0008] To this end, according to a first aspect, the present invention describes a processing method for the execution of a neural network comprising at least N successive neural network layers C_i, i = 1 to N, with N>1, in a neural network accelerator;

[0009] the neural network accelerator comprising:

[0010] - a global control block,

[0011] - a matrix of computing nodes, each node comprising a router and an element processing element, also called PE associated with a set of respective local memory registers, the processing element, allowing to perform calculation operations among multiplications and accumulations of data stored in its registers, the router allowing to perform several independent data routings in parallel to distinct outputs of the router, the routers being connected to each other;

[0012] - at least one row of local buffers, each memory row local buffer being positioned on one side of the compute node matrix, and each local buffer is linked to a router of an associated compute node;

[0013] the routers being adapted to route the data, through the matrix of computing nodes in parallel with the execution by the computing nodes, according to a control word associated with each data;

[0014] according to which at the beginning of the execution by the accelerator of a C_i layer, ifmap input data of the layer are extracted from at least some of said local buffers and are provided, with their associated control word, to the routers of the associated compute nodes;

[0015] according to which during the execution of the C_i layer by the accelerator, ofmap output data from the C_i layer from data delivered by the compute node routers associated with local buffers are stored in said local buffers;

[0016] said method being characterized in that the accelerator further comprises a packing block disposed between at least some of the local buffer memories and the nodes associated with said local buffer memories, and said packing block determines the control word associated with the ifmap input data of layer C_i+1 according to a configuration word defined for said layer C_i+1, when one or the other of the following arrangements i and j is implemented in the accelerator:

[0017] disposition i /

[0018] it / during the execution of layer C_i by the accelerator:

[0019] - before said ofmap output data from layer C_i are stored in said local buffer memories, said packing block determines, according to the configuration word associated with layer C_i+1, the control words, for each of the input data ifmap of layer C_i+1 which correspond to the output data ofmap of layer C_i;

[0020] - then said ofmap output data are inserted into data packets stored in said local buffers with the associated control words thus determined for layer C_i+1 by the packing block;

[0021] i2 / at the beginning of the execution of layer C_i+1 by the accelerator, said data input ifmap of the layer are extracted from the local buffers where they were stored since step il and are provided, with their associated control word for layer C_i+1, to the routers of the compute nodes to which said local buffers are associated;

[0022] disposition j /

[0023] j 1 / at the beginning of the execution of layer C_i+1 by the accelerator, the data of output ofmap from layer C_i having been stored in said buffers local at the end of the execution of layer C_i, said packing block determines, based on the configuration word associated with layer C_i+1, the control words, for layer C_i+1, which are associated with the ifmap input data of layer C_i+1, which correspond to the ofmap output data;

[0024] j2 / then the said ifmap input data of the layer extracted from the buffers local buffers where they were stored since the end of the execution of layer C_i are inserted into packets with their associated control word determined for layer C_i+1, and are thus provided to the routers of the computing nodes to which said local buffers are associated.

[0025] In embodiments, such a method will further comprise at least one of the following features:

[0026] - the determination of the check word for a data point for layer C_i+1 is performed based on at least one field of the configuration word indicating the type of layer C_i+1 and the local buffer storing the ofmap data corresponding to the ifmap data of layer C_i+1;

[0027] - the determination of the check word for a data point for layer C_i+1 is performed based on at least one element among the position of said local buffer and the position of the node associated with said local buffer;

[0028] - the determined control word indicates a type of communication among unicast, broadcast, multicast diagonal, multicast vertical, multicast horizontal and further indicates a direction if the type of communication is multicast;

[0029] - several of said data associated with the same control word and delivered by a The same computing node is inserted into a common data packet associated with the control word determined by the packing block.

[0030] According to another aspect, the invention describes a neural network accelerator adapted for the execution of a neural network comprising at least N successive neural network layers C_i, i = 1 to N, with N>1,

[0031] the neural network accelerator comprising:

[0032] - a global control block,

[0033] - a matrix of computing nodes, each node comprising a router and an element processing element, also called PE associated with a set of respective local memory registers, the processing element being adapted to perform computational operations among multiplications and accumulations of data stored in its registers, the router being adapted to perform several independent data routings in parallel to separate outputs of the router, the routers being linked to each other;

[0034] - at least one row of local buffers, each memory row local buffer being positioned on one side of the compute node matrix, and each local buffer is linked to a router of an associated compute node;

[0035] the routers being adapted to route the data, through the matrix of computing nodes in parallel with the execution by the computing nodes, according to a control word associated with each data;

[0036] said accelerator being adapted so that, at the beginning of the execution by the accelerator of a C_i layer, ifmap input data of the layer are extracted from at least some of said local buffers and are provided, with their associated control word, to the routers of the associated compute nodes

[0037] and so that during the execution of the C_i layer by the accelerator, ofmap output data from the C_i layer from data delivered by the routers of computing nodes associated with local buffers are stored in said local buffers;

[0038] said accelerator being characterized in that it further comprises a packing block disposed between at least some of the local buffers and the nodes associated with said local buffers, and said packing block is adapted to determine the control word associated with the ifmap input data of layer C_i+1 as a function of a configuration word defined for said layer C_i+1, when one or the other of the following provisions i and j are implemented in the accelerator:

[0039] disposition i /

[0040] it / during the execution of layer C_i by the accelerator:

[0041] - before said ofmap output data from layer C_i are stored in said local buffer memories, said packing block is adapted to determine, based on the configuration word associated with layer C_i+1, the control words, for each of the input data ifmap of layer C_i+1 which correspond to the output data ofmap of layer C_i;

[0042] - the accelerator being adapted for said output data ofmap are then inserted into data packets stored in said local buffers with the associated control words thus determined for layer C_i+1 by the packing block;

[0043] i2 / at the beginning of the execution of layer C_i+1 by the accelerator, said accelerator is adapted so that said ifmap input data from the layer are extracted from the local buffers where they were stored since step il and are provided, with their associated control word for layer C_i+1, to the routers of the compute nodes to which said local buffers are associated;

[0044] disposition j /

[0045] j 1 / at the beginning of the execution of layer C_i+1 by the accelerator, the output data ofmap of layer C_i having been stored in said local buffers at the end of the execution of layer C_i, said packing block is adapted to determine as a function of the configuration word associated with layer C_i+1, the control words, for layer C_i+1, which are associated with the input data ifmap of layer C_i+1, which correspond to the output data ofmap;

[0046] j2 / the accelerator is adapted so that the said ifmap input data of the layer extracted from the local buffers where they were stored since the end of the execution of layer C_i are then inserted into packets with their associated control word determined for layer C_i+1, and are thus provided to the routers of the computing nodes to which said local buffers are associated.

[0047] In embodiments, such an accelerator shall further comprise at least one of the following features:

[0048] - the packaging block is adapted to determine the check word for a data for layer C_i+1 depending on at least one field of the configuration word indicating the type of layer C_i+1 and the local buffer storing the ofmap data corresponding to the ifmap data of layer C_i+1;

[0049] - the determination of the check word for a data point for layer C_i+1 is performed by the packing block based on at least one element among the position of said local buffer and the position of the node associated with said local buffer;

[0050] - the determined control word indicates a type of communication among unicast, broadcast, multicast diagonal, multicast vertical, multicast horizontal and further indicates a direction if the type of communication is multicast;

[0051] - several of said data associated with the same control word and delivered by a same computation node (14) are inserted into a common data packet associated with said control word determined by the packing block.

[0052] In one embodiment of the invention, said method further comprises the following provision:

[0053] - a communication mechanism manages the communication of data through the matrix of computational nodes in parallel with the execution of computational operations by the computational nodes;

[0054] a respective configuration word has been previously defined for each of the N layers, the configuration word comprises at least three fields indicating respectively at least the type of layer C_i, the number of filters and the number of input images involved by said layer; a mechanism for sending data, from a local memory to a computing node or vice versa, defines the recipient(s) of the data according to the current configuration word;

[0055] the global control block includes a finite state machine orchestrating the execution of each layer C_i by the grid of nodes and each compute node includes a finite state machine orchestrating the node's operations and the storage of data in its local memory registers, the finite state machines of the nodes and the global control block running in parallel, with synchronization points between them, during the execution of a layer launched by the global control block; and

[0056] said process comprises the following steps:

[0057] before the successive execution of each layer C_i, the global control block obtains the respective configuration word defined for layer C_i and configures its finite state machine according to said at least three fields in said configuration word;

[0058] The execution of the finite state machine of the global control block includes sending to the nodes at least a part of said configuration word defined for the layer and each node configures its finite state machine according to at least said part of said configuration word that was sent to it.

[0059] In embodiments, such a method will further comprise at least one of the following features:

[0060] - the finite state machine of the global control block includes at least the states following:

[0061] loading filter data;

[0062] loading data from an input image;

[0063] waiting for the completion of calculations based on the data loaded with filters and the data loaded with an input image;

[0064] the configuration of its finite state machine by the global control block including the definition of the transitions between said states as a function of said at least three fields in said configuration word;

[0065] - the said calculation operations performed by the processing element are put implemented by one or more dynamically programmed logic gates implemented on FPGA;

[0066] - the configuration word further indicates a number of MAC operators per processing element and each node, before the execution of said layer, dynamically programs one or more logic gates according to said number appearing in said part of said configuration word which was sent to it.

[0067] Similarly, in one embodiment, in the accelerator according to the invention, a communication mechanism is adapted to manage data communication across the matrix of computing nodes in parallel with the execution by the computing nodes of computational operations; a respective configuration word having been previously defined for each of the N layers, the configuration word comprising at least three fields indicating respectively at least the type of layer C_i, the number of filters and the number of input images involved by said layer, said accelerator includes a mechanism for sending data, from local memory to a computing node or vice versa, defining the recipient(s) of the data according to the current configuration word;

[0068] The global control block includes a finite state machine adapted to orchestrate the execution of each layer C_i by the grid of nodes, and each compute node includes a finite state machine adapted to orchestrate the node's operations and the storage of data in its local memory registers, the finite state machines of the nodes and the global control block running in parallel, with synchronization points between them, during the execution of a layer launched by the global control block; and

[0069] the global control block is adapted to, before the successive execution of each layer C_i, obtain the respective configuration word defined for layer C_i and to configure its finite state machine according to said at least three fields in said configuration word;

[0070] the execution of the finite state machine of the global control block including sending to the nodes at least a part of said configuration word defined for the layer and each node being adapted to configure its finite state machine according to at least said part of said configuration word which was sent to it.

[0071] In embodiments, such an accelerator shall further comprise at least one of the following features:

[0072] - the finite state machine of the global control block includes at least the states following:

[0073] loading filter data;

[0074] loading data from an input image;

[0075] waiting for the completion of calculations based on the data loaded with filters and the data loaded with an input image;

[0076] the configuration of its finite state machine by the global control block including the definition of the transitions between said states as a function of said at least three fields in said configuration word;

[0077] - the processing element comprises one or more programmable logic gates dynamically implemented on FPGAs implementing said calculation operations performed by the processing element;

[0078] - the configuration word further indicates a number of MAC operators per element of processing and each node, before the execution of said layer, is adapted to dynamically program one or more logic gates according to said number appearing in said part of said configuration word which was sent to it. Brief description of the drawings

[0079] The invention will be better understood and other features, details and advantages will become clearer from the following description, given by way of non-limiting reason, and from the accompanying figures, given by way of example.

[0080] [Fig-1] Fig. 1 represents a neural network processing system in a method of implementing the invention;

[0081] [Fig.2] Fig.2 represents a neural network processing method implemented work in an embodiment of the invention;

[0082] [Fig. 3] [Fig. 3] represents a memory controller implemented in a mode realization of the invention;

[0083] [Fig.4] Figure [Fig.4] represents the states of a finite state machine of a controller of memory implemented in an embodiment of the invention;

[0084] [Fig. 5] Figure 5 represents a state machine of a global control block used in an embodiment of the invention;

[0085] [Fig.6] Fig.6 represents the structure of a configuration word considered in a method of implementing the invention;

[0086] [Fig.7] Fig.7 represents a computation node considered in a mode of realization of the invention;

[0087] [Fig.8] Fig.8 represents the formats of the data packets in a mode of realization of the invention;

[0088] [Fig.9] Fig.9 represents an activation mask implemented in a mode of realization of the invention;

[0089] [Fig. 10] Fig. 10 represents a Pooling block in one embodiment of the invention;

[0090] [Fig. 11] Figure 11 illustrates the operation of the activation and packaging blocks in an embodiment of the invention;

[0091] [Fig. 12] Fig. 12 illustrates different types of data movement in the accelerator in one embodiment of the invention;

[0092] [Fig. 13] [Fig. 13] illustrates the execution of a first convolution-type layer in an embodiment of the invention;

[0093] [Fig. 14] Fig. 14 illustrates the execution of a hidden layer of the convolution type or depthwise in an embodiment of the invention;

[0094] [Fig. 15] Fig. 15 illustrates the execution of a pointwise hidden layer in an embodiment of the invention;

[0095] [Fig. 16] Fig. 16 illustrates the execution of a hidden layer of the fully connected in an embodiment of the invention;

[0096] [Fig. 17] The [Fig. 17] represents a neural network processing method implemented in one embodiment of the invention;

[0097] [Fig. 18] The [Fig. 18] illustrates a mechanism for recovering communications by calculation implemented in an embodiment of the invention;

[0098] [Fig. 19] The [Fig. 19] illustrates a stage of convolution calculations;

[0099] [Fig.20] The [Fig.20] illustrates another stage of convolution calculations;

[0100] [Fig.21] The [Fig.21] illustrates yet another stage of convolution calculations.

[0101] Identical references may be used in different figures when they refer to identical or comparable elements. Description of the implementation methods

[0102] A DNN comprises successive layers of neural networks of various types, each layer being associated with a set of filters: an input layer, followed by one or more hidden layers (i.e. other than the first and last layers), followed by an output layer.

[0103] The input data is received by the DNN's input layer of neurons and the output data, calculated by the hidden layers of the neural network, is delivered by the output layer.

[0104] Examples of types of neuronal layers include input layers, output layers, rectified linear unit layers, fully connected layers (fc), recurrent layers, long / short-term memory layers, convolutional layers (conv), depthwise layers (dw), dropout layers, pointwise layers (pw), and pooling layers (pool), which include maximum pooling layers,These are called "max pooling" (the maximum value in a subgroup of data is selected) and pooling layers using averaging, called "average pooling" (the average of the values ​​in a subgroup of data is chosen).

[0105] A convolutional layer, for example, analyzes by zones, using each filter (for example: horizontal Sobel, vertical Sobel, etc., or any other filter considered, particularly one derived from training) from the set of filters, at least one data matrix, which is provided to it as input, called Input Feature Map (also called IN or ifmap hereafter) and delivers as output at least one data matrix, called here Output Feature Map (also called Ofmap hereafter), which allows you to keep only what is sought according to the filter considered.

[0106] By way of illustration of such a convolution layer, we consider regular square matrices, although this is not essential for the present invention. In this particular case, we have a regular Ifmap matrix of n rows and n columns, a regular filter matrix F of f rows and f columns, and an Ofmap matrix of m rows and m columns. In a simple case (without padding, and with a stride of '1'), we then have: m = n - f + l. The exact, more complex formula is:

[0107]

[0108] m=(n-f+2p) / s +1, where

[0109] m: Ofmap (mxm)

[0110] n: Ifmap (nxn) [YES] f: filter (fxf)

[0112] p: margin at 0 ("0-padding")

[0113] s: not (stride).

[0114] For example, f = 3 or 5 or 9 or 11.

[0115] More generally, considering that: - the input matrix IFMAP is characterized by its height (i.e. its number of rows) Hin, its width (i.e. its number of columns) Win and its number of channels Cin (a channel thus defines, in a known way, a respective "image" hereafter named Ifmap; the size of the input matrix IFMAP is therefore [Hin Win Cin] (for example [Hin Win Cin] = [255 255 64]); the size of each Ifmap matrix is ​​[Hin Win] and it is made up of its coefficients named ifmaps, ifmaps data or Ifmaps data); - the filter matrix, 'Kernel', is characterized by its height HK, its width WK and its number of channels CK (for example [HK WK CK] = [7 7 128]); - the step corresponds to the Stride matrix, characterized by its height HK and its width WK (for example, [Hs WK] = [3 4]); - the padding is characterized by the Padding matrix: [P_H1, P_H2, P_W1, P_W2] (for example Padding =

[2222] ); Then, the height Hout, the width Wout, and the number of channels Cout of the output OFMAP matrix (each channel thus defining a respective "image" hereafter named Ofmap) satisfy the following conditions:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124] - Hout = (Hin - Hk +P_H1 +P_H2) / Hs +1 ; (in the example shown: Hout = (255 - 7 + 2 + 2) / 3 + 1 = 85) ; - Wout = (Win - WK + P_W 1 + P_W2) / WK + 1; (in the example shown: Wout = (255 - 7 + 2 + 2) / 4 + 1 = 64); - Cost = CK); (in the example shown: Cost = 128). The convolutions performed, in a known manner, correspond, for example, to the following process: the filter matrix is ​​placed on the upper left corner of the IN matrix; a product of each pair of overlapping coefficients is calculated; all the products are summed, giving the value of pixel (1,1) of the output Ofmap matrix. The filter matrix is ​​then shifted horizontally one cell to the right (stride), and the process is repeated, providing the value of pixel (1,2) of the Ofmap matrix, and so on. Once the end of the row is reached, the filter is moved down one cell vertically, the process is repeated starting again from the right, and so on until the entire Ifmap matrix has been traversed. Convolution calculations are generally implemented by neural network computing units, also called artificial intelligence accelerators or NPUs (Neural Processing Units), comprising a network of PE processor elements. In an example, a calculation performed classically in a convolution layer implemented by an accelerator is presented below. Consider the filter F consisting of the following weights: [Tables 1] fl fl f3 f4 f5 f6 f7 f8 fç Consider the following Ifmap matrix: [Tableaux2] ini in2 in3 in4 in5 in6 in7 in8 in9 inio inn ini2 inn ini4 inis inie inn inis ini9 in20 in2i in22 in23 in24 in25 And consider the following Ofmap matrix [Tables 3] OUti out2 out3 out4 out5 out6 out7 out8 OUtg

[0125] The expression of each coefficient of the Ofmap matrix is ​​a weighted sum corresponding to an output of a neuron whose inputs would be mises and the weights applied to the inputs by the neuron would be fj and which would calculate the value of the coefficient.

[0126] Now consider a grid of computational unit elements pe, having as many rows as the filter F (f = 3 rows) and as many columns as the Ofmap matrix has columns (m = 3): [pei,j] i = 0 to 2 and j = 0 to 2. The following is an example of using the grid to calculate the coefficients of the Ofmap matrix.

[0127] As shown in [Fig.19], to each unit calculation element pe of the (i+1)th row of pe, the (i+1)th row of the filter matrix, i=0 to 2, is provided. The grid of pe is then provided with the matrix IN: thus the first row of IN is provided to the unit calculation element peOO; the second row of Ifmap to pelO and peOl, located on the same diagonal; the third row of Ifmap is provided to the unit elements pe20, pel1 and pe02, located on the same diagonal; the fourth row of Ifmap is provided to the elements pe21 and pel2 on the same diagonal, and the fifth row of Ifmap is provided to pe22.

[0128] In a first round of calculations, also shown in [Fig. 19], a convolution (combination of multiplications and additions) is performed in each pe between the filter row provided to it and the first f coefficients of the row of the Ifmap matrix provided to it, yielding a named partial sum (the shaded cells of the Ifmap row are not used for the current calculation). Thus, peOO calculates fl.inl + f2.in2 + f3.in3 etc. Then, the three partial sums determined by the pes in the same column are summed progressively: the partial sum determined by pe2j is provided to pelj, which adds it to the partial sum it previously calculated; this new partial sum resulting from the accumulation is then in turn provided by pelj to peOj, which adds it to the partial sum it had calculated, j = 0 to 2: the total thus obtained is equal to the first coefficient of the j+th row of the Ofmap matrix.

[0129] In a second batch of calculations shown in [Fig. 20], a convolution is performed in each pe between the filter row provided to it and the f = 3 coefficients, starting from the 2nd coefficient, of the row of the matrix IN provided to it, yielding a partial sum (thus peOO calculates fl.in2 + f2.in3 + f3.in4). Then the three partial sums determined by the pes of the same column are summed progressively as described above, and the total thus obtained is equal to the second coefficient of the j+th row of the matrix Ofmap.

[0130] In a third calculation round represented in [Fig. 21], a convolution is performed in each pe between the filter row provided to it and the 3 coefficients f = 3, starting from the 3rd coefficient, of the row of the Ifmap matrix provided to it, yielding a named partial sum. Thus, peOO calculates fl.in3 + f2.in4 + f3.in5 etc. Then the three partial sums determined by the pes of the same column are summed progressively as described above, and the total thus obtained is equal to the third coefficient of the j+th row of the Ofmap matrix.

[0131] In the calculation process described here as an example, the i-th column of pes thus allows the i-th row of Ofmap to be constructed successively, i = 1 to 3.

[0132] It follows from this example that the lines of manipulated data (filter weights, Input Feature Map data and partial sums) are spatially reused between unit processor elements: here for example, the same filter data are used by the pes of the same horizontal line and the same Ifmap data are used by all the pes of diagonal lines, while the partial sums are transferred vertically and then reused.

[0133] Other types of reuse and other directions of spatial propagation may nevertheless be considered, depending on the type of operations to be carried out.

[0134] It is therefore important that, in an accelerator, the communication of these data and the calculations involved are feasible in an optimized manner in terms of transfer time and calculation of access to the central memory initially delivering these data, and this regardless of the dimensions of the input and output data, the types of propagation which are implemented in the PE network, the type of neural layer or the calculations implemented.

[0135] To this end, with reference to [Fig. 1], a neural network processing system 2 comprises a neural network accelerator 1. System 2 also comprises, in the example considered, a DMA controller 18 (DMA stands for "Direct Memory Access") and a DRAM memory 19 (DRAM stands for "Dynamic Random Access Memory") as well as a main processor (CPU, GPU) which are external to the accelerator 1.

[0136] The accelerator 1, in one embodiment of the invention, comprises a global control block 10, a grid 11 of compute nodes 14, a memory controller 12, and a set 13 of buffer memories (e.g., SRAM). The memory controller 12 is adapted to orchestrate the exchanges between the accelerator 1 and the outside world (DMA CTRL 18, DRAM 19, and CPU / GPU subsystem) and to request data transfers between external memories (DRAM 19) and internal memories of the accelerator (buffer memories 13). The global control block 10 is adapted to control the progress of the treatment in accelerator 1, in particular to ensure its overall synchronization.

[0137] The grid 11 of computing nodes comprises computing nodes 14 arranged in a network, for example according to a matrix organization in N rows and M columns and connected by horizontal and vertical communication links allowing the exchange of data packets between nodes 14 (in [Fig.1], N = 3 and M = 5 only as an illustration).

[0138] The buffer set 13 is adapted to store input data (filter and ifmaps) from DRAM 19 before it is provided to grid 11 for layer execution. The buffer set 13 is also adapted to store output data (ofmaps) from a layer.

[0139] A data sending mechanism (from the set 13 of memories to the computing nodes or vice versa) defines the recipient(s) of the data using control packets associated with the data packets to be sent.

[0140] In the example considered, the set 13 of buffer memories comprises:

[0141] - a first subset 13_1 of buffer memories called IGB: IGB_i, i = 0 to N-l+M-1 (here i = 0 to 6) (each of these IGB memories is associated in the example considered with a respective computing node from the set 13_11 of nodes composed of the nodes 14 of the first column or from the set 13_12 of nodes composed of the nodes of the last row of grid 11: each local IGB buffer memory is connected to the router of the computing node which is associated with it);

[0142] - a second subset 13_2 of buffer memories, called OGB: OGB j, j = 0 to Ml (each of these OGB memories is associated in the example considered with a respective computing node among the set of nodes composed of the nodes 14 of the first row of grid 11: each local OGB buffer memory is connected to the router of the computing node which is associated with it).

[0143] The size of the minimum accelerator architecture depends on the type of operations to be performed. Here, in the case of a convolution operation, the size of the minimum accelerator architecture is based on the number of filter rows and the number of Ofmap rows. For example, if the filter size is 6 x 6, then the number of rows in the grid of 11 compute nodes would be 6, and the number of columns would be equal to the number of rows in the Ofmap. The general formula giving the size (Hout, Wout, Cout) of the Ofmap output was given above.

[0144] The neighbors located to the North (N), South (S), East (E) and West (W) will be discussed below: these directions are indicated on [Fig.1] (relative to a location of a node symbolized by a cross).

[0145] Accelerator 1, for example, has an architecture based on a Network on Chip (NOC). The NOC is, for example, designed for simultaneous data transfer (between node routers in particular) via parallel and parametric multipath routing to ensure high bandwidth in different topologies, for example as described in FR 3133936.

[0146] In embodiments, the accelerator 1 further comprises a pooling block 15 and / or a packaging block 16 and / or an activation block 17.

[0147] In one embodiment, each computing node 14 comprises, with reference to [Fig.7], a processing element 50, a router 52 and a clock gating control block 53.

[0148] The processing element 50, also called PE 50 (for "Processing Element" in English), comprises (and this in a parameterizable manner in an embodiment of the invention described below) one or more unit calculation blocks 51, each comprising:

[0149] - memory registers (RF) in particular intended to store filter data (RF 510) and the Input Feature Map input data (RF 511) received by compute node 14 and

[0150] - a MAC 57 operator adapted to perform MAC-type calculation operations in particular based on this data stored in registers 510, 511, using a logical multiplication cell 512, a logical accumulation cell 513 and an logical addition cell 514.

[0151] The processing element 50 further comprises a PE controller 58 and multiplexers (mux) 55, 56, each driven by output selection logic in this PE controller 58. The multiplexer 55, for example, has as inputs the respective outputs of the unit computing blocks 51, and only one of these inputs is chosen as the output of the mux 55 (for example, the selected partial sum Psum will then be transmitted, via the router 52, to a neighboring computing node, for example, the one to the North). In some embodiments, the processing element 50 includes additional memory registers for storing results (partial sums, accumulations of partial sums) calculated by the PE 50.

[0152] The PE 50 therefore does not include a processor programmable by reading software instructions, it consists of ALU units (“Arithmetic Logic Unit”).

[0153] Router 52 is adapted to route data communications entering or leaving node 14. All routers of accelerator 1 are interconnected using one of several possible topologies (mesh, torus or ring) to form the NoC network, which provides communication between the IGB 13_1 / OGB 13_2 buffers and the compute nodes 14, and between the compute nodes 14 themselves.

[0154] The result of a calculation carried out by a PE 50, called partial sum (PSum), will be transmitted to the neighboring calculation node 14 to the North to finally carry out the final accumulation operation generating the Ofmap.

[0155] The clock control block 53, in node 14, is adapted to optimize the device's energy consumption. In one embodiment, the architecture of accelerator 1 remains fixed for the execution of a given DNN, and since the shape of the layers of the same DNN is variable, the clock gating mechanism ensures that only the PE 50s of nodes 14 allocated for execution remain active. The others are deactivated by the mechanism. The clock control block 53, comprising an "AND" logic gate powered by the global clock and a "1" signal controlled by the activation control logic, compares an activation mask (see example mask in [Fig. 9]) with the local address of the node and decides whether or not to activate the node, depending on its position. These activation masks are stored in IGBs 13_1 between the layers of the DNN network.Horizontal multicasting is performed across the entire grid 11, and only PE 50s from nodes 14 with a Y value lower than the mask values ​​remain active. The others will be deactivated (IF Y of PE[X, Y]>m => PE inactive).

[0156] In order to enable the computational communication recovery mechanism described below in an example, the routing device 52 is adapted to manage routing paths in parallel. To this end, it includes, by reference, a block of parallel routing controls, a block of parallel arbitrators, a block of parallel switches, and a block of parallel input buffers.

[0157] Indeed, through these different buffering modules (for example FIFO, First-In-First-Out) of the parallel input buffer block, different data communication requests (filters, IN data or psums) received in parallel (for example from a neighboring block to the East (e_in), to the West (o_in), to the North (n_in), to the South (s_in), or locally (L_in) of the PE or registers) can be stored without any loss.

[0158] These requests are then processed simultaneously in several control modules within the parallel routing control block, based on the Flit (Flow control unit) headers of the data packets. These routing control modules ensure deterministic control of data transfer according to a static XY routing algorithm (for example) and manage different types of communication (unicast, horizontal, vertical or diagonal multicast, and broadcast).

[0159] The resulting requests issued by the routing control modules are provided as input to the parallel arbiter block. Parallel arbitration of the priority order of processing incoming data packets according to, for example, the Round-Robin arbitration policy based on planned access allows for better collision management, i.e. That is to say, a request that has just been granted will have the lowest priority in the next arbitration cycle. In the event of simultaneous requests for the same output to the East, West, North, and South (given that the outputs are labeled as follows in [Fig. 7]: for East (e_out), for West (o_out), for North (n_out), for South (s_out), or Local (L_out)), the requests are logged to avoid deadlock or data loss (i.e., two simultaneous requests for the same output within the same router 12 are not served in the same cycle). The arbitration decision is then communicated to the parallel switch block.

[0160] Parallel switching simultaneously directs data to the correct outputs according to the Wormhole switching rule, for example, that is to say that the connection between one of the inputs and one of the outputs of a router is maintained until all the elementary data of a message packet are sent and this simultaneously through the different switching modules for their respective direction North, East, South, West, Local.

[0161] The data packet format is shown in [Fig. 8]. The packet is of configurable size Wdata (32 bits in the figure) and consists of a header flit followed by payload flits. The packet size depends on the size of the interconnection network, since the more routers there are, the more bits are needed to encode the addresses of the senders or recipients. Similarly, the packet size varies with the size of the payloads (filter weights, input activations, or partial sums) to be circulated through the grid. The header value determines the communication to be handled by router 5. Several types of communication are possible: unicast, horizontal multicast, vertical multicast, diagonal multicast, broadcast, and memory access.Router 52 first receives the control packet, also called the control word, containing the communication type and the destination or source, identified by its coordinates (Addr_source or dest_X, Addr_source or dest_Y) in the grid, as shown in [Fig. 8]. Router 52 decodes this control word and then allocates the communication path to transmit the payload packet (associated with the control word), which arrives in the cycle following the reception of the control packet. The corresponding packet pairs are shown in [Fig. 8] (i, ii, iii). Once the payload packet has been transmitted, the allocated path will be freed for other transfers.

[0162] For memory access, reading from IGB 13_1 is a multicast communication; writing to an OGB 13_2 is a type of communication that handles a direct parallel unicast from the first rows of nodes 14 to the OGB. Sending a configuration word is handled in broadcast mode. Router 52 receives first the control packet containing the type of communication and the source or destination address.

[0163] System 2 thus uses three levels of hierarchical storage:

[0164] - the DRAM memory 19 which contains all the input data to be provided to accelerator 1, previously loaded from a central processor (CPU or GPU type); the output data delivered by the neural network will be transmitted from accelerator 1 to DRAM 19 and then interpreted by the main processor;

[0165] - the set 13 of buffer memories which are distributed on the grid 1 of nodes of computation to facilitate parallel access to data; communication between DRAM 19 and the buffer memories of set 13 is managed by the memory controller 12 via the global control block 10 and the DMA controller 18; as will be described below, these buffer memories can be used to exploit data reuse, thus masking the access latency to DRAM 19 and storing intermediate data between distinct layers of the neural network running on the accelerator 1; the size of set 13 of buffer memories is for example equal to 1 megabyte;

[0166] - the RF registers (for example referenced 510, 511), distributed in each node 14 compute nodes that are directly connected to the MAC operators 57 of this compute node; an RF communicates with the RFs of neighboring compute nodes 14 and buffers of the set 13 neighbors via the local router 52 of the node concerned; the size of an RF does not exceed 1 kilobyte.

[0167] Accessing data via these different levels involves different energy costs. The highest cost is DRAM and the lowest is RF.

[0168] The Pooling block 15 is adapted to perform a classic "pooling" processing in DNNs, i.e. of pooling corresponding to a kind of subsampling of the data which are provided as input to block 15, by choosing a maximum, an average value, among several neighboring data.

[0169] An example of a Pooling Block 15 is shown in [Fig. 10]. The pooling layer applies a kernel to the input ifmap image in a 2D representation. There are different types of pooling, such as maximum pooling and average pooling. This layer is used to make feature maps robust to the location of the features to be detected. This layer is generally applied to the outputs after the convolution calculation. For this reason, this module was placed separately from the grid 11 of compute nodes 14 to perform the pooling to the ofmaps before storing them in the OGB 13_2. The global control block 10 sends: - (i) a 2-bit "p" signal to the C controllers to inform them of the type of pooling (max or average) and step size, called "stride" (1 or 2), - (ii) 2-bit "salt" signal to the multiplexers to select the data to be sent to the OGB 13_2 (i.e., data without pooling (N) or data with pooling with stride of 1 (SI) or 2 (S2)).

[0170] Depending on the pooling type and stride size, controller C performs a comparison between two adjacent input data. The output of controller C represents the result (S1) of pooling with stride 1 for the multiplexer associated with that C. The wiring has been predefined to associate the output of C with a neighboring multiplexer to represent the result (S2) of pooling with stride 2.

[0171] Fig. 11 illustrates the operation of the activation blocks 17 and packaging blocks 16 (in another embodiment, their direction is reversed: the set 170 of the data 01, 02, 03 which are Ofmap data delivered by the grid 11 and from the accumulations of Psum then pass first through the packaging block 16, and then through the activation block 17).

[0172] Activation block 17 is suitable for applying:

[0173] - a mathematical function (a so-called 'sat' function, defined for example in a generic parameter when generating an architecture configuration) on the data 01, 02, 03 provided as input, for example the Rectified Linear Unit function, commonly called ReLU, which replaces all negative values ​​received as input with zeros (ReLU(x) = max(0,x)) (alternatively, the activation function is the hyperbolic tangent function (tanh(x)), the saturating hyperbolic tangent function (|tanh(x)|), the sigmoid function (1+ex)*); then

[0174] - an offset based on the content of the "Shift" field (defining the number of right shift bits to apply) of the configuration word,

[0175] For example, the value returned by the sat function is a saturated value of the 8-bit weighted sum. In the case where the activation function is a ReLU, the sat function will set the variable x to 0 if x < 0. Then, in the case of an 8-bit integer calculation, this variable will be shifted to the right by a certain number of bits defined in the "Shift" field of the configuration word.

[0176] The data conditioning by the packing block 16 includes, in the embodiment considered by way of example, a process consisting of encapsulating a set 170 of several Ofmaps data, here NPACk (here NPACk = 3), successive Ofmaps output data (n_x 8 bits) from the same node (named 01, P2, 03 in [Fig. 11]) in the same payload packet 160, and then adding the associated control word to prepare the data for the next layer. These packets are then, for example, sent to the OGB 13_2, awaiting execution by the next layer.

[0177] The processing of data streams by system 2 is based in particular on:

[0178] - adaptive data reuse through data transfers multidirectional in parallel, as illustrated in [Fig. 12] in particular, which reduces costly access to memories and electrical energy consumed;

[0179] - a recovery of communications by calculation, by separating the control of These two aspects mask the cost of data transfers to improve performance;

[0180] - an execution on demand, driven by the PE 53 clock control blocks.

[0181] - a suitable synchronization: between Globally Synchronous Locally Globally Asynchronous Locally Synchronous (GALS), which ensures optimal spatiotemporal movement of data.

[0182] For example, with reference to [Fig.1], calling G1 the group of all the nodes 14 of the grid 11 for the calculation of their respective Ofmap outputs, calling G2 the group of nodes N_00, N_10 and N_20 (corresponding to 1 row of Ofmap) and calling G3 the group consisting of node N_04 (for the calculation of the Psums): G1 / G2 GAIS: the nodes of the same column in a group G2 perform the calculation synchronously with each other (they receive the data at the same time: therefore Locally Synchronous within G2), but the columns constituting the group G1 are asynchronous with each other; which means that the nodes belonging to different columns are asynchronous (Globally Asynchronous); and GLG3 GSLA: each node represented by a group G3 (for example N_04) performs the calculation independently of the other nodes of the same group G3 (Locally Asynchronous);Group G1 must wait for the calculation to finish in all nodes of group G3 in order to trigger the execution of a new layer of a DNN (Globally Synchronous Network).

[0183] Illustration, in an example, of the recovery of communications by computation

[0184] The communication recovery mechanism by computation implemented in accelerator 1 is illustrated with respect to [Fig. 18] where the set of data in the (i+l)th row of weights in the filter matrix is ​​denoted Frowi, i = 0 to p-1; the set of data in the (i+l)th row of the IN matrix is ​​denoted inrowi, i = 0 to n-1; the data resulting from a partial sum calculation performed by the PEÿ is denoted psumÿi = 0 to 3 and j = 0 to 3. The computation nodes 14 are arranged in a network, for example here according to a matrix organization in N=4 rows and M=4 columns.

[0185] In [Fig. 18], a calculation node 14 (and similarly its PE 50) is identified by its row and column position in the grid. Thus, the node (i,j), comprising the PEij, is located on the i+1st row and j+1st column of the grid 11, i = 0 to 3 and j = 0 to 3.

[0186] Each node 14 not located on the edge of the network thus has 8 neighboring nodes, in the following directions: one to the North (N), one to the South (S), one to the West (W), one to the East (E), one to the Northeast, one to the Northwest, one to the Southeast, and one to the Southwest.

[0187] The global control block 10 is adapted to synchronize the computational processing in the PE and data transfer between or within nodes and implemented in the accelerator 1. All these processes are timed by a clock of the accelerator 1.

[0188] There will have been a preliminary step of configuring grid 11 to select the set of PEs to be used, from among the PEs available in the maximum hardware architecture of accelerator 1, for applying the considered filter from a layer of the neural network to an Ifmap matrix. During this configuration, the number of "active" rows of grid 11 is set equal to the number of rows of the filter (p) and the number of "active" columns of grid 11 is taken equal to the number of rows of the OUT matrix (m). In the case represented in [Fig. 18], these numbers p and m are equal to 4 and the number n of rows of the Ifmap matrix is ​​equal to 7.

[0189] The arrows in [Fig. 18] represent how the data are reused in grid 11. Indeed, the rows of the same filter, Frowi, i = 0 to p-1, are reused horizontally across the PEs (this is therefore a horizontal multicast of the filter weights), the inrowi rows of IN, i = 0 to n-1 are reused diagonally across the PEs (a diagonal multicast of the input image, here implemented by the succession of a horizontal multicast and a vertical multicast) and the partial sums psum are accumulated vertically across the PEs (this is a unicast of the psum), as represented by the vertical dashed arrows.

[0190] The accelerator 1 is adapted to implement, in the inference phase of the neural network, the parallel reuse, described above, by the PEs of the three types of data which are the filter weights, the input data of the Input Feature Map matrix and the partial sums, as well as the recovery, by calculation, of the communications, in an embodiment of the invention.

[0191] Accelerator 1 is particularly suitable for implementing the steps described below.

[0192] In a first step, a parallel feed of the grid 11 is made with the filter weights and the input data of the Ifmap matrix, via a bus between the buffer memories 13_1 and the grid 11.

[0193] Thus, at the processing cycle T0 (the cycles are timed by the accelerator 1 clock):

[0194] the first column of grid 11 is fed by the respective lines of the filter: the Frowi weight line, i = 0 to 3 is provided as input to the node (i, 0), here by the IGB_i 13_1 (not shown in [Fig. 18]);

[0195] The first column and the last row of grid 11 are fed by the respective rows of the Input Feature Map matrix: the row inrowi, i = 0 to 3 is provided as input to the node (i, 0) (here by the IGB_i 13_1 (not shown in [Fig. 18])) and the row in rowi, i = 4 to 6 is provided as input to the node (3, i-3) here by the IGB_i 13_1 (not shown in [Fig. 18]).

[0196] In cycle T1 following cycle T0, the weights and data from the Ifmap matrix received by each of these nodes 14 are stored in respective registers 511, 510 of node 14.

[0197] In a subsequent step, the dissemination of filter weights and input data within the network is iterated: it is carried out in parallel, by horizontal multicast of filter weight lines and diagonal multicast of Input Feature Map lines.

[0198] Thus in cycle T2:

[0199] The first column sends, by horizontal diffusion, to the second column of grid 11 the respective lines of the previously stored filter: the Frowi weight line, i = 0 to 3, is provided as input to node (i, 1) by node (i, 0); and in parallel

[0200] Each of the nodes (i, 0) transmits the line inrowi, i = 1 to 3, and each of the nodes (3, i-3) transmits the line inrowi, i = 4 to 6), to the processing block 10 which is respectively neighboring them in the NE direction (for example the node (3,0) transmits to the node (2,1)): to reach this destination, it will take in the present case, two transmissions: a horizontal transmission and a vertical transmission (for example for the data to arrive from the node (3,0) to the node (2,1), it will go from the node (3,0) to the node (3,1), then to the node (2,1): so first the neighbors to the East of the nodes (i, 0), i = 1 to 3 and of the nodes (3, i-3) receive the line;

[0201] the first column of nodes having the filter weights and the input data of the Ifmap matrix, the PEs of these nodes implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psumOj by the PEOj, j = 0 to 3, is stored in a register of the node.

[0202] At cycle T3, the filter weights and data from the Ifmap matrix received at T2 by these nodes at T2 are stored in respective registers of each of these nodes.

[0203] In cycle T4, in parallel:

[0204] the second column feeds, by horizontal diffusion, the third column of grid 11 through the respective lines of the filter previously stored: the Frowi weight line, i = 0 to 3, is supplied as input to node (i, 2) by node (i,l);

[0205] the nodes (i-1, 1) receive the line inrowi, i = 1 to 3 and each of the nodes (2, i-2) receive the line inrowi, i = 4 to 5;

[0206] the second column of the nodes 14 having the filter weights and the input data of the Ifmap matrix, the PEs of these nodes implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum^ by the PEij, j = 0 to 3, is stored in a local memory register at the node.

[0207] At cycle T5, the filter weights and data from the Ifmap matrix received at T4 by these nodes are stored in respective registers of each of these nodes.

[0208] In cycle T6, in parallel:

[0209] the third column feeds, by horizontal diffusion, the fourth column of grid 11 by the respective lines of the filter previously stored, thus completing the diffusion of the filter weights in grid 11: the Frowi weight line, i = 0 to 3, is supplied as input to node (i, 3) by node (i,2);

[0210] nodes having received at time T4 a line of the Ifmap matrix and having a neighbor in the direction of the NE in turn emit this line of Ifmap matrix towards this neighbor.

[0211] At cycle T7, the filter weights and data from the Ifmap matrix received at T4 by these nodes are stored in respective registers of each of these nodes.

[0212] At cycle T8, the third column of nodes having the filter weights and the input data of the Ifmap matrix, the PEs of these nodes implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum2j by the PE2j, j = 0 to 3, is stored in a register of the node.

[0213] nodes having received at time T6 a line of the Ifmap matrix and having a neighbor in the direction of the NE in turn emit this line of Ifmap matrix towards this neighbor.

[0214] Diagonal diffusion continues.

[0215] At cycle T12, the node (03) in turn received the line inrow3.

[0216] The fourth column of nodes having the filter weights and the input data of the IN matrix, the PEs of these nodes implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum3j by the PE3j, j = 0 to 3, is stored in a register of the node.

[0217] Next, a parallel transfer of the psum partial sums is carried out, and these psums are accumulated: the nodes of the last row of grid 11 each send the calculated partial sum to their neighbor located to the North. This neighbor adds this received partial sum to the one it previously calculated and in turn sends the accumulated partial sum to its North neighbor, which repeats the same operation, and so on, until the nodes of the first row of grid 11 have This accumulation was performed (all these processes being carried out in time with the clock of accelerator 1). This final accumulation, performed by each node (0,j) j = 0 to 3, corresponds to (a part of) the data from row j of the OUT matrix. It is then delivered by node (0,j) to the OGB 13_2 buffer for storage.

[0218] The Output Feature maps of the convolution layer considered are thus determined as a function of the outputs Outl<)WI, i = 0 to 3.

[0219] The diffusion of filter weights is carried out in accelerator 1 (a multicast of filter weights with horizontal reuse of filter weights across nodes) in parallel with the diffusion of input data of the Ifmap matrix (multicast of image rows with diagonal reuse across nodes).

[0220] Overlapping communications through computation reduces data transfer costs by improving parallel program execution time and decreasing the effective contribution of data transfer time to the overall application execution time. Computations are decoupled from data communication in the grid so that the PE 50 perform computational work while the communication infrastructure (routers 52 and NOC communication links) performs data transfer. This partially or completely hides the overhead of communication, although perfect overlap is only possible if computation time exceeds communication time and the hardware supports this paradigm.

[0221] In the embodiment described above with respect to [Fig. 18], all psums are expected to be calculated before their accumulation. In another embodiment, psum accumulation is initiated on the first columns of the network while the transfer of filter data and Ifmap matrix data continues in the columns further east, and therefore the psums for these columns have not yet been calculated. In this case, there is an overlap of communications by the communications of the partial psums, which further reduces the contribution of data transfers to the total execution time of the application and thus improves performance. The first columns can then, optionally, be used more quickly for other storage and calculations, thereby further improving the overall processing time.

[0222] The operations have been described above for example with respect to the first layer in the particular case of an RS (Row Stationary) Dataflow and a convolutional layer of type Conv2D (cf. Y. Chen et al. 2017. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-State Circuits 52, 1 (Nov. 2017), 127-138).

[0223] Other types of Dataflow execution (WS Weight-Stationnary Dataflow, IS Input-Stationnary Dataflow, OS Output-Stationnary Dataflow, etc.) involving other data reuse schemes between PEs, therefore other transfer paths, other computational organizations, other types of CNN layers (Fully Connected, PointWise, depthWise, Residual), etc., can be implemented according to the invention: the data transfers of each data type (filter, ifmap, psum), for their parallel reuse, must thus be able to be carried out in any of the possible directions in the routers, and this in parallel with the data transfers of each other type (it should be noted that embodiments may of course use only some of the proposed options: for example, the spatial reuse of only a subset of data types among filter type data, Input Feature Maps, partial sums).To this end, the accelerator is configurable in order to adapt access to input data in memory and to adapt the direction of data transmission, depending in particular on the type of neural network layer to be executed.

[0224] Two innovative aspects are now described. In the example considered, they are both implemented in the same system 2. However, they are independent of each other, and each can be implemented in a system without the other. These two innovative aspects are:

[0225] - FSM-based architecture and corresponding configuration method

[0226] - architecture with data packaging and corresponding method

[0227] and are described in the chapters below.

[0228] FSM-based circuit architecture and corresponding CONFIGURATION method

[0229] In one embodiment, the global control block 10 includes, with reference to Figures 1 and 5, a configuration register 20 and a finite state machine 40, referred to as FSM 40.

[0230] In one embodiment, this FSM 40 of the global control block 10 is configurable according to a configuration word.

[0231] The global control block 10 drives the operation of the accelerator 1 for the calculation of ofmaps by each layer of a DNN neural network running on it. To do this, the global control block 10, by unwinding the FSM 40, based on a configuration word selectively associated with each layer, is adapted to identify the shape of the layer C_i that is now to be executed, to configure its FSM 40 according to this configuration word, and then to unwind this configured FSM, which triggers the feeding of the computation nodes 14 with the characteristic input data (ifmaps) and filters stored in the set 13 of buffers, while waiting for the completion of the current task. and allows accelerator 1 to be synchronized to trigger a new execution. This is repeated until the end of the computation, when the output results of the layer are retrieved from buffer set 13 and sent to DRAM 19, via the DMA controller 18.

[0232] The FSM 40 of the global control block 10 comprises, in the example considered, five states 41 to 45, which it goes through in a loop, in the manner described below with reference to [Fig.2],

[0233] The five states are “INACTIVE” 41, “LOADING CONFIGURATION” 42, “LOADING FILTER” 43, “LOADING ifmap” 44 and “WAITING” 45.

[0234] Rules govern, on the one hand, the number of times a given state 4i is repeated (rule rü) and, on the other hand, the triggering of a transition from a state 4i to a state 4j (rule ry), with i and j being integers from 1 to 5. These rules are, in particular, a function of receiving control signals and current counter values. Looping allows the system to remain in a state until the completion of the task to be performed by that state. Looping or the transition from one state to another allows the system to move on to another function to be performed by the accelerator 1.

[0235] In the "INACTIVE" state 41, the global control block 10 is inactive (it loops back in this state) until it receives a signal indicating that an execution of a neural network is to be launched by the accelerator 1 (ri2).

[0236] The "LOAD CONFIGURATION" state 42 allows configuration of the FSM 40 according to the configuration word associated with the current layer to be executed, as described further below with reference to [Fig.2], and then sending a start execution control signal from the C_i layer to the memory controller 12 and the compute nodes 14.

[0237] The "LOAD filter" state 43 triggers, via the sending of the "Load filter" control signal by the global control block 10, to the nodes 14, the transmission of filter data used by the layer in question, from predefined buffers of the set 13 of buffers to the routers of the nodes 14 directly connected to them. This filter data, like all data circulating in the grid 11 between the routers 52 and communicated between neighboring routers, is accompanied by associated communication control words allowing the routers 52 of the grid 11 to route the received data. Depending on this control word, the routers transmit the associated data in unicast, multicast, or broadcast.

[0238] Rule r33 conditions the number of times the FSM loops back in this state 43; it therefore depends on the number of data in a line of a filter; for example, it is implemented with a dedicated counter whose maximum value is defined in the configuration word 50 (field 50_13, see [Fig.6]), defined later.

[0239] The "LOADING ifmap" state 44 triggers, via the sending of the "Load ifmap" control signal by the global control block 10, to the nodes 14, the sending of the data The ifmaps used by the layer in question are sent from predefined buffers in the set of 13 buffers to the routers directly connected to them. Depending on the control word associated with the data in a payload packet (i.e., an ifmap or NPAck data item, depending on the implementation: see below), the routers transmit this data in unicast, multicast, or broadcast. Loopback in state 44 is conditioned by the application of the rule r^, a function of the number of data items in an ifmap row, via a dedicated counter whose maximum value is defined in the configuration word 50 (field Ifmap_H 50_14 indicating the number of rows in the ifmap matrix, see [Fig. 6]), defined later.

[0240] When the filter data and ifmaps data have been sent to grid 11, the FSM goes to the "WAIT" state 45 (rule r45): the start_compute signal is then sent by the global control block 10 to the compute nodes 14: the MACs operations in the compute nodes 14 then take place; once a partial sum is calculated in a node 14 of the last row of grid 1, the partial sum is sent to the compute node located to the North of the compute node in question; this compute node 14 adds the received partial sum to the partial sum it has calculated locally and in turn sends the result to the neighboring compute node 14 located to the North; this processing is continued until the accumulation of sums in a compute node 14 located on the first row of the grid;the corresponding partial sum is then stored in the buffer of set 13 associated with the computation node (in the example the result from N_0k is stored in OGB_k, k= 0 to 4), if necessary after additional processing carried out by the pooling block 15 and / or the activation block 17 and / or the packaging block 16. When this processing is completed for all nodes 14 of the first row of the grid, an end of task execution signal (the "End computation" signal received by the FSM) is sent by the node located in the northeast corner of the grid (node ​​N_04 in the example), signaling the end of the execution of the current layer (the loopback rule r55 in state 45 is therefore a function of the presence or absence of this signal), the FSM returns to the "LOADING CONFIGURATION" state 42 (r52).

[0241] The partial sum data stored in the buffer of set 13 associated with the calculation node will then, depending on the case, be sent to DRAM memory or directly reinjected into grid 11 for processing the next layer.

[0242] The r54 rule for controlling the switching or not from state 45 to state 44 is conditioned by the number of ifmaps in a layer; it is implemented via a dedicated counter whose maximum value is defined in the configuration word 50 (field 50_25, see [Fig.6]), defined later.

[0243] The r53 rule for controlling the switching or not from state 45 to state 43 is conditioned by the number of filters in a layer; the rule is implemented via a dedicated counter, the maximum value of which is defined in the configuration word 50 (field 50_23).

[0244] Rule r52 triggering the switchover or not from state 45 to state 42 is conditioned by the "End computation" signal and the reaching of the maximums of the two counters mentioned previously in reference to rules r53 and r54.

[0245] The r2iest rule is conditioned by the end of all layers.

[0246] The memory controller 12 includes, in one embodiment, with reference to [Fig.3], a simple finite state machine 30 (FSM) for triggering the reading of input data from DRAM 19 to buffer memories of set 13 (for example, in the embodiment considered, to IGB 13_1) and the writing of output data from buffer memories of set 13 (for example, in the embodiment considered, data from OGB 13_2) to DRAM 19, based on control signals indicating the state of the memories and the processing on the accelerator 1. The memory controller 12 also includes a multiplexer (MUX) 36 and a demultiplexer (DEMUX) 35 for managing all-to-one and one-to-all memory accesses, respectively.The FSM 30 sends control signals to the DEMUX 35 to select which of the IGBs to fill and sends control signals to the MUX 36 to select which of the OGBs whose content will be delivered.

[0247] As shown in [Fig.4], the loop states of FSM 30 of memory controller 12 are successively:

[0248] - "INACTIVE" state 31 (in English IDEE), until a starting signal is received (from the external host (GPU or CPU)), then switching to state 32;

[0249] - state "Configuration Filling" 32, in which the configuration words associated with the respective layers of the neural network to be executed are obtained and loaded into the accelerator 1, as described later; the switch from state 32 to state 33 is controlled by a configuration word counter whose threshold value therefore depends on the number of layers;

[0250] - "IGBs Filling" state 33, in which the DRAM data (data input and filter data) are loaded via the DMA controller 18 and the DEMUX 35, into the IGB buffer memories, until the IGBs are full (which is indicated via a counter of the number of input and filter data, which will be stored in the IGB memories); then switching to state 34;

[0251] - "EoE waiting" state 34: this state corresponds to waiting for the end of execution (EoE (for English "End of Execution") calculations relating to data loaded into IGBs and not requiring new IGB refills, until a signal indicating the end of the execution of all layers of a DNN network is received (from global control block 10), then switching to state 35;

[0252] - state "Read OGBs" 35, in which the OGBs buffers are read, then transmitted to MUX 36 for storage in DRAM 19 via the DMA controller 18, until all OGBs have been read (which will be indicated via a counter previously set by the number of ofmaps of the last layer), then switching back to the "INACTIVE" state 31 (the switch to state 31 is triggered when the counter reaches the maximum value representing the number of all ofmaps of the last layer).

[0253] In one embodiment, the PE 58 controller of the processing element 50 of a computing node 14 shown in [Fig. 7] also includes, for example, a configurable finite state machine governing its behavior, in particular controlling:

[0254] - triggering or not the update of filter data or ifpmaps;

[0255] - triggering a specific operation among multiplication, addition, accumulation.

[0256] In one embodiment, the states of the PE 58 controller's finite state machine are configured according to the content of at least some of the fields of the configuration word relating to the layer being processed, as described below.

[0257] Configuration method (of FSMs in particular) based on the configuration word

[0258] In one embodiment, the neural network processing system 2 is adapted to implement the neural network processing method 100 schematically represented in [Fig.2] and comprising a configuration method.

[0259] Consider a predefined DNN neural network, comprising n successive predefined neural network layers C_1, ..., C_n. System 2 is implemented to provide the necessary input data to accelerator 1 to compute the corresponding output data of this DNN neural network.

[0260] A configuration word CONFIG_i has been predefined for each of these layers C_i, i = 1 to n, in the manner represented in [Fig.6] in one embodiment.

[0261] The configuration word 50 is presented here for example in the form of a double control word 50_1 and 50_2 (i.e. 2 x 32 bits in total).

[0262] The configuration word 50 comprises, distributed between the two control words 50_1 and 50_2, which include the parameter values ​​defining the layer associated with the configuration word:

[0263] - a 50_12 field, called a "layer" field, indicating the type of network layer of neurons (typically conv, dw, pw, pool, fc, ...);

[0264] - a 50_13 field, called the "Filter_H" field, indicating the filter height;

[0265] - a 50_14 field, called the "Ifmap_H" field, indicating the height of the input data ifmaps;

[0266] - a 50_15 field, called the "Shift" field, indicating the size of the bit jump (in English " shift") in the activation function;

[0267] - a 50_16 field, called the "NumMac" field, indicating the number of MAC operators per calculation node 14;

[0268] - a 50_17 field, called the "stride" field, indicating the pitch;

[0269] - a 50_18 field, called the "padding" field, indicating the margin;

[0270] - a 50_22 field, called the "F" field, indicating '0' in the case of a single filter considered in the network layer and indicating '1' if the number of filters in the network layer under consideration is greater than or equal to 2;

[0271] - a field 50_23, called the "numFilter" field, indicating the number of filters in the layer considered;

[0272] - a 50_24 field, called the "I" field, indicating '0' in the case of a data matrix input, ifmap, unique considered in the network layer and indicating '1' if the number of input images of the considered network layer is greater than or equal to 2;

[0273] - a 50_25 field, called the "Channel_I" field, indicating the number of ifmap images input for the layer in question.

[0274] As in the present case, the configuration word 50 is in the form of a double control word 50_l and 50_2 (i.e. 2 x 32 bits in total), each control word 50_l and 50_2 here also includes a first 4-bit field (called opcode field), respectively 50_l 1 and 50_21 indicating that it is a component of the configuration word 50.

[0275] With reference to [Fig.2], in a step 101, prior to the execution by the accelerator 1 of the processing to calculate the output data of a predefined DNN neural network to which input data are provided as input, the ordered list of configuration words CONFIG_i, i = 1 to n, in the order of succession of the layers from the input layer to the output layer, is stored in the system 2 and is made accessible to the global control block 10: it is considered here that it is for example stored in the configuration register 20 of the global control block 10.

[0276] The global control block 10 then implements step 102 before the actual processing, by the accelerator 1, of each layer C_i, i = 1 to n.

[0277] Global control block 10 extracts the configuration word CONFIG_i from configuration register 20 for the next layer C_i to be processed: at this stage, the FSM 40 of global control block is in state 42 "LOADING CONFIGURATION" "(if all neural network layers have been executed by accelerator 1, FSM 40 returns to the "INACTIVE 41" state).

[0278] In this state 42, the initial values ​​of counters, on which the rules for repeating the same state or on which the transitions from one state to another depend, are fixed according to the relevant fields of the configuration word CONFIG_i.

[0279] Thus, in this step 102, in a substep 102_l, the global control block 10, its FSM 40 being in state 42, configures this FSM 40 (which will be used to orchestrate the part incumbent upon the global controller 10, of the execution of layer C_i) according to this configuration word CONFIG_i. In particular, via the configuration of at least some of the transition / loopback rules according to the configuration word CONFIG_i: - the number of times a given state (for example state 43 or 44) will be iterated during the processing of layer C_i is determined based on this configuration word. - the number of times we will transition from one defined state to another defined state during the processing of the C_i layer is fixed, (for example, going from state 45 "WAITING" to state 44 "LOADING ifmap", or the number of times we will go from state 45 "WAITING" to state 43 "LOADING filter", or the cases where we will go directly from state 43 to state 45 (without going through state 44)).

[0280] For example, the transitions between states must be adapted for the execution of a layer where:

[0281] - in the case of a layer involving k filters (which is indicated by the field 50_23 "numFilter" of the configuration word) and t ifmap images (which is indicated by field 50_25 "Channel_I" of the configuration word), state 43 "LOADING filter" will be iterated k times and state 44 "LOADING ifmap" will be iterated t times;

[0282] - several images and several filters need to be processed, which can lead to iterations of sequences "state 43 followed by state 44 followed by state 45");

[0283] - if several ifmaps input images are to be processed by the same filter during the layer processing (indicated by the 50_23 "numFilter" field of the configuration word and by the 50_12 "layer" field defining the layer type): several 44 "LOADING ifmap" states will be performed, with calculations (state 45) following each load, without having to reload the filters (which will result in iterations of the "state 44 followed by state 45" sequence); or

[0284] - if several filters are to be applied to the same input ifmap image: several states 43 "LOADING filter" (number defined by the content of field 50_23) will be to be carried out with, following each load, the implementation of the calculations (state 45), without having to reload the ifmap data (which will give rise to several iterations of the sequence "state 43 followed by state 45" without going through state 44).

[0285] In a node 14, the computing block 51, including each MAC operator 57 it contains, is driven by the FSM 40 of the global control block 10. It is adapted to remain in the inactive state until the global control block 10 triggers execution (by sending the "start execution" control signal), and then to proceed accordingly based on the control signals sent by the global control block 10: - when the load_filter signal is received by a compute node 14, the compute block 51 of node 14 stores the incoming payload in the filter register (RF 511); once all compute nodes 14 have received their corresponding filter data (i.e., once the compute node 14 in the top right of grid 11 has received it), the controller 58 of this compute node 14 in the top right of grid 11 (each node knows its position in grid 11) then sends a signal to the global control block 10 so that the FSM 40 moves to the next state (which corresponds to the transition rule r34); in one embodiment, the transition to state 44 "LOADING ifmap" of FSM 40 (which corresponds to the transition rule r34) takes place one cycle after the loading state of the filters, which ensures the simultaneous propagation of two types of data in grid 11 and ensures communication / communication overlap in order to reduce execution time;- when the load_ifmap signal is received by a computing node 14, the computing block 51 of node 14 stores the incoming payload in the RF ifmaps 510.Once all nodes 14 have received their corresponding ifmaps data, the compute node 14 sends a signal to the global control block 10 to move to the next state of its FSM 40 (which corresponds to the transition rule r45); in one embodiment, the transition to the WAIT state of the FSM 40 (which corresponds to the transition rule r45) takes place one cycle after the ifmap load state: this cycle ensures that the ifmap data has been sent to the first column (on the left) and the first row (in one embodiment: the bottom [the first layer of the DNN] or the top [the deep layers of the DNN]) of the grid 11; therefore, a computation trigger will be made in parallel with the propagation of the ifmaps data in the network, which ensures communication / computation overlap in order to reduce execution time; . - The computation block 51 of node 14 begins the MAC operation on its filter data and ifmaps data as soon as this data is present in RF 511, 510 and generates the PSum. Then, it either sends the PSum to the North node (if it is a node in the bottom row), or (ii) adds the incoming PSum from the South node to the local PSum and sends the result to the North node or to the associated OGB 13_2 (if it is a node in the top row). In this phase, the communication-computation overlap is also performed by the NPE, which has received the required data. At the end of the computation, an end_computation signal will be sent by the last node 14 (the last node, in this case, is in the top right corner of the active grid) to inform the control block 10 of the end of the execution, in order to proceed to the next execution and the loading of new data.

[0286] In one embodiment, at the end of or in parallel with step 102_1, in a substep 102_2, in state 42 “LOADING configuration”, the global control block 10 further transmits the configuration word 50 (or at least some of the information contained in this configuration word) of the current C_i layer to be processed, for example in parallel with the “start execution” signal, to the compute nodes 14 and / or the memory controller 12, and upon receipt of this configuration word, a compute node 10 and / or the memory controller 12 configures its processing (for example by configuring a local FSM used to orchestrate its part of the execution of the C_i layer) according to this configuration word CONFIG_i.

[0287] The computing node 14 includes, for example, a PE controller 58 comprising a state machine (FSM) adapted to, following receipt of the configuration word CONFIG_i sent by the global control block 10, synchronize with the global control block 10 and configure this FSM according to the configuration word.

[0288] This provision allows, for example, fixing how many operations are to be performed locally at the node for the C_i layer.

[0289] The FSM of the PE 58 controller is for example composed of 4 states: "Idle", "Mul", "acc", "add". The transition from "idle" to "mul" is triggered by the reception of the 'start execution' signal (from the global control block 10), the availability of data in RF 511,510 (filter and ifmap), and in embodiments depending in addition on other signals configured by the configuration word in particular by fields 50_12, 50_13 and 50_14.

[0290] The transition from "mul" to "acc" is systematically triggered in the following cycle.

[0291] The transition from "acc" to "add" is triggered when a dedicated counter reaches its maximum value which is the height of the filter minus 1 (the height information is extracted from field 50_13 of the configuration word).

[0292] The "add" state loops back on itself while waiting for the arrival of the partial sum PSum from the neighboring PE to the south. Once this partial sum arrives, it performs the required addition and creates the control word associated with this updated partial sum data and sends it to the neighboring PE to the north. The system then transitions to the "mul" or "idle" state depending on the conditions below: - the looping of the "add" state to "mul" is a function of the current value of a dedicated counter whose maximum value is that of the size of a line ofmap (the formula is given above and uses the values ​​of the fields Filter_H 50_13, Ifmap_H 50_14, stride 50_17 and padding 50_l8 of the configuration word 50: cf. [Fig.6]); - the transition from "add" to "idle" is triggered when the counter reaches the max value of the map; at this moment, the PE 50 sends the "End_computation" signal.

[0293] These provisions thus allow in nodes 14, the PE 58 controllers to configure their FSM to trigger MAC calculations upon receipt of filter data by reusing previously loaded ifmaps data already used for a previous calculation, for example if several filters are to be applied to the same ifmap data when processing a given layer (or vice versa to trigger MAC calculations upon receipt of ifmaps data by reusing previously loaded filter data already used for a previous calculation, for example if the same filter is to be applied to several ifmaps).

[0294] The layer type determines whether for a new calculation, a PE 58 expects new ifmap / filter data, or alternatively retains the data already loaded for the previous calculation, as well as the contents of some of the configuration word fields, such as field 50_23 "numFilter", field 50_25 "Channel_I", field 50_12 "layer".

[0295] In one embodiment, the PE 50 of a node 14 is parametric: it is dynamically parameterized for the execution of the layer to be processed, it thus includes a MAC 57 operator (for example to process a convolution layer dependent on an RS operating mode), or several MAC 57 operators (for example to process several convolutions in parallel in the same node 14 or to manage two decision branches in a dynamic DNN).

[0296] For example, if the DNN is to be run in static Row Stationary (RS) mode with 3x3 size filters, a single MAC 57 operator is generated to run three successive MACs over three cycles, or three MAC 57 operators are generated for parallel execution of these in a single cycle.

[0297] In the case of execution on a dynamic DNN where there is a second branch for decision-making in parallel with the initial branch of the network In a classic DNN, generating two MAC 57 operators allows processing both branches simultaneously and achieving the fastest possible decision-making without compromising computation. Alternatively, four additional MAC 57 operators can be generated to provide three functional operators for each branch, adding a second level of parallelism.

[0298] In the case of several MAC operators 57 configured in a PE 50, they are all interfaced with the same router 52 and the same clock controller 53 of the PE 50.

[0299] In one embodiment, the accelerator is implemented on an FPGA comprising dynamically programmable logic elements (logic gates). When the PE 58 controller receives the configuration word from the global control block 10, it configures the PE 50 processing element so that the latter has a number of calculation blocks (and therefore MAC operators) equal to the number indicated in the 50_16 "NumMac" field by programming the FPGA's logic gates according to the 50_16 "NumMac" field. Thus, unlike the clock gating mechanism, this is not a case of putting unnecessary calculation operators to sleep, but rather a redefinition of the circuits.

[0300] In one embodiment, a data sending mechanism, from a local memory of the set 13 of buffers to a computing node or vice versa, defines the recipient(s) of the data (appearing in the data packet discussed in [Fig. 8]) according to the current and / or following configuration word. The input data of the first layer and the data of all filters in the given DNN are stored beforehand in the buffers 13_1 (IGB) according to the form given in [Fig. 8]. This is done via the memory controller 12. The filter data are associated with control words ensuring horizontal multicast, and the input data are associated with control words ensuring diagonal multicast.

[0301] The output data from the calculation of each layer will become the inputs of the next layer. In this case, in one embodiment, the packaging module 16 generates the control words to be associated with this data according to the type of layer to be processed subsequently.

[0302] The provisions set out in this chapter make it possible to obtain a compact architecture on the one hand and on the other hand easily adaptable and configurable, thus allowing the execution of different types of DNN layers and neural networks.

[0303] Indeed, while it is known until now that there are 2 types of AI coprocessor:

[0304] - either a fully wired architecture that always executes the same thing (row stationary or other), the data is always sent in the same way, resulting in a compact, efficient but non-reconfigurable architecture;

[0305] - either a programmable SIMD or MIMD architecture - each node must do the process (fetching data - decoding instructions - executing instructions), which gives a reconfigurable architecture, but not compact (a processor is needed in each node) and power-intensive.

[0306] The proposed solution, through the use of "simple FSM" (instead of processor) and a lightweight FSM parameterization mechanism by configuration word, allows for a relatively compact and reconfigurable architecture to be able to implement any type of neural network.

[0307] Circuit architecture with data packaging

[0308] As seen previously, the data circulating in accelerator 1 is transferred by the router of each node according to a control word associated with each piece of data (cf. [Fig.8]).

[0309] In one embodiment, the packing block 16 is adapted to, between the processing implemented by the grid 11 to process a layer, C_i, i = 1 to Nl, of the neural network and the processing implemented by the grid 11 to process the next layer, C_i+1, determine, as a function of the configuration word CONFIG_i+l, which is provided to it by the global control block 10, the control word associated with at least some of the input data of the layer C_i+1 when they also come from the output data of the grid 11 after its processing relative to the layer C_i, (i.e. the ifmap image data of the layer C_i+1, which are the psum accumulation results of the layer C_i.

[0310] Thus, in one embodiment, the packing block 16 determines, according to the type of layer indicated in the "layer" field 50_12 of the configuration word for layer C_i+1 and a lookup table stored for example in the packing block 16 (or which is accessible to it) the type of broadcast (unicast, broadcast, multicast), as well as the recipient (unicast) or the direction of the broadcast (multicast) among the following directions: E or W (if horizontal multicast), S or N (if vertical multicast), NE or NW, or SE, or SW (if diagonal multicast): this information conditions the structure of the control words considered.

[0311] For example, the lookup table indicates:

[0312] - a diagonal SO multicast communication if the layer is a conv layer hidden;

[0313] - unicast communication to all nodes of grid 11 if the layer is a fully connected layer (fc);

[0314] - unicast communication to all nodes of grid 11 if the layer is a pointwise layer;

[0315] - a vertical SO multicast communication if the layer is a depthwise layer.

[0316] There are several possible implementations of this packaging mechanism.

[0317] For example, in one embodiment, the following treatment process 200 is implemented by the accelerator 1, with reference to [Fig. 17].

[0318] In a step 201, an output data ofmap resulting from the accumulations carried out during the processing by grid 11 of layer C_i is delivered to the packing block 16 (optionally, this data is first processed by the activation block 17, then the resulting data is provided to the packing block 16) by a node 14 charged with the last accumulation carried out before providing the result of this accumulation from the grid, for example, in the examples described above the node N_0i (i= 0 to 4) of the first row of grid 11 (although in reality, the accumulations can alternatively by reconfiguration be carried out, vertically but towards the south, or horizontally towards the E or the W; for example via a static reconfiguration based on a generic parameter or via a dynamic reconfiguration, based on the exploitation of a dedicated field in the configuration word and indicating the direction of the accumulation).

[0319] In step 202, as soon as the packing block 16 has received NPACK data ofmaps provided by node N_0i (NPACK is a configurable integer greater than or equal to 1; in the example shown in [Fig. 11], NPACK is strictly greater than 1, for example it was chosen to be 3), it inserts them into a payload data packet 160 and generates the control word - common - associated with this ofmap data, according to the lookup table and according to (as indicated above) the configuration word CONFIG_i+1 of the future layer, C_i+1, of the network to be executed. Then the resulting double data packet (the control packet containing the control word and the associated payload data packet, see [Fig. 8]) is provided to the OGB buffer _i, associated with node N_0i, to be stored there (if the activation has not been performed upstream of the packing block, it (can be performed if necessary before storage in the buffer).

[0320] The payload data packets (or words), also called payload packets, are created based on the information in field 50_14 (I fmap_H) of the configuration word for the future layer. For example, in a case where NPACk = 3: - if field 50_14 indicates 6, the following "split" is made: two useful data words, each containing 3 ifmap data; - if the field 50_14 indicates 5, two useful data words are formed, the first containing 3 ifmap data and the second containing 2 ifmap data padded with zeros (zero padding).

[0321] The number of control words to generate depends on the values ​​of fields 50_14 and 50_25: it is equal to (Ceil (Ifmap_W / Npack)*Ifmap_H) * Channel_I, where Ceil (x) is a mathematical function that returns the smallest integer greater than or equal to x. In the case of a regular matrix considered here as an illustration: Ifmap_W= Ifmap_H, which corresponds to the field 50_14.

[0322] The choice of 3 for NPACK relates to the size of the data carried in grid 11, which in this case is 32 bits. Given that the operators in this case manipulate 8-bit data (the results are also 8 bits), a 32-bit packet can contain a 4-bit header flit plus a maximum of 3 8-bit words. If operators manipulating 6-bit, 4-bit, or other data sizes are used, more than 3 words will be inserted into the same packet.

[0323] In step 203, the execution of the neural network layer C_i+1 is initiated by the global control block 10; it is determined, if necessary, based on the configuration word by the control block 10 that the ifmaps input data of layer C_i+1 are those then stored in the OGB buffers _i (for example, a mechanism in the global controller 10 increments a counter at each newly encountered configuration word and when the value is strictly greater than 1, infers that the configuration word is that of a deep layer and therefore that the input data are already stored in the OGBs). The control block 10 thus triggers the provisioning of the packets stored in the OGB memory _i to the grid's input node, which is associated with this memory in the grid 11, i.e., here, node N_0i.

[0324] As indicated, node N_0i therefore begins the dissemination of the received ifmap data according to the communication type (unicast, horizontal multicast, vertical multicast, diagonal multicast, or broadcast) indicated in the control word associated with the input data. Data dissemination can then be carried out from node to node, based on the associated control word.

[0325] Grouping data into a data packet NPACKdonnées ofmaps allows the buffer to be loaded (same for the removal of memory to the grid during the execution of the next layer C_i+1) only once rather than NPACk times.

[0326] In another embodiment, the NPACK data ofmaps are stored in a data packet in the OGB_i memory, but this time without a packing operation by the packing block 16. And when the execution of the neural network layer C_i+1 is launched by the global control block 10, it is determined according to the configuration word by the control block 10 that the input data ifmaps of the layer C_i+1 are those then stored in the OGB_i buffers, and the control block 10 then triggers the provisioning of the packets stored in the OGB_i memory to the packing block 16.

[0327] In this other embodiment, the packing block 16, once it has received the packet of NPAcKdonnées ofmaps foumi(s), determines, as indicated in step 202, the common control word associated with this ofmaps data, according to the lookup table and according to (as indicated above) the CONFIG_i+1 configuration word of layer C_i+1 whose execution has just been triggered. Then the resulting double data packet, consisting of the useful data packet and the associated control word packet, is provided to the input node (corresponding to memory OGB_i) in grid 11, i.e. here node N_0i (the control word is also a function of the execution mode chosen when there are several possible ones, for example WS, IS, OS).

[0328] And so, for each layer C_ i+1, the type and direction of communication of the input data ifmap of the layer are defined according to the type of layer indicated in the configuration word, according to the storage memory 13 (therefore the grid node, which is associated with this storage memory and to which the data extracted from this memory will be provided at the beginning of the execution of layer C_i+1) and according to the output data ofmaps of the previous layer in the grid 11 of the accelerator 1.

[0329] The packaging process in the two embodiments described in the section above, using a local packaging block 16, located between the nodes 14 and the local memory 13, autonomous in its packaging operation (independent of the global control block 10) simplifies data communication operations, reduces access to memories external to the accelerator 1 (for example the DRAM 19) and ultimately allows greater parallelization of operations.

[0330] It will be noted that all the nodes 14 are constituted in the same way. The input nodes of the grid (i.e. that receive the ifmaps or filter data from a current layer to be executed) or the output nodes of the grid (i.e. that deliver the accumulation results outside the grid 11) know, before a layer execution, that they are respectively input or output nodes (for example, the nodes are told that the nodes of such and such a row or such and such a column are concerned) of the grid, and they then execute the specific processing functionalities of these input or output nodes (typically, receiving the ifmaps of the buffer data or providing the accumulation data to a buffer, or to an activation block or a packaging block).

[0331] The following are described cases implementing an accelerator in embodiments of the invention:

[0332] Fig. 12 gives an overview of different types of data movement in accelerator 1 (left section of Fig. 12) for the first layer of a DNN considered already discussed above to illustrate the communication / calculation), right-hand section for the hidden layers of the DNN). The right-hand section of [Fig. 12] illustrates the processing of a convolution-type hidden layer in one embodiment of the invention, where the ifmaps are sent diagonally from the OGB_i (Inrowi+i coming from OGB_i, i = 0 to 2) (by application of method 200 described above), the filters are sent horizontally (Frowi+i coming from IGB_i, i = 1 to 3), and the PSum accumulation is performed by accumulating the results from each node in a column by moving them vertically. Finally, the nodes in the top row store the ofmap data respectively in the OGBs of the memory set 13_2.

[0333] Considering now [Fig. 13], which illustrates the execution of a first convolution layer with a stride value of 1, an ifmap size of 7x5 [row*column], a filter size of 3x3, an ofmap size of 5x3, and a grid size of 3x5, there are 7 IGBs and 5 OGBs used in this example to explain the execution pattern. Seven input ifmap rows, rl, r2, ..., r7, are sent via the IGBs using Northeast diagonal multicast to grid 11 from nodes 14, and three filter rows are sent to grid 11 using vertical multicast. Once all the required data is available in a column of nodes, the nodes in that particular column begin the computation while the other nodes continue to receive their data. In this way, an overlap between communication calculations is established.Once a node is ready with a PSum, the data is sent to its North node using vertical unicast. On the receiving side, the node waits for the data to arrive from the South node (an exception is made for nodes in the bottom row that have no nodes connected to their South); once the data is available at the node's input data pin, the local PSum is added to the incoming data and sent to the North node. This computational chain is executed until the last PSum ofmap is accumulated and stored in the OGB (line Ri+1 from Ofmaps to OGB_i from N_0i, i = 0 to 4). This process is repeated until all the ifmaps and filters of that particular convolution layer have been used to compute the required ofmaps.

[0334] Figure 14 illustrates the execution of a hidden layer of the convolution or depthwise type: the ofmaps of the previous layer are initially already stored in the OGBs (for example, following the processing of a first layer as illustrated in Figure 13); these data become the ifmaps of the current layer (process 200 has been applied, for example). Here, the ifmaps are sent diagonally southwest from the OGBs (line Ri+1 of Ofmaps from OGB_i to N_0i, i = 0 to 4). The filters are sent horizontally from the IGBs. This is followed by the PSum accumulation phase, and finally the Ofmap data are stored respectively in their OGBs. Here, the ifmap data is sent from the 0GBs to grid 11 using diagonal multicast, and the filter data is sent to grid 11 using horizontal multicast. Once all the required data is available in a column of nodes, the nodes in that particular column begin the computation while the other nodes continue to receive their data. Once a node is ready with a PSum, this data is sent to its vertical node using vertical unicast. On the receiving side, the node waits for the data to arrive from the South node (an exception is made for nodes in the bottom row that have no nodes connected to their South). Once the data is available at the node's input, the local PSum is added to the incoming data and sent to the North node. Such a computation chain is executed until the last ofmap PSum is accumulated and stored in the OGBs.This process is repeated until all the ifmaps and filters of this particular convolution layer are used to calculate the required ofmaps (Ofmap RU, R2', R3' lines).

[0335] Figure 15 illustrates the execution of a pointwise layer: the ifmaps are sent unicast from the OGB buffer set 13_2, and the filter data are sent horizontally from the IGB 13_1. This is followed by the PSum accumulation phase, and finally the ofmap data are stored in their respective OGBs. Here, 5 rows (rij, i = 1 to 5) of 3 different ifmaps (Ifmapj, j = 1 to 3) are sent from the OGBs to grid 11 using unicast or parallel unicast, and 3 copies of the filter data are sent to grid 11 using horizontal multicast. This data distribution is performed to utilize hardware resources. Once all the required data is available in a column of nodes, the nodes in that column begin computation while the other nodes continue to receive their data.Once a node is ready with a PSum, the data is sent to the northern node using vertical unicast. On the receiving side, the node waits for the data to arrive from the southern node (an exception is made for nodes in the bottom row that have no nodes connected to their south). Once the data is available at the node's input, the local PSum is added to the incoming data and sent to the northern node. This computational chain is executed until the last PSum ofmap is accumulated and stored in the OGB (data stored in 0GB_i, i = 0 to 4, yields row Ri+1 ofmap). This process is repeated until all the ifmaps and filters of that particular convolutional layer have been used to compute the required ofmaps.

[0336] Figure 16 shows the execution of a fully connected layer. Due to the CNN architecture, ofmaps of the layer preceding the fully connected layer take the form of a single column which is stored in the first OGB. Thus, the ifmaps for the fully connected layers are sent from the first OGB using unicast. The filter data, divided into 3 equal lines, is sent unicast. This is followed by the PSum accumulation phase, and finally the ofmap data is stored in its respective OGB.

[0337] Here, the ifmap data, divided into 3 rows (i.e., the number of rows in grid 11), is sent to the grid of nodes 11 using unicast, and the filter data is divided into 3 rows and sent to grid 11 using unicast. Once all the required data is available in a column of nodes, these nodes begin the calculation to compute the PSum. Once all nodes have their PSum ready, the accumulation of PSum in each row is performed in parallel by unicasting data to the west node. On the receiving side, the node waits for the data to arrive from the east node (an exception is added for nodes in the rightmost column that have no nodes connected to their east). Once the data is available at the node's input, the local PSum is added to the incoming data and sent to the west node.This computational chain is executed until the leftmost nodes have accumulated their PSum. Then, the data is sent to their vertical node using vertical unicast. On the receiving side, the node waits for the data to arrive from the South node (an exception is made for nodes in the bottom row that have no nodes connected to their South). Once the data is available at the node's input, the local PSum is added to the incoming data and sent to the North node. The top-left node (i.e., PE_0_0) stores the accumulated value in OGB_0.

[0338] It should be noted that the "configuration (of FSMs in particular) according to the configuration word" aspect and the "data packaging" aspect can be implemented independently of each other and that in accelerator embodiments, only one of the two aspects is implemented.

[0339] The invention thus proposes a parameterizable data flow execution solution, enabling flexibility of processing and reduction of energy consumption in specialized architectures of convolutional neural networks (CNNs) for inference.

[0340] The invention has been described above in an implementation on NoC. Other types of Dataflow architectures may nevertheless be used.

[0341] In the example considered, the set 13 of buffer memories comprises buffer memories 13_1 for storing the ifmaps data and filter data, and buffer memories 13_2 for storing the results (ofmaps) of the first layer, delivered by the nodes of the first row of the grid after accumulation of the psum results of the nodes by column, which are also, in some embodiments, ifmaps data of hidden layers. Other memory configurations Buffers can be used, for example, with more or fewer buffers: for instance, the same buffers can first store the ifmaps and filter data, then the accumulation results, with appropriate flow orchestration and varying processing times. At least one row of local memory is provided, with each row of local memory positioned on one side of the compute node matrix and each local memory connected to a router of a compute node.

Claims

1. Demands Processing method for running a neural network comprising at least N successive neural network layers C_i, i = 1 to N, with N>1, in a neural network accelerator (1); the neural network accelerator (1) comprising: - a global control block (10), - a matrix (11) of computation nodes (14), each node (14) comprising a router (52) and a processing element (50), also called PE, associated with a set of respective local memory registers (510, 511), the processing element (50), allowing to perform computation operations among multiplications and accumulations of data stored in its registers, the router (52) allowing to perform several independent data routings in parallel to distinct outputs of the router, the routers being connected to each other; - at least one row of local buffers (13), each row of local buffers being positioned on one side of the matrix (11) of compute nodes, and each local buffer is connected to a router of a compute node associated with it; the routers (52) being adapted to route the data, through the matrix of computing nodes in parallel with the execution by the computing nodes, according to a control word associated with each data; according to which at the beginning of the execution by the accelerator (1) of a C_i layer, ifmap input data of the layer are extracted from at least some of said local buffers and are provided, with their associated control word, to the routers of the associated compute nodes; according to which, during the execution of the C_i layer by the accelerator, ofmap output data from the C_i layer, derived from data delivered by the compute node routers associated with local buffers, are stored in said local buffers; said method being characterized in that the accelerator (1) further comprises a packing block (16) disposed between at least some of the local buffer memories and the associated nodes said local buffers, and said packing block (16) determines the control word associated with the ifmap input data of layer C_i+1 according to a configuration word defined for said layer C_i+1, when implementing either of the following provisions i and j in the accelerator: provision i / it / during the execution of layer C_i by the accelerator: - before the said output data ofmap of layer C_i is stored in the said local buffer memories (13), the said packing block (16) determines, according to the configuration word associated with layer C_i+1, the control words, for each of the input data ifmap of layer C_i+1 which correspond to the output data ofmap of layer C_i; - then said ofmap output data are inserted into data packets stored in said local buffers (13) with the associated control words thus determined for layer C_i+1 by the packing block (16); i2 / at the beginning of the execution of layer C_i+1 by the accelerator (1), said ifmap input data of the layer are extracted from the local buffers (13) where they were stored since step il and are provided, with their associated control word for layer C_i+1, to the routers (52) of the computing nodes (14) to which said local buffers (13) are associated; arrangement j / j 1 / at the beginning of the execution of layer C_i+1 by the accelerator, the output data ofmap of layer C_i having been stored in said local buffers (13) at the end of the execution of layer C_i, said packing block (16) determines according to the configuration word associated with layer C_i+1, the control words, for layer C_i+1, which are associated with the input data ifmap of layer C_i+1, which correspond to the output data ofmap; j2 / then the said ifmap input data of the layer extracted from the local buffers (13) where they were stored since the end of the execution of layer C_i are inserted into packets with their associated control word determined for layer C_i+1, and are thus provided to the routers (52) of the computing nodes (14) to which the said local buffers (13) are associated.

2. Processing method for running a neural network according to claim 1, wherein the determination of the control word for a data for layer C_i+1 is carried out as a function of at least one field of the configuration word indicating the type of layer C_i+1 and the local buffer (13) storing the ofmap data corresponding to the ifmap data of layer C_i+1.

3. Processing method for running a neural network according to claim 2, wherein the determination of the control word for a data point for layer C_i+1 is carried out as a function of at least one element among the position of said local buffer (13) and the position of the node (14) associated with said local buffer.

4. A processing method according to any one of the preceding claims, wherein the determined control word indicates a type of communication among unicast, broadcast, diagonal multicast, vertical multicast, horizontal multicast and further indicates a direction if the type of communication is multicast.

5. Processing method according to any one of the preceding claims, wherein several of said data associated with the same control word and delivered by the same computing node (14) are inserted into a common data packet associated with said control word determined by the packing block (16).

6. A neural network accelerator (1) adapted for running a neural network comprising at least N successive neural network layers C_i, i = 1 to N, with N > 1, the neural network accelerator (1) comprising: - a global control block (10), - a matrix (11) of compute nodes (14), each node (14) comprising a router (52) and a processing element (50), also called a PE, associated with a set of respective local memory registers (510, 511), the processing element (50) being adapted to perform computational operations among multiplications and accumulations of data stored in its registers, the router (52) being adapted to perform several independent data routings in parallel to distinct outputs of the router, the routers being interconnected; - at least one row of local buffers (13), each row of local buffers being positioned on one side of the matrix (11) of compute nodes, and each local buffer is linked to a router of a compute node associated with it; the routers (52) being adapted to route the data, through the matrix of compute nodes in parallel with execution by the compute nodes, according to a control word associated with each data; said accelerator being adapted so that, at the beginning of the execution by the accelerator (1) of a C_i layer, ifmap input data of the layer are extracted from at least some of said local buffers and are provided, with their associated control word, to the routers of the associated compute nodes and so that during the execution of the C_i layer by the accelerator, ofmap output data from the C_i layer, derived from data delivered by the routers of compute nodes associated with local buffers, are stored in said local buffers; said accelerator being characterized in that it further comprises a packing block (16) disposed between at least some of the local buffers and the nodes associated with said local buffers, and said packing block (16) is adapted to determine the control word associated with the ifmap input data of layer C_i+1 as a function of a configuration word defined for said layer C_i+1, during the implementation of either of the following dispositions i and j in the accelerator: disposition i / during the execution of layer C_i by the accelerator: - before the said output data ofmap of layer C_i is stored in the said local buffers (13), the said packing block (16) is adapted to determine, according to the configuration word associated with layer C_i+1, the control words, for each of the input data ifmap of layer C_i+1 which correspond to the output data ofmap of layer C_i; - the accelerator being adapted for said output data ofmap are then inserted into data packets stored in said local buffer memories (13) with the associated control words thus determined for layer C_i+1 by the packing block (16); i2 / at the beginning of the execution of layer C_i+1 by the accelerator (1), said accelerator is adapted so that said ifmap input data of the layer are extracted from the local buffers (13) where they were stored since step il and are provided, with their associated control word for layer C_i+1, to the routers (52) of the compute nodes (14) to which said local buffers (13) are associated; arrangement j / j 1 / at the beginning of the execution of layer C_i+1 by the accelerator, the output data ofmap of layer C_i having been stored in said local buffers (13) at the end of the execution of layer C_i, said packing block (16) is adapted to determine as a function of the configuration word associated with layer C_i+1, the control words, for layer C_i+1, which are associated with the input data ifmap of layer C_i+1, which correspond to the output data ofmap;j2 / the accelerator is adapted so that the said ifmap input data of the layer extracted from the local buffers (13) where they were stored since the end of the execution of layer C_i are then inserted into packets with their associated control word determined for layer C_i+1, and are thus provided to the routers (52) of the computing nodes (14) to which the said local buffers (13) are associated.;

7. Accelerator (1) according to claim 6, wherein the packing block (16) is adapted to determine the control word for data for layer C_i+1 as a function of at least one field of the configuration word indicating the type of layer C_i+1 and the local buffer (13) storing the ofmap data corresponding to the ifmap data of layer C_i+1.

8. Accelerator (1) according to claim 7, wherein the determination of the control word for a data for layer C_i+1 is carried out by the packing block (16) as a function of at least one element among the position of said local buffer (13) and the position of the node (14) associated with said local buffer.

9. Accelerator (1) according to any one of claims 6 to 8, wherein the determined control word indicates a type of communication among unicast, broadcast, multicast diagonal,

10. vertical multicast, horizontal multicast and further indicates a direction if the type of communication is multicast. Accelerator (1) according to any one of claims 6 to 9, wherein several of said data associated with the same control word and delivered by the same computing node (14) are inserted into a common data packet associated with said control word determined by the packing block (16).

Images