PROCESSING METHOD IN A NEURAL FOLDING NETWORK ACCELERATOR AND ACCELERATOR FOR IT

Description

Domaine technique :

[0001] The invention lies in the field of artificial intelligence and deep neural networks, and more particularly in the field of accelerating inference computation by convolutional neural networks. Technique antérieure :

[0002] Artificial intelligence (AI) algorithms are currently a vast field of research, as they are destined to become essential components of next-generation applications. These applications will rely on intelligent processes to make decisions based on knowledge of their environment, such as detecting objects like pedestrians in a self-driving car or recognizing activity in a health-tracking smartwatch. This knowledge is gathered by sensors combined with highly efficient detection and / or recognition algorithms.

[0003] In particular, deep neural networks (DNNs), and especially convolutional neural networks (CNNs – see, for example, Y. LeCun et al. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278-2324), are good candidates for integration into such systems due to their excellent performance in detection and recognition tasks. They are based on filter layers that perform feature extraction and then classification. These operations require significant computation and memory, and integrating such algorithms into systems necessitates the use of accelerators. These accelerators are electronic devices that primarily perform parallel computation of the multiply-accumulate (MAC) operations that are numerous in CNN algorithms.The goal of these accelerators is to improve the execution performance of CNN algorithms to meet application constraints and enhance the system's energy efficiency. They are primarily based on a large number of processing elements involving operators optimized for MAC execution and a memory hierarchy enabling efficient data storage.

[0004] Most hardware accelerators are based on a network of Processing Elements (PEs) implementing MACs and use local buffers to store frequently reused data, such as filter parameters or intermediate data. Communication between the PEs themselves and between the PEs and memory is a crucial aspect to consider when designing a CNN accelerator. Indeed, CNN algorithms exhibit high inherent parallelism and data reuse capabilities. Therefore, the on-chip communication infrastructure must be carefully designed to leverage the large number of PEs and the specific characteristics of CNN algorithms, which improve both performance and energy efficiency.For example, multicasting or broadcasting particular data in the communication network will allow target PEs to simultaneously process different data with the same filter using a single memory read.

[0005] XiaoShanlin ET AL: "NeuronLink: An Efficient Chip-to-Chip Interconnect for Large-Scale Neural Network Accelerators", IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS., vol. 28, no. 9, pages 1966-1978, describes an intra-chip and inter-chip communication coupling for neural network accelerators.

[0006] Several factors have contributed to limiting or complicating the scalability and flexibility of existing CNN accelerators on the market. These factors manifest themselves through: (i) limited bandwidth due to the lack of an efficient broadcast medium, (ii) excessive energy consumption due to memory size (e.g., 40% of energy consumption in some architectures is induced by memory) and the memory capacity wall problem, and (iii) limited data reuse and the need for efficient support for processing different communication patterns.

[0007] Therefore, there is a need to increase the processing efficiency in the neural accelerators of CNN architectures, taking into account the high number of PEs and the particularities of CNN algorithms. Summary of the invention:

[0008] To this end, according to a first aspect, the present invention describes a processing method in a convolutional neural network accelerator comprising a grid of unit processing blocks, each unit processing block comprising a unit computation element PE associated with a set of respective local memories and performing computational operations among multiplications and accumulations of data stored in its local memories, according to claim 1.

[0009] During respective processing cycles timed by an accelerator clock, unit blocks of the grid receive and / or transmit data from or to neighboring unit blocks in the grid in at least one selected direction, depending on said data, from at least the vertical and horizontal directions in the grid; during said same cycles, unit blocks of the grid perform one of said computational operations relating to data stored in their set of local memories during at least one previous processing cycle.

[0010] Such a process makes it possible to guarantee flexible processing and reduce energy consumption in CNN architectures including an accelerator.

[0011] It offers a DataFlow execution model that distributes, gathers and updates among the many distributed processing elements (PEs), operands and allows for different degrees of parallelism on the different types of shared data (weights, Ifmaps and Psums) in CNNs, reduces the cost of data exchange without degrading performance and finally facilitates the processing of different CNN networks and different layers of the same network (Conv2D, FC, PW, DW, Residual, etc.).

[0012] In some embodiments, such a process will further include at least one of the following features: at least during one of said processing cycles: at least one unit block of the grid receives data from several neighboring unit blocks in the grid located in distinct directions from said unit block; and / or at least one unit block of the grid transmits data to several neighboring unit blocks in the grid in distinct directions; a unit block performs a transmission of a type selected between broadcast and multicast according to a header of the packet to be transmitted and according to which the unit block applies at least one of said rules: for a packet to be transmitted as broadcast originating from a neighboring block located in a given direction from said block to perform the transmission, said block transmits the packet during one cycle in all directions except that of said neighboring block;for a packet to be transmitted via multicast: if the packet comes from the PE of the unit block, the multicast implemented by the block is bidirectional; otherwise, the multicast implemented by the block is unidirectional, directed in the opposite direction from the neighboring processing block from which said packet originates; The reception and / or transmission of data implemented by a unit processing block is implemented by a routing block included in said unit block, implementing parallel data routing functions during the same processing cycle, according to communication directions associated with the data; in the event of at least two simultaneous transmission requests to the same direction by a unit block during a processing cycle, the priority between said requests is arbitrated, the priority arbitrated request is transmitted in said direction and the other request is memorized, then transmitted to said direction in a later processing cycle.

[0013] According to another aspect, the invention describes a convolutional neural accelerator comprising a grid of unit processing blocks and a clock, each unit processing block comprising a unit computation element PE associated with a set of respective local memories and adapted to perform computational operations among multiplications and accumulations of data stored in its local memories, according to claim 6.

[0014] In this accelerator, unit blocks of the grid are adapted to, during respective processing cycles timed by the accelerator clock, receive and / or transmit data from or to neighboring unit blocks in the grid in at least one direction selected, according to said data, from among at least the vertical and horizontal directions in the grid; and unit blocks of the grid are adapted to, during said same cycles, perform one of said computational operations relating to data stored in their set of local memories during at least one previous processing cycle.

[0015] In some embodiments, such an accelerator will further include at least one of the following features: at least during one of said processing cycles: at least one unit block of the grid is adapted to receive data from several neighboring unit blocks in the grid located in distinct directions with respect to said unit block; and / or at least one unit block of the grid is adapted to transmit data to several neighboring unit blocks in the grid in distinct directions; a unit block is adapted to perform a transmission of a type selected between broadcast and multicast based on a header of the packet to be transmitted and the unit block is adapted to apply at least one of said rules: for a packet to be transmitted as broadcast originating from a neighboring block located in a given direction with respect to said block to perform the transmission, said block transmits the packet during one cycle in all directions except that of said neighboring block;for a packet to be transmitted in multicast: if the packet comes from the PE of the unit block, the multicast implemented by the block is bidirectional; otherwise, the multicast implemented by the block is unidirectional, directed in the opposite direction to the neighboring processing block from which said packet originates; a unit block includes a routing block adapted to implement said data receptions and / or transmissions by the unit block, said routing block being adapted to implement parallel data routing functions during the same processing cycle, according to communication directions associated with the data;In the event of at least two simultaneous transmission requests to the same direction by a unit block during a processing cycle, the routing block of the unit block is adapted to arbitrate the priority between said requests, the priority arbitrated request then being transmitted in said direction and the other request being memorized, then transmitted to said direction in a subsequent processing cycle. Brief description of the figures:

[0016] The invention will be better understood and other features, details and advantages will become clearer from the following description, given by way of non-limiting example, and from the accompanying figures, given by way of example. [ Fig. 1 ] There figure 1 represents a neural network accelerator in one embodiment of the invention; [ Fig. 2 ] There figure 2 represents a unit processing block in one embodiment of the invention; [ Fig. 3 ] There figure 3 represents a method in an embodiment of the invention; [ Fig. 4 ] There figure 4 represents the communication packet structure in the accelerator in one embodiment; [ Fig. 5 ] There figure 5 represents a routing block in one embodiment of the invention; [ Fig. 6 ] There figure 6 diagrams the computing and communication control architecture in one embodiment of the invention; [ Fig. 7 ] There figure 7 illustrates a stage of convolution calculations; [ Fig. 8 ] There figure 8 illustrates another stage of convolution calculations; [ Fig. 9 ] There figure 9 represents another stage of convolution calculations; [ Fig. 10 ] There figure 10 illustrates step 101 of the process of the figure 3 ; Fig. 11 ] There figure 11 illustrates step 102 of the process of the figure 3 ; Fig. 12 ] There figure 12 illustrates step 103 of the process of the figure 3 .

[0017] Identical references may be used in different figures when they refer to identical or comparable elements. Description détaillée :

[0018] A CNN comprises different types of successive neural network layers, including convolutional layers, each layer being associated with a set of filters. A convolutional layer analyzes, by region, using each filter (for example: horizontal Sobel, vertical Sobel, etc., or any other filter considered, particularly one derived from training) from the set of filters, at least one data matrix, which is provided as input, called the Input Feature Map (also referred to as IN hereafter), and delivers at least one data matrix as output, called the Output Feature Map (also referred to as OUT hereafter), which allows only the desired elements to be retained according to the filter considered.

[0019] The IN matrix is ​​a matrix of n rows and n columns. A filter F is a matrix of p rows and p columns. The OUT matrix is ​​a matrix of m rows and m columns. In special cases, m = n - p + 1, given that the exact formula is: m = n − f + 2 p / s + 1 , Or m: ofmap (mxm) - size may not be regular n: ifmap (nxn) - size may not be regular f: filter (fxf) p: margin to 0 (0-padding) s: stride.

[0020] For example, p = 3 or 5 or 9 or 11.

[0021] The convolutions performed, for example, correspond to the following process: the filter matrix is ​​placed in the upper left corner of the IN matrix; a product of each pair of overlapping coefficients is calculated; the sum of all these products is then given, yielding the value of pixel (1,1) in the output matrix OUT. The filter matrix is ​​then shifted horizontally one position to the right (strad), and the process is repeated, providing the value of pixel (1,2) in the OUT matrix, and so on. Once the end of the row is reached, the filter is moved down one position vertically, the process is repeated starting again from the right, and so on, until the entire IN matrix has been traversed.

[0022] Convolution calculations are generally implemented by neural network computing units, also called artificial intelligence accelerators or NPUs (Neural Processing Units), comprising a network of PE processor elements.

[0023] In an example, a calculation performed classically in a convolution layer implemented by an accelerator is presented below.

[0024] Consider the filter F consisting of the following weights: [Table 1] f 1 f 2 f 3 f 4 f 5 f 6 f 7 f 8 f 9

[0025] Consider the following matrix IN: [Table 2] in 1 in 2 in 3 in 4 in 5 in 6 in 7 in 8 in 9 in 10 in 11 in 12 in 13 in 14 in 15 in 16 in 17 in 18 in 19 in 20 in 21 in 22 in 23 in 24 in 25

[0026] And we consider the following OUT matrix: [Table 3] out 1 out 2 out 3 out 4 out 5 out 6 out 7 out 8 out 9

[0027] The expression of each coefficient of the OUT matrix is ​​a weighted sum corresponding to an output of a neuron whose inputs would be in i and the weights applied to the inputs by the neuron would be fj and which would calculate the value of the coefficient.

[0028] Let us now consider a grid of computational unit elements pe, having as many rows as the filter F (p = 3 rows) and as many columns as the OUT matrix has rows (m = 3): [pei,j] i = 0 to 2 and j = 0 to 2. The following is an example of using the grid to compute the coefficients of the OUT matrix.

[0029] As depicted in figure 7 , to each coefficient of the (i+1)th row of the pe, we provide the (i+1)th row of the filter matrix, i=0 to 2. We then provide the grid of pe the matrix IN: thus the first row of IN is provided to the unit element of calculation pe00; the second row of IN to the coefficients pe10 and pe01, located on the same diagonal; the third row of IN is provided to the unit elements pe20, pe11 and pe02, located on the same diagonal; the fourth row of IN is provided to the elements pe21 and pe12 on the same diagonal, and the fifth row of IN is provided to pe22.

[0030] In an initial round of calculations also shown on the figure 7 A convolution (combination of multiplications and additions) is performed in each pe between the filter row it was given and the first p coefficients of the row of the matrix IN it was given, yielding a named partial sum (the shaded cells of the IN row are not used for the current calculation). Thus, pe00 calculates f1.in1 + f2.in2 + f3.in3, etc. Then, the three partial sums determined by the pes in the same column are summed progressively: the partial sum determined by pe2j is given to pe1j, which adds it to the partial sum it previously calculated; this new partial sum resulting from the accumulation is then in turn given by pe1j to pe0j, which adds it to the partial sum it had calculated, j = 0 to 2: the total thus obtained is equal to the first coefficient of the j+1st row of the matrix OUT.

[0031] In a second round of calculations represented in figure 8 A convolution is performed in each pe between the filter row it was given and the p = 3 coefficients, starting from the 2nd coefficient, of the row of the IN matrix it was given, yielding a partial sum. Thus pe00 calculates f1.in2 + f2.in3 + f3.in4 etc. Then the three partial sums determined by the pes of the same column are summed progressively as described above and the total thus obtained is equal to the second coefficient of the j+1st row of the OUT matrix.

[0032] In a third round of calculations represented in figure 9 A convolution is performed in each pe between the filter row it was given and the p = 3 coefficients, starting from the 3rd coefficient, of the row of the IN matrix it was given, yielding a named partial sum. Thus pe00 calculates f1.in3 + f2.in4 + f3.in5 etc. Then the three partial sums determined by the pes of the same column are summed progressively as described above and the total thus obtained is equal to the third coefficient of the j+1st row of the OUT matrix.

[0033] In the calculation process described here as an example, the i-th line of pes thus allows us to successively construct the i-th column of OUT, i = 1 to 3.

[0034] It follows from this example that the lines of manipulated data (filter weights, Input Feature Map data and partial sums) are spatially reused between unit processor elements: here for example, the same filter data is used by the pes of the same horizontal line and the same IN data is used by all pes of diagonal lines, while the partial sums are transferred vertically, then reused.

[0035] It is therefore important that the communication of this data and the calculations involved are carried out in an optimized manner in terms of transfer time and calculation of access to the main memory initially delivering this data, regardless of the dimensions of the input and output data or the calculations implemented.

[0036] To that end, with reference to the figure 1 , a CNN neural network accelerator 1 in an embodiment of the invention comprises a grid 2 of unit processing blocks 10, a global memory 3 and a control block 30.

[0037] The grid 2 of unit processing blocks 10 comprises unit processing blocks 10 arranged in a network, connected by horizontal and vertical communication links allowing the exchange of data packets between unit blocks, for example according to a matrix organization in N rows and M columns.

[0038] Accelerator 1, for example, has an architecture based on a NoC (Network on Chip).

[0039] In one embodiment, each processing block 10 includes, with reference to the figure 2 , a PE (Processing Element) processor 11 adapted to perform calculation operations, of the MAC type in particular, a memory set 13, including for example several registers, intended to store filter data, input data from Input Feature Map received by the processing block 10 and the results (partial sums, accumulations of partial sums) calculated by the PE 11 in particular, and a router 12 adapted to route incoming or outgoing data communications.

[0040] A unit processing block 10 (and similarly its PE) is identified by its row and column position in the grid, as shown in figures 1 , 10 , 11 And 12 . Thus the processing block 10 (i,j), including the PE ij 11, is located on the i+1st row and j+1st column of grid 2, i = 0 to 3 and j = 0 to 3.

[0041] Each processing block 10 not located at the edge of the network thus has 8 neighboring processing blocks 10, in the following directions: one to the North (N), one to the South (S), one to the West (W), one to the East (E), one to the Northeast, one to the Northwest, one to the Southeast, and one to the Southwest.

[0042] The control block 30 is adapted to synchronize the calculation processes in the PE and data transfer between unit blocks 10 or within unit blocks 10 and implemented in the accelerator 1. All these processes are timed by a clock of the accelerator 1.

[0043] There will have been a preliminary step of configuring grid 2 to select the set of PEs to use, from among the PEs available in the maximum hardware architecture of accelerator 1, for applying the filter in question from a layer of the neural network to an IN matrix. During this configuration, the number of "active" rows of grid 2 is set equal to the number of rows of the filter (p), and the number of "active" columns of grid 2 is taken equal to the number of rows of the OUT matrix (m). In the case represented in figures 1 , 10 , 11 And 12 , these numbers p and m are equal to 4 and the number n of rows of the matrix IN is equal to 7.

[0044] Global memory 3 (for example, external DRAM or global SRAM buffer) contains all the initial data: the filter matrix weights and the input data from the Input Feature Map matrix to be processed. Global memory 3 is also suitable for storing the output data delivered by grid 2; in this example, by the power outputs (PEs) on the north edge of grid 2. A set of communication buses (not shown) connects global memory 3 and grid 2 to handle these data exchanges.

[0045] In what follows and in the figures, the set of data in the (i+1)th row of the weights in the filter matrix is ​​denoted F rowi , i = 0 to p-1 ; the set of data in the (i+1)th row of the matrix IN is denoted in rowi , i = 0 to n-1 ; the data from a partial sum calculation performed by the PE ij is denoted psum ij i = 0 to 3 and j = 0 to 3.

[0046] The arrows in Figure 1 represent how the data is reused in grid 2. Indeed, the rows of the same filter, F rowi , i = 0 to p-1, are reused horizontally across the PEs (this is therefore a horizontal multicast of the filter weights), the rows in rowi of IN, i = 0 to n-1 are reused diagonally across the PEs (a diagonal multicast of the input image, here implemented by the succession of a horizontal multicast and a vertical multicast) and the partial sums psum are accumulated vertically across the PEs (this is a unicast of the psum), as represented by the vertical dashed arrows.

[0047] During the computation of deep CNNs, each piece of data can be accessed many times by MAC operations implemented by the PEs. Repeatedly loading this data from global memory 3 would introduce an excessive number of memory accesses. The energy consumption of global memory accesses can be much greater than that of logical computations (e.g., MAC operations). Reusing data from processing blocks 10, made possible by communication between these blocks within the accelerator 1, limits accesses to global memory 3 and thus reduces the resulting energy consumption.

[0048] Accelerator 1 is adapted to implement, in the inference phase of the neural network, the parallel reuse, described above, by the PEs of the three types of data which are the filter weights, the input data of the Input Feature Map matrix and the partial sums, as well as the recovery, by calculation, of the communications, in an embodiment of the invention.

[0049] Accumulator 1 is particularly suitable for implementing the steps described below of a treatment process 100, with reference to the figure 3 and to figures 10 , 11 , 12 .

[0050] In a step 101, with reference to figures 3 And 10 , a parallel feed of the gate 2 is made with the filter weights and the input data of the IN matrix, via the bus between the global memory 3 and the gate 2.

[0051] Thus, at the processing cycle T0 (the cycles are timed by the accelerator 1 clock): The first column of grid 2 is fed by the respective rows of the filter: the weight row F rowi , i = 0 to 3 is provided as input to processing block 10 (i, 0); the first column and the last row of grid 2 are fed by the respective rows of the Input Feature Map matrix: the row in rowi , i = 0 to 3 is provided as input to processing block 10 (i, 0) and the row in rowi , i = 4 to 6 is provided as input to processing block 10 (3, i-3).

[0052] In cycle T1 following cycle T0, the weights and data from the IN matrix received by each of these blocks 10 are stored in respective registers of memory 13 of block 10.

[0053] In step 102, with reference to figures 3 And 11The dissemination of filter weights and input data within the network is iterative: it is performed in parallel, by horizontal multicast of filter weight lines and diagonal multicast of input feature map lines, as shown sequentially in figure 11 and synthesized into figure 3 .

[0054] Thus, in cycle T2: the first column sends, by horizontal diffusion, to the second column of grid 2 the respective lines of the filter previously stored: the line of weight F rowi , i = 0 to 3, is provided as input to the processing block 10 (i, 1) by the processing block (i,0);and in parallel each of the processing blocks 10 (i, 0) transmits the line in rowi , i = 1 to 3 - and each of the processing blocks 10 (3, i-3) transmits the line in rowi , i = 4 to 6 - to the processing block 10 which is next to them in the NE direction (for example the block (3,0) transmits to the block (2,1)): to arrive at the destination, this will take in the present case, to reach this neighbor, it is actually necessary to make two transmissions: a horizontal transmission and a vertical transmission (for example for the data to arrive from block 10 (3,0) to block 10 (2,1), it will go from block (3,0) to block (3,1), then to block (2,1): so first the neighbors to the East of the processing blocks 10 (i, 0), i = 1 to 3 and of the processing blocks 10 (3, i-3) receive the line first;the first column of the processing blocks 10 having the filter weights and the input data of the matrix IN, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum 0j by the PE 0j , j = 0 to 3, is stored in a register of memory 13. ;

[0055] At cycle T3, the filter weights and data from the IN matrix received at T2 by these blocks 10 at T2 are stored in respective registers of memory 13 of each of these blocks 10.

[0056] In cycle T4, in parallel: the second column feeds, by horizontal diffusion, the third column of grid 2 with the respective lines of the filter previously stored: the line of weight F rowi , i = 0 to 3, is provided as input to the processing block 10 (i, 2) by the processing block (i, 1); the processing blocks 10 (i-1, 1) receive the line in rowi , i = 1 to 3 and each of the processing blocks 10 (2, i-2) receives the line in rowi , i = 4 to 5; the second column of the processing blocks 10 having the filter weights and the input data of the matrix IN, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum 1j by the PE 1j , j = 0 to 3, is stored in a register of memory 13.

[0057] At cycle T5, the filter weights and data from the IN matrix received at T4 by these blocks 10 are stored in respective registers of memory 13 of each of these blocks 10.

[0058] In cycle T6, in parallel: the third column feeds, by horizontal diffusion, the fourth column of grid 2 with the respective lines of the filter previously stored, thus completing the diffusion of the filter weights in grid 2: the line of weight F rowi , i = 0 to 3, is provided as input to the processing block 10 (i, 3) by the processing block (i,2); the processing blocks 10 having received at time T4 a line of the IN matrix and having a neighbor in the direction of NE in turn emit this line of IN matrix towards this neighbor.

[0059] At cycle T7, the filter weights and data from the IN matrix received at T4 by these blocks 10 are stored in respective registers of memory 13 of each of these blocks 10.

[0060] At cycle T8, the third column of the processing blocks 10 having the filter weights and the input data of the IN matrix, the PEs of these blocks implement a convolution calculation between the filter and (some at least of) these input data; the result thus calculated of partial sum psum 2j by the PE 2j , j = 0 to 3, is stored in a register of memory 13.

[0061] The blocks of 10 processing that received at time T6 a line of the IN matrix and having a neighbor in the direction of the NE in turn emit this line of the IN matrix towards this neighbor.

[0062] The diagonal spread continues.

[0063] In cycle T12, block 10 (03) in turn received the line in row3.

[0064] The fourth column of the processing blocks 10 having the filter weights and the input data of the IN matrix, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum 3j by the PE 3j , j = 0 to 3, is stored in a register of memory 13.

[0065] In step 103, with reference to figures 3 And 12A parallel transfer of the partial sums (psums) is performed, and these psums are accumulated: the processing blocks 10 of the last row of grid 2 each send the calculated partial sum to their neighbor located to the north. This neighbor adds this received partial sum to the one it previously calculated and, in turn, sends the accumulated partial sum to its northern neighbor, which repeats the same operation, and so on, until the processing blocks 10 of the first row of grid 2 have performed this accumulation (all these processes being performed in a timed manner by the accelerator clock 1). This final accumulation, performed by each processing block (0,j), j = 0 to 3, corresponds to (a part of) the data from row j of the OUT matrix. It is then delivered by the processing block (0,j) to global memory 3 for storage.

[0066] The Output Feature maps of the considered convolution layer are thus determined as a function of the outputs Out rowi, i = 0 to 3.

[0067] As has been demonstrated with reference to the figure 3 , the diffusion of filter weights is carried out in accelerator 1 (a multicast of filter weights with horizontal reuse of filter weights across processing blocks 1) in parallel with the diffusion of input data of matrix IN (multicast of image lines with diagonal reuse across processing blocks 10).

[0068] Overlaying communications through computation reduces data transfer costs by improving the execution time of parallel programs and decreasing the effective contribution of data transfer time to the overall application execution time. Computations are decoupled from data communication in the grid so that the PEs (Program Electronics) perform computational work while the communication infrastructure (routers and communication links) handles data transfer. This partially or completely hides the overhead of communication, although perfect overlap is only possible if computation time exceeds communication time and the hardware can support this paradigm.

[0069] In the embodiment described above with respect to the figure 3 It is expected that all psums will be calculated before they are accumulated. In another embodiment, psum accumulation is initiated on the first columns of the network while the transfer of filter data and IN matrix data continues in the columns further east, and therefore the psums for these columns have not yet been calculated. In this case, there is an overlap of communications by the communications of the partial psums, which further reduces the contribution of data transfers to the total execution time of the application and thus improves performance. The first columns can then, optionally, be used more quickly for other storage and calculations, thereby further improving the overall processing time.

[0070] The operations described above were performed in the specific case of a Row Stationary (RS) Dataflow and a Conv2D convolutional layer (see Y. Chen et al. 2017. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-State Circuits 52, 1 (Nov. 2017), 127-138). However, other types of Dataflow execution (WS Weight-Stationary Dataflow, IS Input-Stationary Dataflow, OS Output-Stationary Dataflow, etc.) involve different data reuse schemes between PEs, and therefore different transfer paths, different computational organizations, different types of CNN layers (Fully Connected, PointWise, DepthWise, Residual), etc.can be implemented according to the invention: the transfers of data of each type of data (filter, ifmap, psum), for their parallel reuse, must thus be able to be carried out in any of the possible directions in the routers, and this in parallel with the transfers of data of each other type (it should be noted that embodiments may of course use only some of the proposed options: for example, the spatial reuse of only a subset of the data types among the data of type filter, Input Feature Maps, partial sums).

[0071] To this end, the routing device 12 includes, with reference to the figure 5 , a block of parallel routing controls 120, a block of parallel arbiters 121, a block of parallel switches 122 and a block of parallel input buffers 123.

[0072] Indeed, through these different buffering modules (for example FIFO, First-In-First-Out) of block 123, different data communication requests (filters, IN data or psums) received in parallel (for example from a neighboring block 10 to the East (E), West (W), North (N), South (S), or locally from the PE or registers) can be stored without any loss.

[0073] These requests are then processed simultaneously in several control modules within the parallel routing control block 120, based on the Flit (Flow control unit) headers of the data packets. These routing control modules ensure deterministic control of data transfer according to a static XY routing algorithm (for example) and manage different types of communication (unicast, horizontal, vertical, or diagonal multicast, and broadcast).

[0074] The resulting requests issued by the routing control modules are provided as input to the parallel arbiter block 122. Parallel priority arbitration of the order incoming data packets are performed according to, for example, the Round-Robin arbitration policy based on scheduled access. This allows for better collision management; that is, a request that has just been granted will have the lowest priority in the next arbitration round. In the case of simultaneous requests for the same output (E, W, N, S), the requests are logged to avoid deadlock or data loss (i.e., two simultaneous requests for the same output within the same router 12 are not served in the same round). The arbitration decision is then communicated to the parallel switch block 122.

[0075] Parallel switching simultaneously routes data to the correct outputs according to the Wormhole switching rule, for example, that is, the connection between one of the inputs and one of the outputs of a router is maintained until all the elementary data of a message packet are sent, and this simultaneously through the different switching modules for their respective N, E, S, W, L directions.

[0076] The data packet format is represented in Figure 4 The packet has a configurable size of W data (32 bits in the figure) and consists of a header flit followed by payload flits. The packet size depends on the size of the interconnection network, since the more routers there are, the more bits are needed to encode the addresses of the senders or recipients. Similarly, the packet size varies with the size of the payloads (filter weights, input activations, or partial sums) to be circulated in the grid. The header value determines the communication to be handled by the router. Several communication types are possible: unicast, horizontal multicast, vertical multicast, diagonal multicast, broadcast, and memory access. Router 1 first receives the control packet containing the communication type and the recipient or source, identified by its coordinates (i,j) in the grid, as shown in Figure 1. figure 4 Router 11 decodes this control word and then allocates the communication path to transmit the useful data packet that arrives in the cycle following the reception of the control packet. The corresponding packet pairs are represented in figure 4 (a, b, c) Once the useful data packet has been transmitted, the allocated path will be freed up to perform other transfers.

[0077] In one embodiment, router 12 is adapted to prevent back-transfer during multicast (multicast and broadcast communications), in order to avoid transfer loops and to better control the data transmission delay throughout the grid 12. Indeed, during broadcast according to the invention, packets originating from one or more directions will be forwarded to the other directions, the source direction(s) being inhibited. This makes the maximum broadcast delay in a network of size NxM equal to [(N-1) + (M-1)]. Thus, when a packet to be broadcast arrives at the input of router 12 of a processing block 10 (block A) from a neighboring block 10 located in an E, W, N, or S direction relative to block A, this packet is forwarded in parallel in all directions except that of said neighboring block.

[0078] Furthermore, in one embodiment, when a packet is to be transmitted by multicast (horizontal or vertical) from a processing block 10: if said block is the source (i.e. the packet comes from the PE of the block), the multicast is bidirectional (it is carried out in parallel towards E and W for a horizontal multicast, towards S and N for a vertical multicast); otherwise the multicast is unidirectional, directed in the opposite direction from the neighboring processing block 10 from which the packet originates.

[0079] In one embodiment, to guarantee and facilitate the recovery of communications by calculation, with reference to the figure 6 , the control block 30 includes a global control block 31, a calculation control block 32 and a communication control block 33: the communication control is carried out independently of the calculation control, while keeping synchronization points between the two processes in order to facilitate their simultaneous execution.

[0080] The arithmetic controller 32 manages multiplication and accumulation operations, as well as read and write operations to local memory (e.g., register banks), while the communication controller 33 handles data transfers between global memory 3 and local memory 13, as well as arithmetic data transfers between processing blocks 10. Synchronization points between the two controllers are implemented to prevent data overwriting or loss. With this communication control mechanism independent of the arithmetic control mechanism, weight transfers can be performed in parallel with data transfers, and communication operations can be executed in parallel with arithmetic. This allows not only a single communication to be covered by a single calculation, but also one communication to be covered by another.

[0081] The invention thus proposes a data stream execution solution based on computational communication overlay to improve performance and on the reuse, for example configurable, of data (filters, input images, and partial sums) to reduce multiple memory accesses, thereby ensuring processing flexibility and reducing energy consumption in specialized convolutional neural network (CNN) inference architectures. The invention also proposes parallel routing to guarantee the characteristics of data stream execution by ensuring any-to-any data exchange with wide interfaces to support long data bursts. This routing is designed to support flexible communication with numerous multicast / broadcast requests with non-blocking transfers.

[0082] The invention has been described above in an implementation on NoC. Other types of Dataflow architectures can nevertheless be used.

Claims

1. Processing method in a convolutional neural network accelerator (1) comprising an array (2) of unitary processing blocks (10), each unitary processing block (11) comprising a router (12) and a unitary computational element PE (11) associated with a set (13) of respective local memories, the unitary computational element being configured to perform computational operations selected from multiplications and accumulations on data stored in its local memories, the router being configured to perform a plurality of independent data routings in parallel toward distinct outputs of the router, said method comprising the following steps performed in parallel by the same unitary processing block during a same respective processing cycle clocked by a clock of the accelerator: - receiving and / or transmitting, through the router of the unitary block, first and second data from or to neighboring unitary blocks in the array according to first and second directions selected, on the basis of said data, from among at least the vertical and horizontal directions in the array; - performing, by the unitary computational element, one of said computational operations in relation to data stored in said set of local memories during at least one previous processing cycle.

2. The processing method according to claim 1, wherein said router comprises a parallel routing control block (120), a parallel arbiter block (121), a parallel switch block (122), and a parallel input buffer block (123), the router being capable of receiving and processing different data communication requests in parallel.

3. Processing method according to claim 1 or 2, wherein said accelerator comprises a global control block (31), a computational control block (32), and a communication control block (33), communication control being performed independently of computation control, the computational controller (32) being configured to control the computational operations performed by the unitary computational elements, as well as read and write operations of the associated local memories, and the communication controller (33) being configured to manage data transfers between a global memory (3) and the local memories (13), as well as data transfers between the processing blocks (10).

4. Processing method according to one of the preceding claims, wherein a unitary block (10) performs a transmission of a type selected from broadcast and multicast as a function of a header of the packet to be transmitted, and wherein the unitary block applies at least one of said following rules: for a packet to be transmitted in broadcast originating from a neighboring unitary block (10) located in a given direction relative to said block that is to perform the transmission, said block transmits the packet during one cycle in all directions except the direction of said neighboring block; for a packet to be transmitted in multicast: if the packet originates from the PE of the unitary block, the multicast implemented by the block is bidirectional according to two opposite directions; otherwise, the multicast implemented by the block is unidirectional and directed opposite the neighboring processing block from which said packet originates.

5. Processing method according to one of the preceding claims, wherein, in the event of at least two simultaneous transmission requests toward the same direction by a unitary block during a processing cycle, priority among said requests is arbitrated, the request having the arbitrated priority being transmitted in said direction, and the other request being stored and subsequently transmitted toward said direction during a subsequent processing cycle.

6. Convolutional neural network accelerator comprising an array (2) of unitary processing blocks (10) and a clock, each unitary processing block (11) comprising a router (12) and a unitary computational element PE (11) associated with a set (13) of respective local memories, the unitary computational element being configured to perform computational operations selected from multiplications and accumulations on data stored in its local memories, the router being configured to perform a plurality of independent data routings in parallel toward distinct outputs of the router, wherein the same unitary processing block (10) of the array is configured, during a same processing cycle clocked by the clock of the accelerator, to: - receive and / or transmit, through the router of the unitary block, first and second data from or to neighboring unitary blocks in the array according to first and second directions selected, on the basis of said data, from among at least the vertical and horizontal directions in the array; - perform one of said computational operations in relation to data stored in its set of local memories during at least one previous processing cycle.

7. Convolutional neural network accelerator (1) according to the preceding claim, wherein said router comprises a parallel routing control block (120), a parallel arbiter block (121), a parallel switch block (122), and a parallel input buffer block (123), the router being capable of receiving and processing different data communication requests in parallel.

8. Convolutional neural network accelerator (1) according to claim 6 or 7, comprising a global control block (31), a computational control block (32), and a communication control block (33), communication control being performed independently of computational control, the computational controller (32) being configured to control the computational operations performed by the unitary computational elements, as well as read and write operations of the associated local memories, and the communication controller (33) being configured to manage data transfers between a global memory (3) and the local memories (13), as well as data transfers between the processing blocks (10).

9. Convolutional neural network accelerator (1) according to one of claims 6 to 8, wherein a unitary block (10) is configured to perform a transmission of a type selected from broadcast and multicast as a function of a header of the packet to be transmitted, and the unitary block (10) is configured to apply at least one of the following rules: for a packet to be transmitted in broadcast originating from a neighboring block 10 located in a given direction relative to said block that is to perform the transmission, said block transmits the packet during one cycle in all directions except the direction of said neighboring block; for a packet to be transmitted in multicast: if the packet originates from the PE of the unitary block, the multicast implemented by the block is bidirectional according to two opposite directions; otherwise, the multicast implemented by the block is unidirectional and directed opposite the neighboring processing block from which said packet originates.

10. Convolutional neural network accelerator (1) according to one of claims 6 to 9, wherein, in the event of at least two simultaneous transmission requests toward the same direction by a unitary block (10) during a processing cycle, the routing block of the unitary block is configured to arbitrate priority among said requests, the request having the arbitrated priority being transmitted in said direction, and the other request being stored and subsequently transmitted toward said direction during a subsequent processing cycle.