Processing method in a convolutional neural network accelerator and associated accelerator
The CNN accelerator addresses scalability and energy efficiency issues by employing a grid of processing blocks with optimized data exchange and transmission methods, improving flexibility and reducing energy consumption while maintaining performance across different CNN networks and layers.
Patent Information
- Application Number
- FR2022002559
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-23
AI Technical Summary
Existing convolutional neural network (CNN) accelerators face limitations in scalability, flexibility, and energy efficiency due to issues such as limited bandwidth, excessive memory energy consumption, and inefficient data reuse, which complicates the processing of different communication patterns.
A CNN accelerator architecture utilizing a grid of unit processing blocks with local memories and a communication infrastructure that enables data exchange in vertical and horizontal directions, implementing multicast and broadcast transmission based on packet headers, and prioritizing simultaneous transmission requests to optimize data flow and reduce energy consumption.
This approach enhances the flexibility and reduces energy consumption while maintaining performance by optimizing data reuse and parallelism, facilitating the processing of various CNN networks and layers without degrading performance.
Smart Images

Figure 00000021_0000 
Figure 00000021_0001 
Figure 00000022_0000
Abstract
Description
Title of the invention: Processing method in a convolutional neural network accelerator and associated accelerator Technical field
[0001] The invention lies in the field of artificial intelligence and deep neural networks and more particularly in the field of accelerating inference calculations by convolutional neural networks. Prior art
[0002] Artificial intelligence (AI) algorithms are currently a vast field of research, as they are intended to become essential components of next-generation applications, relying on intelligent processes to make decisions based on knowledge of their environment, for example, in relation to the detection of objects such as pedestrians for a self-driving car or activity recognition for a health-tracking smartwatch. This knowledge is gathered by sensors associated with high-performance detection and / or recognition algorithms.
[0003] In particular, deep neural networks (DNNs) and among them, especially convolutional neural networks (CNNs - see for example Y. Lecun et al. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (Nov. 1998), 2278-2324) are good candidates to be integrated into such systems due to their excellent performance in detection and recognition tasks. They are based on filter layers that perform feature extraction and then classification. These operations are computationally and memory intensive, and the integration of such algorithms into systems requires the use of accelerators. These accelerators are electronic devices that mainly calculate in parallel the multiply-accumulate (MAC) operations that are numerous in CNN algorithms.The objective of these accelerators is to improve the execution performance of CNN algorithms to meet application constraints and improve the energy efficiency of the system. They are mainly based on a high number of processing elements involving operators optimized for MAC execution and a memory hierarchy allowing efficient data storage.
[0004] Most hardware accelerators are based on a network of elementary processors (Processing Elements - PEs) implementing MACs and use local buffer memories to store data that is frequently reused, such as filter parameters or intermediate data. Communications between PEs themselves and those between PEs and memory are a very important aspect to consider when designing a CNN accelerator. Indeed, CNN algorithms have high intrinsic parallelism as well as data reuse possibilities. The on-chip communication infrastructure must therefore be carefully designed to exploit the high number of PEs and the particularities of CNN algorithms, which allow for improvements in both performance and energy efficiency. For example, multicasting or broadcasting particular data in the communication network will allow target PEs to simultaneously process different data with the same filter using a single memory read.
[0005] Several factors have contributed to limiting or complicating the scalability and flexibility of existing CNN accelerators on the market. These factors are manifested through: (i) limited bandwidth linked to the absence of an efficient broadcast support, (ii) energy overconsumption linked to the size of the memory (e.g.: 40% of the energy consumption in certain architectures is induced by the memory) and the memory capacity wall problem (iii) as well as limited data reuse and a need for efficient support for the processing of different communication patterns.
[0006] There is therefore a need to increase the processing efficiency in neural accelerators of CNN architectures, taking into account the high number of PEs and the particularities of CNN algorithms. Summary of the invention
[0007] To this end, according to a first aspect, the present invention describes a processing method in a convolutional neural network accelerator comprising a grid of unit processing blocks, each unit processing block comprising a unit calculation element PE associated with a set of respective local memories and performing calculation operations among multiplications and accumulations of data stored in its local memories.
[0008] said method comprising the following steps:
[0009] during respective processing cycles clocked by a clock of the accelerator, unit blocks of the grid receive and / or transmit data from or to neighboring unit blocks in the grid in at least one direction selected, as a function of said data, from at least the vertical and horizontal directions in the grid;
[0010] during said same cycles, unit blocks of the grid carry out one of said calculation operations relating to data stored in their set of local memories during at least one previous processing cycle.
[0011] Such a method makes it possible to guarantee flexible processing and to reduce energy consumption in CNN architectures comprising an accelerator.
[0012] It offers a DataFlow execution model distributing, gathering and updating among the numerous distributed processing elements (PE), the operands and allows to ensure different degrees of parallelism on the different types of shared data (weights, Ifmaps and Psum) in the CNNs, to reduce the cost of data exchanges without degrading the performances and finally to facilitate the processing of different CNN networks and different layers of the same network (Conv2D, FC, PW, DW, Residual, etc.).
[0013] In embodiments, such a method will further comprise at least one of the following features:
[0014] - at least during one of said treatment cycles:
[0015] at least one unit block of the grid receives data from several neighboring unit blocks in the grid located in distinct directions relative to said unit block; and / or
[0016] at least one unit block of the grid transmits data to several neighboring unit blocks in the grid in distinct directions;
[0017] - a unit block performs a transmission of selected type between broadcast and multicast based on a header of the packet to be transmitted and according to which the unit block applies at least one of said rules:
[0018] for a packet to be transmitted in broadcast originating from a neighboring block located in a given direction relative to said block to carry out the transmission, said block transmits the packet during a cycle in all directions except that of said neighboring block;
[0019] for a packet to be transmitted in multicast: if the packet comes from the PE of the unit block, the multicast implemented by the block is bidirectional, otherwise the multicast implemented by the block is unidirectional, directed opposite the neighboring processing block from which said packet comes;
[0020] the receptions and / or transmissions of data implemented by a unitary processing block are implemented by a routing block included in said unitary block, implementing parallel data routing functions during the same processing cycle, according to communication directions associated with the data;
[0021] in the event of at least two simultaneous transmission requests towards the same direction by a unit block during a processing cycle, the priority between said requests is arbitrated, the priority arbitrated request is transmitted in said direction and the other request is stored, then transmitted towards said direction at a subsequent processing cycle.
[0022] According to another aspect, the invention describes a convolutional neuron accelerator comprising a grid of processing unit blocks and a clock, each processing unit block comprising a computing unit element PE associated with a set of respective local memories and adapted to perform computing operations among multiplications and accumulations of data stored in its local memories.
[0023] wherein unit blocks of the grid are adapted to, during respective processing cycles clocked by the accelerator clock, receive and / or transmit data from or to neighboring unit blocks in the grid in at least one direction selected, as a function of said data, from at least the vertical and horizontal directions in the grid;
[0024] and unit blocks of the grid are adapted to, during said same cycles, carry out one of said calculation operations relating to data stored in their set of local memories during at least one previous processing cycle.
[0025] In embodiments, such an accelerator will further comprise at least one of the following features:
[0026] - at least during one of said treatment cycles:
[0027] at least one unit block of the grid is adapted to receive data from several neighboring unit blocks in the grid located in distinct directions relative to said unit block; and / or
[0028] at least one unit block of the grid is adapted to transmit data to several neighboring unit blocks in the grid in distinct directions;
[0029] - a unit block is adapted to carry out a transmission of selected type between broadcast and multicast depending on a header of the packet to be transmitted and the unit block is adapted to apply at least one of said rules:
[0030] for a packet to be transmitted in broadcast originating from a neighboring block 10 located in a given direction relative to said block having to carry out the transmission, said block transmits the packet during a cycle in all directions except that of said neighboring block;
[0031] for a packet to be transmitted in multicast: if the packet comes from the PE of the unit block, the multicast implemented by the block is bidirectional, otherwise the multicast implemented by the block is unidirectional, directed opposite the neighboring processing block from which said packet comes;
[0032] - a unit block comprises a routing block adapted to implement said receptions and / or transmissions of data by the unit block, said routing block being adapted to implement parallel data routing functions during of the same processing cycle, depending on communication directions associated with the data;
[0033] - in the event of at least two simultaneous transmission requests to the same direction by a unit block during a processing cycle, the routing block of the unit block is adapted to arbitrate the priority between said requests, the priority arbitrated request then being transmitted in said direction and the other request being stored, then transmitted towards said direction at a subsequent processing cycle. Brief description of the drawings
[0034] The invention will be better understood and other characteristics, details and advantages will appear more clearly on reading the following description, given without limitation, and thanks to the appended figures, given by way of example.
[0035] [Fig-1] [Fig.l] represents a neural network accelerator in a mode of realization of the invention;
[0036] [Fig.2] [Fig.2] represents a processing unit block in a mode of realization of the invention;
[0037] [Fig.3] [Fig.3] represents a method in one embodiment of the invention;
[0038] [Fig.4] [Fig.4] represents the structure of communication packets in the accelerator in one embodiment;
[0039] [Fig.5] [Fig.5] shows a routing block in one embodiment of the invention;
[0040] [Fig.6] [Fig.6] shows the calculation control architecture and communication in one embodiment of the invention;
[0041] [Fig.7] [Fig.7] illustrates a stage of convolution calculations;
[0042] [Fig.8] [Fig.8] illustrates another stage of convolution calculations;
[0043] [Fig.9] [Fig.9] represents another stage of convolution calculations;
[0044] [Fig. 10] [Fig. 10] illustrates step 101 of the method of [Fig.3];
[0045] [Fig. 11] [Fig. 11] illustrates step 102 of the method of [Fig.3];
[0046] [Fig. 12] [Fig. 12] illustrates step 103 of the method of [Fig.3].
[0047] Identical references may be used in different figures when they designate identical or comparable elements. Description of the embodiments
[0048] A CNN comprises different types of successive neural network layers, including convolution layers, each layer being associated with a set of filters. A convolution layer analyses by zones, using each filter (for example: horizontal Sobel, vertical Sobel,... or any other filter considered, in particular resulting from learning) of the set of filters, at least one data matrix, which is provided to it as input, called Input Feature Map (also called IN subsequently) and outputs at least one data matrix, called here Output Feature Map (also called OUT subsequently), which allows only what is searched for according to the filter considered to be kept.
[0049] The IN matrix is a matrix of n rows and n columns. A filter F is a matrix of p rows and p columns. The OUT matrix is a matrix of m rows and m columns. In special cases, m = n - p + 1, knowing that the exact formula is:
[0050] m=(n-f+2p) / s +1, where
[0051] m:ofmap(mxm) - size may not be regular
[0052] n: ifmap(nxn) - size may not be regular
[0053] f: filter (fxf)
[0054] p: margin at 0 (0-padding)
[0055] s:step (stride).
[0056] For example, p = 3 or 5 or 9 or 11.
[0057] In a known manner, the convolutions carried out correspond for example to the following process: the filter matrix is placed on the top left corner of the matrix IN, a product of each pair of coefficients thus superimposed is carried out; all of the products are summed, which gives the value of the pixel (1,1) of the output matrix OUT. The filter matrix is then shifted by one box (stride) horizontally to the right, and the process is repeated, providing the value of the pixel (1,2) of the matrix OUT, etc. Once the end of the line has been reached, the filter is moved down vertically by one box, the process is repeated then starting again from the right, etc. until the entire matrix IN has been covered.
[0058] Convolution calculations are generally implemented by neural network calculation units, also called artificial intelligence accelerators or NPUs (Neural Processing Units), comprising a network of PE processor elements.
[0059] In one example, an example of a calculation carried out conventionally in a convolution layer implemented by an accelerator is presented below.
[0060] We consider the filter F consisting of the following weights:
[0061] [Tables 1] inn ini2 inn inu inis inie inn ini8 ini9 in20 in2i in22 in23 in24 in25
[0064] And we consider the following OUT matrix:
[0065] [Tables3] ouT out2 out3 out4 out5 out6 out7 out8 out9
[0066] The expression of each coefficient of the OUT matrix is a weighted sum corresponding to an output of a neuron of which in; would be the inputs and fj would be the weights applied to the inputs by the neuron and which would calculate the value of the coefficient.
[0067] Let us now consider a grid of unit calculation elements pe, comprising as many rows as the filter F (p = 3 rows) and as many columns as the matrix OUT has rows (m = 3): [pei,j] i = 0 to 2 and j = 0 to 2. The following is an example of using the grid to calculate the coefficients of the matrix OUT.
[0068] As shown in [Fig.7], to each coefficient of the (i+1) th row of the pe, we provide the (i+1) th row of the filter matrix, i=0 to 2. We then provide the grid of the pe with the matrix IN: thus the first row of IN is provided to the unitary calculation element peOO; the second row of IN to the coefficients pelO and peOl, located on the same diagonal; the third row of IN is provided to the unitary elements pe20, pel 1 and pe02, located on the same diagonal; the fourth row of IN is provided to the elements pe21 and pel2 on the same diagonal, and the fifth row of IN is provided to pe22.
[0069] In a first burst of calculation also shown in [Fig.7], a convolution (combination of multiplications and sums) is carried out in each pe between the filter row supplied to it and the first p coefficients of the row of the matrix IN supplied to it, delivering a named partial sum (the gray boxes of the row of IN are not used for the current calculation). Thus peOO calculates fl.inl + f2.in2 + f3.in3 etc. Then the three partial sums determined by the pe of the same column are progressively summed: the partial sum determined by pe2j is supplied to pelj which adds it to the partial sum that it previously calculated, this new partial sum resulting from the accumulation is then in turn supplied by pelj to peOj which adds it to the partial sum that it had calculated, j = 0 to 2: the total thus obtained is equal to the first coefficient of the j+1th row of the matrix OUT.
[0070] In a second burst of calculation shown in [Fig.8], a convolution is carried out in each pe between the filter row supplied to it and the p = 3 coefficients, starting from the 2nd coefficient, of the row of the IN matrix supplied to it, delivering a partial sum. Thus peOO calculates fl.in2 + f2.in3 + f3.in4 etc. Then the three partial sums determined by the pe of the same column are progressively summed as described above and the total thus obtained is equal to the second coefficient of the j+1th row of the OUT matrix.
[0071] In a third burst of calculation shown in [Fig.9], a convolution is carried out in each pe between the filter row supplied to it and the p = 3 coefficients, starting from the 3rd coefficient, of the row of the IN matrix supplied to it, delivering a named partial sum. Thus peOO calculates fl.in3 + f2.in4 + f3.in5 etc. Then the three partial sums determined by the pe of the same column are progressively summed as described above and the total thus obtained is equal to the third coefficient of the j+1th row of the OUT matrix.
[0072] In the calculation process described here as an example, the ith line of pes thus makes it possible to successively construct the ith column of OUT, i = 1 to 3.
[0073] It follows from this example that the lines of manipulated data (weights of the filters, data of the Input Feature Map and partial sums) are reused spatially between the unitary processor elements: here for example, the same filter data are used by the pe of the same horizontal line and the same IN data are used by all the pe of diagonal lines, while the partial sums are transferred vertically, then reused.
[0074] It is therefore important that the communications of this data and the calculations involved are carried out in an optimized manner in terms of transfer time and calculation of access to the central memory initially delivering this data, and this regardless of the dimensions of the input data and output data or the calculations implemented.
[0075] For this purpose, with reference to [Fig.l], a CNN neural network accelerator 1 in one embodiment of the invention comprises a grid 2 of unit processing blocks 10, a global memory 3 and a control block 30.
[0076] The grid 2 of unit processing blocks 10 comprises unit processing blocks 10 arranged in a network, connected by horizontal and vertical communication links allowing the exchange of data packets between unit blocks, for example according to a matrix organization in N rows and M columns.
[0077] Accelerator 1 has for example an architecture based on a NoC (Network on Chip).
[0078] In one embodiment, each processing block 10 comprises, with reference to [Fig.2], a PE (Processing Element in English) processor 11 adapted for carry out calculation operations, of the MAC type in particular, a memory assembly 13, comprising for example several registers, intended to store the filter data, the Input Feature Map input data received by the processing block 10 and the results (partial sums, accumulations of partial sums) calculated by the PE 11 in particular, and a router 12 adapted to route incoming or outgoing data communications.
[0079] A unit processing block 10 (and similarly its PE) is identified by its row and column rank in the grid, as shown in figures 1, 10, 11 and 12. Thus the processing block 10 (i,j), comprising the PE;j 11, is located on the i+1th row and j + 1th column of the grid 2, i = 0 to 3 and j = 0 to 3.
[0080] Each processing block 10 not located at the edge of the network thus comprises 8 neighboring processing blocks 10, in the following directions: one to the North (N), one to the South (S), one to the West (W), one to the East (E), one to the North-East, one to the North-West, one to the South-East, and one to the South-West.
[0081] The control block 30 is adapted to synchronize, with each other, the calculation processing in the PEs and the data transfer between unit blocks 10 or within unit blocks 10 and implemented in the accelerator 1. All these processings are clocked by a clock of the accelerator 1.
[0082] There will have been a prior step of configuring the grid 2 to select the set of PEs to be used, among the available PEs of the maximum hardware architecture of the accelerator 1, for the application of the filter considered from a layer of the neural network to a matrix IN. During this configuration, the number of “active” rows of the grid 2 is set equal to the number of rows of the filter (p) and the number of “active” columns of the grid 2 is taken equal to the number of rows of the matrix OUT (m). In the case represented in figures 1, 10, 11 and 12, these numbers p and m are equal to 4 and the number n of rows of the matrix IN is equal to 7.
[0083] The global memory 3, for example of the external DRAM memory type or global SRAM buffer memory, here contains all the initial data: the weights of the filter matrix and the input data of the Input Feature Map matrix to be processed. The global memory 3 is also adapted to store the output data delivered by the grid 2, in the example considered, by the PEs at the North edge of the grid 2. A set of communications buses (not shown) connects for example the global memory 3 and the grid 2 to ensure these data exchanges.
[0084] In the following and in the figures, the set of data of the (i+1)th row of the weights in the filter matrix is denoted Frowi, i = 0 to p-1; the set of data of the (i+1)th row of the matrix IN is denoted inrowi, i = 0 to n-1; the data resulting from a calculation of partial sums carried out by the PEÿ is denoted psumÿi = 0 to 3 and j = 0 to 3.
[0085] The arrows in [Fig.l] represent the way in which the data are reused in grid 2. Indeed, the rows of the same filter, Frowi, i = 0 to p-1, are reused horizontally across the PEs (this is therefore a horizontal multicast of the filter weights), the rows inrowi of IN, i = 0 to n-1 are reused diagonally across the PEs (a diagonal multicast of the input image, here implemented by the succession of a horizontal multicast and a vertical multicast) and the partial sums psum are accumulated vertically across the PEs (this is a unicast of the psum), as represented by the dotted vertical arrows.
[0086] During the calculation of the deep CNNs, each data item can be exploited many times by MAC operations implemented by the PEs. The repeated loading of this data from the global memory 3 would introduce an excessive number of memory accesses. The energy consumption of the accesses to the global memory can be much greater than that of the logical calculations (MAC operation for example). The reuse of the data of the processing blocks 10 enabled by the communications between the blocks 10 of this data in the accelerator 1 makes it possible to limit the accesses to the global memory 3 and thus to reduce the induced energy consumption.
[0087] The accelerator 1 is adapted to implement, in the inference phase of the neural network, the parallel reuse, described above, by the PEs of the three types of data which are the filter weights, the input data of the Input Feature Map matrix and the partial sums, as well as the recovery, by calculation, of the communications, in one embodiment of the invention.
[0088] The accumulator 1 is in particular adapted to implement the steps described below of a treatment method 100, with reference to [Fig. 3] and to figures 10, 11, 12.
[0089] In a step 101, with reference to figures 3 and 10, a parallel supply of the grid 2 is carried out with the filter weights and the input data of the matrix IN, via the bus between the global memory 3 and the grid 2.
[0090] Thus, at processing cycle T0 (the cycles are clocked by the clock of accelerator 1):
[0091] the first column of grid 2 is fed by the respective lines of the filter: the line of weight Frowi, i = 0 to 3 is provided as input to processing block 10 (i, 0);
[0092] the first column and the last row of grid 2 are fed by the respective rows of the Input Feature Map matrix: the row inrowi, i = 0 to 3 is provided as input to the processing block 10 (i, 0) and the row inrowi, i = 4 to 6 is provided as input to the processing block 10 (3, i-3).
[0093] In cycle T1 following cycle T0, the weights and data from matrix IN received by each of these blocks 10 are stored in respective registers of memory 13 of block 10.
[0094] In a step 102, with reference to figures 3 and 11, the diffusion of the filter weights and the input data within the network is iterated: it is carried out in parallel, by horizontal multicast of the filter weight lines and diagonal multicast of the lines of the input image Input Feature Map, as represented sequentially in [Fig.l 1] and synthesized in [Fig.3].
[0095] Thus in cycle T2: - the first column sends, by horizontal diffusion, to the second column of grid 2 the respective lines of the previously stored filter: the line of weight Frowi, i = 0 to 3, is provided as input to processing block 10 (i, 1) by processing block (i, 0); and in parallel - each of the processing blocks 10 (i, 0) transmits the line inrowi, i = 1 to 3,= - and each of the processing blocks 10 (3, i-3) transmits the line inrowi, i = 4 to 6 - to the processing block 10 which is their neighbor in the NE direction (for example, block (30) transmits to block (21)): to reach the destination, in this case, to reach this neighbor, it is actually necessary to make two transmissions: a horizontal transmission and a vertical transmission (for example, for the data to reach from block 10 (30) to block 10 (21), it will go from block (30) to block (31), then to block (21): so first the neighbors to the East of the processing blocks 10 (i, 0), i = 1 to 3 and of the processing blocks 10 (3, i-3) first receive the line; - the first column of the processing blocks 10 having the filter weights and the input data of the matrix IN, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psumOj by the PEOj, j = 0 to 3, is stored in a register of the memory 13.
[0096] In cycle T3, the filter weights and data from the IN matrix received at T2 by these blocks 10 to T2 are stored in respective registers of the memory 13 of each of these blocks 10.
[0097] In cycle T4, in parallel:
[0098] the second column feeds, by horizontal diffusion, the third column of the grid 2 by the respective lines of the filter previously stored: the line of weight Frowi, i = 0 to 3, is provided at the input of the processing block 10 (i, 2) by the processing block (U);
[0099] the processing blocks 10 (i-1, 1) receive the line inrowi, i = 1 to 3 and each of the processing blocks 10 (2, i-2) receive the line inrowi, i = 4 to 5;
[0100] the second column of the processing blocks 10 having the filter weights and the input data of the matrix IN, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum^ by the PEij, j = 0 to 3, is stored in a register of the memory 13.
[0101] At cycle T5, the filter weights and data from the IN matrix received at T4 by these blocks 10 are stored in respective registers of the memory 13 of each of these blocks 10.
[0102] In cycle T6, in parallel:
[0103] the third column feeds, by horizontal diffusion, the fourth column of grid 2 with the respective lines of the previously stored filter, thus completing the diffusion of the filter weights in grid 2: the line of weight Frowi, i = 0 to 3, is provided as input to processing block 10 (i, 3) by processing block (i, 2);
[0104] the processing blocks having received at time T4 a line of the IN matrix and having a neighbor in the direction of the NE in turn transmit this line of IN matrix to this neighbor.
[0105] In cycle T7, the filter weights and data from the IN matrix received at T4 by these blocks 10 are stored in respective registers of the memory 13 of each of these blocks 10.
[0106] In cycle T8, the third column of the processing blocks 10 having the filter weights and the input data of the matrix IN, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum2j by the PE2j, j = 0 to 3, is stored in a register of the memory 13.
[0107] the processing blocks having received at time T6 a line of the IN matrix and having a neighbor in the direction of the NE in turn transmit this line of IN matrix to this neighbor.
[0108] Diagonal diffusion continues.
[0109] In cycle T12, block 10 (03) in turn received line inrow3.
[0110] The fourth column of the processing blocks 10 having the filter weights and the input data of the IN matrix, the PEs of these blocks implement a convolution calculation between the filter and (at least some of) these input data; the result thus calculated of partial sum psum3j by the PE3j, j = 0 to 3, is stored in a register of memory 13.
[0111] In a step 103, with reference to figures 3 and 12, a parallel transfer of the partial sums psum is carried out and these psums are accumulated: the processing blocks 10 of the last row of the grid 2 each send the calculated partial sum to their neighbor located in the North direction. The latter accumulates this partial sum received with the one it previously calculated and in turn sends the accumulated partial sum to its North neighbor who repeats the same operation etc. until the processing blocks 10 of the first row of the grid 2 have carried out this accumulation (all of these processing operations being carried out in a manner timed by the clock of the accelerator 1). This last accumulation carried out by each processing block (0,j) j = 0 to 3, corresponds to (a part of) the data of row j of the matrix OUT. It is then delivered by the processing block (0,j) to the global memory 3 for storage.
[0112] The Output Feature maps results of the convolution layer considered are thus determined according to the outputs Outl<)WI, i = 0 to 3.
[0113] As demonstrated with reference to [Fig.3], the diffusion of the filter weights is performed in the accelerator 1 (a multicast of the filter weights with a horizontal reuse of the filter weights across the processing blocks 1) in parallel with the diffusion of the input data of the matrix IN (multicast of the rows of the image with a diagonal reuse across the processing blocks 10).
[0114] The overlap of communications by computation allows a reduction of the data transfer cost by improving the execution time of parallel programs by reducing the effective contribution of the time dedicated to the transfer of data to the execution time of the complete application. The computations are decoupled from the communication of data in the grid so that the PEs 11 carry out computational work while the communication infrastructure (routers 12 and communication links) carries out the transfer of data. This makes it possible to partially or totally hide the communication overhead, knowing that the overlap can only be perfect if the computation time exceeds the communication time and that the hardware can support this paradigm.
[0115] In the embodiment described above with respect to [Fig. 3], it is expected that all the psums are calculated before proceeding with their accumulation. In another embodiment, the accumulation of the psums is launched on the first columns of the network even while the transfer of the filter data and the IN matrix data continues in the columns further to the East and therefore the psums for these columns are not yet calculated: there is therefore in this case an overlap of the communications by the communications of the partial psum sums, which makes it possible to further reduce the contribution of the data transfers in the total execution time of the application and thus improve the performances. The first columns can then optionally be used more quickly for further storage and calculation, thus further improving the overall processing time.
[0116] The operations have been described above in the particular case of a RS (Row Stationary) Dataflow and a convolutional layer of type Conv2D (cf. Y. Chen ae al. 2017. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE Journal of Solid-State Circuits 52, 1 (Nov. 2017), 127-138). But other types of Dataflow execution (WS Weight-Stationary Dataflow, IS Input-Stationary Dataflow, OS Output-Stationary Dataflow, etc.) involving other data reuse schemes between PEs, therefore other transfer paths, other organizations of calculations, other types of CNN layers (Fully Connected, PointWise, depthWise, Residual) etc.can be implemented according to the invention: the transfers of data of each type of data (filter, ifmap, psum), for their parallel reuse, must thus be able to be carried out in any of the possible directions in the routers, and this in parallel with the transfers of data of each other type (it will be noted that embodiments can of course only use some of the proposed options: for example, the spatial reuse of only a subset of the data types among the filter type data, Input Feature Maps, partial sums).
[0117] For this purpose, the routing device 12 comprises, with reference to [Fig. 5], a parallel routing control block 120, a parallel arbiter block 121, a parallel switch block 122 and a parallel input buffer block 123.
[0118] Indeed, through these different buffering modules (for example FIFOs, First-In-First-Out) of block 123, different data communication requests (filters, IN data or psums) received in parallel (for example from a neighboring block 10 to the East (E), West (W), North (N), South (S), or locally to the PE or registers) can be stored without any loss.
[0119] These requests are then processed simultaneously in several control modules within the parallel routing control block 120, according to the Flit (Flow control unit) headers of the data packets. These routing control modules provide deterministic control of the data transfer according to a static XY routing algorithm (for example) and management of different types of communications (unicast, horizontal, vertical or diagonal multicast and broadcast).
[0120] The resulting requests issued by the routing control modules are provided as input to the parallel arbiter block 122. A parallel priority arbitration of the order of processing of incoming data packets according to, for example, the Round-Robin arbitration policy based on scheduled access makes it possible to better manage collisions, i.e., a request which has just been granted will have the highest priority. low on the next arbitration cycle. In the event of simultaneous requests for the same output (E, O, N, S), the requests are stored to avoid deadlock or data loss (i.e. two simultaneous requests on the same output within the same router 12 are not served on the same cycle). The arbitration carried out is then indicated to the parallel switch block 122.
[0121] Parallel switching simultaneously routes the data to the correct outputs according to the Wormhole switching rule for example, that is to say that the connection between one of the inputs and one of the outputs of a router is maintained until all the elementary data of a packet of the message are sent and this simultaneously through the different switching modules for their respective direction N, E, S, W, L.
[0122] The format of the data packet is shown in [Fig.4]. The packet is of configurable size Wdata (in the figure 32 bits) and is composed of a header flit followed by payload flits. The size of the packet will depend on the size of the interconnection network, since the more the number of routers 11 increases, the more the number of bits to encode the addresses of the recipients or transmitters increases. Similarly, the size of the packet varies with the size of the payloads (weights of the filters, the activations of inputs or the partial sums) to be circulated in the grid 2. The value of the header determines the communication to be ensured by the router. There are several possible types of communication: unicast, horizontal multicast, vertical multicast, diagonal multicast and broadcast, memory access 3.Router 11 first receives the control packet containing the type of communication and the recipient or source, identified by its coordinates (i,j) in the grid, as shown in [Fig.4]. Router 11 decodes this control word and then allocates the communication path to transmit the useful data packet that arrives in the cycle following the reception of the control packet. The corresponding packet pairs are shown in [Fig.4] (a, b, c) Once the useful data packet is transmitted, the allocated path will be freed to carry out other transfers.
[0123] In one embodiment, the router 12 is adapted to prevent back-transfer during multicasting (multicast and broadcast communications), in order to avoid transfer loopback and to better control the data transmission delay in the entire grid 12. Indeed, during broadcasting according to the invention, packets from one or more directions will be transmitted to the other directions, the source direction(s) being inhibited. This makes the maximum broadcast delay in a network of size NxM equal to [(Nl) + (Ml)]. Thus when a packet to be broadcast arrives at the input of a router 12 of a processing block 10 (block A) from a neighboring block 10 located in a direction E, W, N or S relative to block A, this packet is sent in parallel in all directions except that of said neighboring block.
[0124] Furthermore, in one embodiment, when a packet is to be transmitted in multicast (horizontal or vertical) from a processing block 10: if said block is the source (i.e. the packet comes from the PE of the block), the multicast is bidirectional (it is carried out in parallel towards E and O for a horizontal multicast, towards S and N for a vertical multicast); otherwise the multicast is unidirectional, directed opposite the neighboring processing block 10 from which the packet comes.
[0125] In one embodiment, to ensure and facilitate the recovery of communications by the calculation, with reference to [Fig.6], the control block 30 comprises a global control block 31, a calculation control block 32 and a communication control block 33: the communication control is carried out independently of the calculation control, while keeping synchronization points between the two processes in order to facilitate their simultaneous execution.
[0126] The calculation controller 32 makes it possible to control the multiplication and accumulation operations, as well as the reading and writing operations of the local memories (for example, register bank), while the communication controller 33 manages the data transfers of the global memory 3 and the local memories 13, as well as the transfers of calculation data between processing blocks 10. Synchronization points between the two controllers are set up in order to avoid overwriting or loss of data. With this communication control mechanism independent of that of the calculation, it is possible to ensure the transfer of the weights in parallel with the transfer of the data and the execution of the communication operations in parallel with the calculation. It is thus possible to cover not only the communication by the calculation but also a communication by another communication.
[0127] The invention thus proposes a solution for executing the data flow based on the recovery of communications by calculation to improve performance and on the reuse, for example configurable, of data (filters, input images and partial sums) in order to reduce multiple accesses to memories, making it possible to ensure the flexibility of processing and reduce energy consumption in specialized architectures of convolutional neural networks (CNNs) for inferences. The invention also proposes parallel routing to guarantee the characteristics of the execution of the data flow by ensuring "any-to-any" type data exchanges with wide interfaces to support long bursts of data. This routing is designed to support flexible communication with numerous multicast / broadcast requests with non-blocking transfers.
[0128] The invention has been described above in an implementation on NoC. Other types of Dataflow architectures can nevertheless be used.
Claims
Claims
1. Processing method in a convolutional neural network accelerator (1) comprising a grid (2) of unit processing blocks (10), each unit processing block (11) comprising a router (12) and a unit computing element PE (11) associated with a set (13) of respective local memories, the unit computing element making it possible to carry out calculation operations among multiplications and accumulations of data stored in its local memories, the router making it possible to carry out several data routings in parallel according to communication directions associated with the data, said method comprising the following steps carried out in parallel by the same unit processing block during a respective processing cycle clocked by a clock of the accelerator: - receiving and / or transmitting, through the router of the unit block, a set of data from, or to,of neighboring unit blocks in the grid according to first and second directions selected, as a function of said data, from at least the vertical and horizontal directions in the grid; said set of data received, respectively transmitted, comprising first and second data and the first direction, which is selected for the first data, being distinct from the second direction, which is selected for the second data; - performing, by the unitary calculation element, one of said calculation operations relating to data stored in said set of local memories during at least one previous processing cycle.,
2. A processing method according to claim 1, wherein said router comprises a parallel routing control block (120), a parallel arbiter block (121), a parallel switch block (122) and a parallel input buffer block (123), the router being able to receive and process different data communication requests in parallel.
3. A processing method according to claim 1 or 2, wherein said accelerator comprises a global control block (31), a calculation control block (32) and a communication control block (33), the communication control is carried out independently of the calculation control, the calculation controller (32) making it possible to control the calculation operations carried out by the unit calculation elements, as well as the reading and writing operations of the associated local memories, the communication controller (33) managing the data transfers between a global memory (3) and the local memories (13), as well as the data transfers between the processing blocks (10).
4. Processing method according to one of the preceding claims, according to which a unit block (10) carries out a transmission of type selected between broadcast and multicast according to a header of the packet to be transmitted and according to which the unit block applies at least one of said rules: for a packet to be transmitted in broadcast coming from a neighboring unit block (10) located in a given direction relative to said block having to carry out the transmission, said block transmits the packet during a cycle in all directions except that of said neighboring block; for a packet to be transmitted in multicast: if the packet comes from the PE of the unit block, the multicast implemented by the block is bidirectional in two opposite directions, otherwise the multicast implemented by the block is unidirectional, directed opposite the neighboring processing block from which said packet comes.
5. Processing method according to one of the preceding claims, according to which in the event of at least two simultaneous transmission requests towards the same direction by a unit block during a processing cycle, the priority between said requests is arbitrated, the priority arbitrated request is transmitted in said direction and the other request is stored, then transmitted towards said direction at a subsequent processing cycle.
6. Convolutional neuron accelerator comprising a grid (2) of unit processing blocks (10) and a clock, each unit processing block (11) comprising a router (12) and a unit computing element PE (11) associated with a set (13) of respective local memories, the unit computing element making it possible to carry out computing operations among multiplications and accumulations of data stored in its local memories, the router being adapted to carry out several data routings in parallel according to communication directions associated with the data, wherein a single unit processing block (10) of the grid is adapted to, during a processing cycle clocked by the accelerator clock: - receive and / or transmit, through the router of the unit block, a set of data from, or to, neighboring unit blocks in the grid according to first and second directions selected, as a function of said data, from among at least the vertical and horizontal directions in the grid; said set of data received, respectively transmitted, comprising first and second data and the first direction, which is selected for the first data, being distinct from the second direction, which is selected for the second data; - carry out one of said calculation operations relating to data stored in their set of local memories during at least one previous processing cycle.
7. Convolutional neural accelerator (1) according to the preceding claim, wherein said router comprises a parallel routing control block (120), a parallel arbiter block (121), a parallel switch block (122) and a parallel input buffer block (123), the router being able to receive and process different data communication requests in parallel.
8. Convolutional neuron accelerator (1) according to claim 6 or 7, comprising a global control block (31), a calculation control block (32) and a communication control block (33), the communication control is carried out independently of the calculation control, the calculation controller (32) making it possible to control the calculation operations carried out by the unit calculation elements, as well as the reading and writing operations of the associated local memories, the communication controller (33) managing the data transfers between a global memory (3) and the local memories (13), as well as the data transfers between the processing blocks (10).
9. Convolutional neural accelerator (1) according to one of claims 6 to 8, in which a unit block (10) is adapted to carry out a transmission of type selected between broadcast and multicast according to a header of the packet to be transmitted and the unit block (10) is adapted to apply at least one of said rules: for a packet to be transmitted in broadcast originating from a neighboring block 10 located in a given direction relative to said block having to carry out the transmission, said block transmits the packet during a cycle in all directions except that of said neighboring block; for a packet to be transmitted in multicast: if the packet comes from the PE of the unitary block, the multicast implemented by the block is bidirectional in two opposite directions, otherwise the multicast implemented by the block is unidirectional, directed opposite the neighboring processing block from which said packet comes.
10. Convolutional neural accelerator (1) according to one of claims 6 to 9, wherein in the event of at least two simultaneous transmission requests towards the same direction by a unit block (10) during a processing cycle, the routing block of the unit block is adapted to arbitrate the priority between said requests, the priority arbitrated request then being transmitted in said direction and the other request being stored, then transmitted towards said direction at a subsequent processing cycle.
