METHOD AND DEVICE FOR REDUCING THE COMPUTING LOAD OF A DATA PROCESSOR BY A NEURAL NETWORK WITH CONVOLUTION

Description

[0001] Implementation and realization methods of the invention relate to deep learning, in particular deep neural networks, and more particularly the reduction of the computational load of a computing unit, for example a microprocessor, processing an initial set of data by a convolutional neural network.

[0002] Neural networks are used extensively to solve various statistical problems, including the problem of data classification.

[0003] After a generally supervised machine learning phase, i.e. on a reference database of already classified data, a neural network "learns" and becomes capable of applying the same classification to unknown data on its own.

[0004] Convolutional Neural Networks (CNNs) are a type of neural network in which the connection pattern between neurons is inspired by the visual cortex of animals. They effectively enable the recognition of objects or people in images or videos. US 2018 / 232629 A1 (DU YUAN [US] ET AL) August 16, 2018 (2018-08-16) discloses a pooling operation method for a convolutional neural network.

[0005] A convolutional neural network typically contains four types of layers that process information successively: the convolution layer which processes blocks of the image one after the other; the non-linear layer which improves the relevance of the result; the pooling layer which allows several neurons to be grouped into a single neuron; the fully-connected layer (also called the dense layer) which links all the neurons of a layer to all the neurons of the previous layer.

[0006] Each layer takes data as input and delivers output data ("features" in English) after processing by said layer.

[0007] The convolutional layer typically corresponds to a set of dot products between the outputs of the previous layer and the weights of that layer. It usually constitutes the first layer of a convolutional neural network.

[0008] By "weights", which is a term whose meaning in the field of neural networks is well known to those skilled in the art, we mean configurable neuron parameters to obtain good output data.

[0009] The purpose of the convolutional layer is to identify the presence of a specific set of data within the input images. This is achieved through convolutional filtering. The principle is to drag a window representing the data to be detected and calculate the convolution product between the window and each scanned portion of the image.

[0010] The filtering window is known to those skilled in the art as the "convolution kernel" and in matrix form, for example, a 3x3 weight kernel.

[0011] For each convolution product, we obtain output data indicating where the specific data is located on the image.

[0012] The pooling layer is often placed between two convolutional layers. It receives the output data from the convolutional layer as input, and then performs a "pooling" operation to reduce the data size while preserving its important characteristics. This reduces the number of parameters and calculations in the network, thereby improving its efficiency.

[0013] The most common form is a pooling layer with filtering windows in matrix form, for example a 2x2 matrix.

[0014] The fully connected layer is the final layer of a convolutional or non-convolutional neural network. It allows for the classification of the neural network's input data.

[0015] Typical neural network architectures stack a few pairs of convolution and nonlinear layers, then add a pooling layer and repeat this pattern until sufficiently small output data is obtained, and then end with one or more fully connected layers.

[0016] Today, neural networks are increasingly complex and require significant computing power.

[0017] Instructions per second (Instructions Per Second) is a unit of measurement for the performance of a system, for example, a microprocessor.

[0018] The more complex the neural network configuration, the lower the number of instructions executed by the system per second.

[0019] Therefore, there is a need to reduce the computational load, for example, of a microprocessor processing data received or created by a convolutional neural network.

[0020] According to one aspect, a method is proposed for reducing the computational load of a computing unit, for example a microprocessor, processing an initial dataset by a convolutional neural network, said initial dataset being stored in an initial memory in first and second orthogonal directions, said network comprising at least one convolutional layer followed by at least one pooling layer, said processing comprising a first filtering of the initial dataset by said at least one convolutional layer using a first sliding window in the first direction, each slide of said first window producing a first dataset, and a second filtering of the first datasets by said at least one pooling layer using a second sliding window in the second direction.

[0021] The combination of these two filters contributes to reducing the computational load on the processing unit.

[0022] “By initial memory” we mean for example volatile memory, for example RAM (for “Random Access Memory”).

[0023] The first filtering window here is a window performing convolutional filtering. This first filtering window is characteristic of the convolutional layer.

[0024] The second filtering window allows the "pooling" operation to be applied to the output data obtained by the first filtering window. This second filtering window is characteristic of the pooling layer. The second sliding window has a dimension N counted along the first direction, the first filtering comprises N-1 slides of the first sliding window along the first direction, and the first storage of the first N sets of data in a first temporary memory.

[0025] By "first temporary memory" we mean, for example, a buffer memory (commonly referred to by the English term "buffer").

[0026] Each slide of the first filter window yields a first set of output data. Each first set is stored on a line of the first temporary memory, which comprises N lines, where N represents the height of the second sliding window.

[0027] According to one implementation mode, the initial dataset is stored in matrix form by rows and columns, the succession of rows extending in the first direction, for example the vertical direction, and the succession of columns extending in the second direction, for example the horizontal direction, with each individual row extending in the second direction and each individual column extending in the first direction.

[0028] In one implementation method, the first sliding window moves in steps along the first direction, traversing the corresponding line at each step. The step size can be arbitrary. Thus, the first sliding window can move line by line or skip lines.

[0029] According to one embodiment, the second filtering includes slides of said second sliding window, in the second direction, on said first temporary memory storing the first N data sets, each slide producing a second data set, said second filtering including a second storage of M second data sets in a second temporary memory.

[0030] By "second temporary memory" we mean, for example, a buffer memory (commonly referred to by the English term "buffer").

[0031] In other words, each slide of the second sliding window onto said first temporary memory containing N initial data sets, produces a second data set.

[0032] Each second dataset represents the output data from said pooling layer.

[0033] The first storage includes a circular filling of said first temporary memory by the first set of data produced at each slide of the first sliding window.

[0034] The oldest first dataset is replaced by the most recent first dataset.

[0035] According to one implementation method, the second filtering is performed after a number of circular fills of said first temporary memory.

[0036] For example, we can perform the second filtering after two successive slides of the first sliding window which has just produced two new first sets of data.

[0037] According to one implementation method, the first filtering and the second filtering follow one another until all the data in the initial set has been processed.

[0038] According to one implementation mode, said neural network includes at least one non-linear layer between said at least one convolution layer and said at least one pooling layer.

[0039] According to one implementation method, the neural network is defined by parameters implemented in floating point or fixed point.

[0040] By "floating point", we mean a real number represented by a sign, a mantissa and an exponent.

[0041] By "fixed point", we mean a number that has a fixed number of bits after the decimal point.

[0042] By "parameters" we mean, for example, the weights of the neural network and the data defining each layer of the network.

[0043] According to another aspect, a device is proposed comprising an initial memory intended to store an initial set of data in first and second orthogonal directions, and a first memory intended to store parameters defining a convolutional neural network configured to process the initial set of data, said network comprising at least one convolutional layer followed by at least one pooling layer, and a computing unit, for example a microprocessor, configured to perform the operations of the different layers of the neural network, said at least one convolutional layer being configured to perform a first filtering of the initial set of data using a first sliding window in the first direction so as to produce a first set of data at each slide,and said at least one pooling layer being configured to perform a second filtering of the first datasets using a second sliding window in the second direction.

[0044] The second sliding window has a dimension N counted along the first direction, and wherein the device includes processing means configured to limit the slides of the first sliding window to N-1 slides along the first direction, said device further comprising a first temporary memory configured to store the first N sets of data.

[0045] According to one embodiment, the initial data set is stored in matrix form by rows and columns, the succession of rows extending in the first direction, and the succession of columns extending in the second direction, each individual row extending in the second direction and each individual column extending in the first direction.

[0046] According to one embodiment, the processing means are configured to slide the first sliding window step by step along the first direction and to, at each step, traverse the line corresponding to the first sliding window.

[0047] According to one embodiment, said processing means are configured to slide the second window in the second direction over the first temporary memory storing the first N data sets, so as to produce a second data set at each slide, said device also comprising a second temporary memory configured to store M second data sets.

[0048] The processing means are configured to perform a circular filling of the first temporary memory with the first sets of data produced during the slides of the first sliding window.

[0049] According to one embodiment, the processing means are configured to perform the second filtering once the first memory has performed a number of circular fills.

[0050] According to one embodiment, the processing means are configured to successively perform the first and second filterings until all the data in the initial set has been processed.

[0051] According to one embodiment, the neural network includes at least one non-linear layer between said at least one convolution layer and said at least one pooling layer.

[0052] According to one embodiment, the parameters of the neural network are implemented in floating point or fixed precision.

[0053] A microcontroller including said device is also offered.

[0054] Other advantages and features of the invention will become apparent upon reading the detailed description of the implementation and execution methods, which are by no means exhaustive, and the accompanying drawings in which: [ Fig. 1 ] schematically illustrates one embodiment of the invention, [ Fig. 2 ] schematically illustrates a method of implementing and realizing the invention, [ Fig. 3 ] schematically illustrates a method of implementing and realizing the invention, [ Fig. 4A ] schematically illustrates a method of implementing and realizing the invention, [ Fig. 4B ] schematically illustrates a method of implementing and realizing the invention, and [ Fig. 4C ] schematically illustrates a method of implementation and realization of the invention.

[0055] On the figure 1 The MC reference designates an electronic object, for example a microcontroller, a code generator, or any other object that can contain an embedded software or hardware architecture.

[0056] The MC object includes a device MO, for example a module including an initial memory MV, for example a volatile RAM (for "Random Access Memory") configured to store an initial set of data DI.

[0057] The MO device also includes a non-volatile memory MNV, for example a ROM (for "Read-Only Memory") intended to store the parameters defining a convolutional neural network RN.

[0058] By "parameters" we mean the weights of the neural network and the data defining each layer of the neural network.

[0059] These parameters can be implemented in fixed-point or floating-point format.

[0060] By "floating point", we mean a real number represented by a sign, a mantissa and an exponent.

[0061] By "fixed point", we mean a number that has a fixed number of bits after the decimal point.

[0062] This configuration allows for specific processing of DI data stored in the initial MV memory and the delivery of output data.

[0063] To do this, the MO device includes a computing unit CU, for example a microprocessor configured to perform the operations defined by each layer of the neural network RN.

[0064] The processing unit (PU) is therefore coupled to the non-volatile memory (NVM) to extract the parameters from each layer of the neural network (NN). It is also coupled to the volatile memory (VM) to extract the initial set of data (ID) on which the processing will be performed.

[0065] The MO device includes a first temporary memory MEM1 and a second temporary memory MEM2.

[0066] By "temporary" we mean a buffer memory (commonly referred to by the English term "buffer").

[0067] The first and second temporary memories MEM1 and MEM2 are coupled to the processing unit UC.

[0068] The first and second temporary memories MEM1 and MEM2 are configured to store data delivered by the computing unit CU.

[0069] The MO device includes MT processing means coupled to the computing unit UC and configured to control the number of operations to be performed for each layer and consequently, control the storage of data delivered by the computing unit UC.

[0070] The processing methods can be implemented, for example, by a software module within the microprocessor and / or by specific logic circuits.

[0071] There figure 2 schematically illustrates the different processing stages of the initial dataset DI by the layers of the neural network RN.

[0072] The neural network RN includes at least one CV convolution layer followed by at least one CMC pooling layer.

[0073] The CV convolution layer allows the presence of a specific dataset within the initial DI dataset received in step V1 to be identified. To do this, the convolution layer performs convolutional filtering in step V2 using a first sliding window F1, as illustrated in the... figures 4A , 4B And 4C .

[0074] The first sliding window F1 allows the convolution product between the window and each portion of the initial data set DI to be calculated.

[0075] The initial dataset DI is presented as a matrix with rows and columns, the succession of rows extending in a first direction, i.e. vertically, and the succession of columns extending in a second direction, i.e. horizontally.

[0076] The first sliding window F1 slides along the first direction, and therefore vertically, while with each slide it traverses a line of the initial data set DI. Each slide produces a first data set D1 in step V3.

[0077] By "gliding" we mean here that the first sliding window F1 traverses each line of the initial data set DI.

[0078] Each first set of data D1 is stored in the first temporary memory MEM1.

[0079] The neural network can include a non-linear layer (NLL), which improves the relevance of the result. It takes an initial dataset D1 as input in step V4 and produces an intermediate dataset D11 in step V5.

[0080] Intermediate D11 data sets can be placed in intermediate temporary memory.

[0081] Since the non-linear CNL layer is optional, the first D1 datasets can be sent directly to the CMC pooling layer.

[0082] The CMC pooling layer therefore receives the first D1 datasets to which a "pooling" operation is applied in step V6, which consists of reducing the size of the D1 datasets while preserving their important characteristics.

[0083] For this purpose, the CMC pooling layer uses a second sliding window F2, illustrated in the figures 4A , 4B And 4C .

[0084] The second sliding window F2 slides along the second direction and produces a first data set D2 in step V7.

[0085] Each second data set D2 is stored in the second temporary memory MEM2.

[0086] There figure 3 schematically illustrates an algorithm implemented in MT processing means.

[0087] This algorithm allows the MT processing means to control the number of operations to be performed for each layer and consequently, to control the storage of the data delivered by the computing unit CU.

[0088] When the MT processing means detect the presence of a CV convolution layer in step S1, they initialize in step S2 a variable i to 0 and a variable N1 also to 0.

[0089] The variable i represents an index traversing the list of rows generated as output from the convolution layer.

[0090] The variable N1 represents the number of first sets D1 stored in the first temporary memory MEM1.

[0091] The first and second temporary memories MEM1 and MEM2 are empty at stage S2.

[0092] In step S3, the first sliding window F1 slides along the first direction, and therefore vertically, traversing a line of the initial dataset DI with each slide. Each slide produces a first dataset D1. This first dataset D1 can be sent to the nonlinearity layer CNL in step S4, which produces an intermediate dataset D11. This step S4 is optional.

[0093] In step S5, we increment the value of the variable N1 by 1 because we have just stored in the temporary memory MEM1 a first set of data D1.

[0094] In step S6, we carry out a first comparison between the variable N1 and a third variable N representing the dimension of the second sliding window F2 and the maximum number of first data sets D1 that the first temporary memory MEM1 can contain.

[0095] If the value of the variable N1 is equal to the value N, this means that the first temporary memory MEM1 contains the first N data sets D1. It can therefore no longer contain any more.

[0096] If the value of the variable N1 is less than the value N, this means that the first temporary memory MEM1 can store more first sets of data D1.

[0097] Since in this example we have only stored a first set of data D1, the value of the variable N1 is less than the value N and therefore we move to step S7 in which we increment the value of the variable i by 1.

[0098] In step S8, a second comparison is made between the value of the variable i and a variable Max representing the maximum number of lines generated at the output of the convolution layer.

[0099] If the value of variable i is greater than the value of variable Max, this means that all generated rows have been processed. The MT processing method then proceeds to the next CV convolution layer if any remain in step S9.

[0100] Otherwise, we repeat steps S3, S4 and S5 until the value of variable N1 is equal to the value of variable N.

[0101] We then proceed to step S10 in which the CMC pooling layer uses the second filtering window F2 in the second direction on the first N data sets D1 contained in the first temporary memory MEM1 and produces a second data set D2.

[0102] In step S11, a defined number N2 is subtracted from the value of the variable N1, which represents the number of slides of the first window F1 to be made before performing a second filtering.

[0103] The value of the variable N1 is equal to N1-N2.

[0104] In step S12, we increment the value of the variable i by 1 and we check in step S13 whether the value of the variable i is less than the value of the variable Max.

[0105] If not, we proceed to step S14 to the next CV convolution layer which will use another first filtering window F1 and will scan the initial DI data set.

[0106] If so, we continue to filter said initial data set D1 with the first filtering window N2 times, that is, until N1 equals N.

[0107] The first N2 data sets D1 overwrite the oldest first data sets D1 in the first temporary memory MEM1. The processing unit UC therefore performs a circular filling.

[0108] The second sliding window F2 performs a slide on said first temporary memory MEM1 comprising the first two new sets D1 and produces a second set D2.

[0109] This algorithm is repeated until all rows of the initial dataset DI have been processed.

[0110] THE figures 4A , 4B And 4C schematically illustrate the implementation of said algorithm.

[0111] In the figure 4A The first sliding window F1 slides once over the first line of the initial data set DI and produces a first data set D1 which will be stored in the first temporary memory MEM1. N1 is now equal to 1.

[0112] The first window F1 slides a second time onto the second line of the initial data set DI and produces a first data set D1 which will be stored in the first temporary memory MEM1. N2 is now equal to 2.

[0113] We repeat the sliding until N1 is equal to N, here 5. N represents the height of the second sliding window.

[0114] The first temporary memory therefore contains the first N data sets D1.

[0115] The second sliding window F2 slides along the second direction on the first temporary memory MEM1 and produces a second set of data D2 which will be stored in the second temporary memory MEM2.

[0116] We subtract N2 from N1, which is equal to 2 here. Therefore, N1 is equal to 3.

[0117] The first sliding window F1 slides a sixth time on the sixth line of the initial data set DI and produces a first data set D1. N1 equals 4.

[0118] The first dataset D1 overwrites the oldest first dataset D1.

[0119] The first window F1 slides a seventh time on the seventh line of the initial data set DI and produces a first data set D1. N1 equals 5.

[0120] The first dataset D1 overwrites the oldest first dataset D1.

[0121] It should be noted that the first window F1 may, for example, not slide over each line of the initial data set DI. The first window F1 may have a sliding step (or "stride").

[0122] In the figure 4B, the second sliding window F2 slides over the first temporary memory MEM1 containing the first two new data sets D1 and produces a second data set D2 which will be stored in the second temporary memory MEM2.

[0123] We subtract N2 from N1, which is equal to 2 here. Therefore, N1 is equal to 3.

[0124] We repeat the same steps in Figure 3B and 3C until there are no more rows to process in said initial dataset DI.

Claims

1. A method for reducing the computational load of a computing unit (UC) intended to process an initial dataset (DI) by a convolutional neural network (RN), the initial dataset (DI) being stored in an initial memory (MV) along a first and a second orthogonal directions, said network (RN) comprising at least one convolutional layer (CV) followed by at least one pooling layer (CMC), said processing comprising a first filtration of the initial dataset (DI) by the at least one convolutional layer (CV) using a first sliding window (F1) along the first direction, each sliding of the said first window (F1) producing a first dataset (D1), and a second filtration of the first datasets (D1) by at least one pooling layer (CMC) using a second sliding window (F2) along the second direction, wherein the second sliding window (F2) has a dimension N counted along the first direction, the first filtration comprising N-1 slidings of the first sliding window (F1) along the first direction, and a first storage of N first datasets (D1) in a first temporary memory (MEM1), and wherein said first storage includes a circular filling of the said first temporary memory (MEM1) by the first dataset (D1) produced at each sliding of the first sliding window (F1).

2. The method according to claim 1, wherein the initial dataset (DI) is stored in a matrix format with rows and columns, the succession of rows extending along the first direction, and the succession of columns extending along the second direction, each individual row extending along the second direction and each individual column extending along the first direction.

3. The method according to claim 1 or 2, wherein the second filtration comprises slidings of said second sliding window (F2) along the second direction, on said first temporary memory (MEM1) storing the first N datasets (D1), each sliding producing a second dataset (D2), said second filtration including a second storage of M second datasets (D2) in a second temporary memory (MEM2).

4. The method according to claim 2 or claims 2 and 3, wherein the first sliding window slides by steps along the first direction and, at each step, traverses the corresponding line.

5. The method of claim 4, wherein the second filtration is performed after a number (N2) of circular fillings of said first memory (MEM1).

6. The method according to one of the preceding claims, wherein the first filtration and the second filtration succeed each other until all the data from the initial set (DI) have been processed.

7. The method according to one of the preceding claims, wherein the neural network (RN) comprises at least one non-linear layer (CNL) between said at least one convolutional layer (CV) and said at least one pooling layer (CMC).

8. The method according to one of the preceding claims, wherein said neural network (RN) is defined by parameters implemented in floating point or fixed precision.

9. A device (MO) comprising an initial memory (MV) intended to store an initial dataset (DI) along a first and a second orthogonal directions, and a first memory intended to store parameters defining a convolutional neural network (RN) configured to process the initial dataset (DI), said network (RN) comprising at least one convolutional layer (CV) followed by at least one pooling layer (CMC), and a computing unit (UC) configured to perform the operations of the different layers of the neural network, said at least one convolutional layer (CV) being configured to perform a first filtration of the initial dataset (DI) using a first sliding window (F1) along the first direction to produce a first dataset (D1) at each sliding, and said at least one pooling layer (CMC) being configured to perform a second filtration of the first datasets (D1) using a second sliding window (F2) along the second direction, wherein the second sliding window (F2) has a dimension N counted along the first direction, the device comprising processing means (MT) configured to limit slidings of the first sliding window to N-1 slidings along the first direction, said device (MO) further comprising a first temporary memory (MEM1) configured to store N initial datasets (D1), and wherein the processing means (MT) are configured to perform a circular filling of the first temporary memory (MEM1) with the first datasets (D1) produced during the slidings of the first sliding window (F1).

10. The device (MO) according to claim 9, wherein the initial dataset (DI) is stored in a matrix format according to rows and columns, the succession of rows extending along the first direction and the succession of columns extending along the second direction, each individual row extending along the second direction and each individual column extending along the first direction.

11. The device (MO) according to claim 10, wherein the processing means (MT) are configured to slide the second window (F2) along the second direction over the first temporary memory (MEM1) storing the first N datasets (D1), so as to produce a second dataset (D2) at each sliding, said device (MO) also comprising a second temporary memory (MEM2) configured to store M second datasets (D2).

12. The device according to any one of claims 10 and 11, wherein the processing means are configured to slide the first sliding window in steps along the first direction and, at each step, to traverse the row corresponding to the first sliding window.

13. The device (MO) according to claim 12, wherein the processing means (MT) are configured to perform the second filtration once the first memory (MEM1) has completed a number (N2) of circular fillings.

14. The device (MO) according to one of claims 9 to 13, wherein the processing means (MC) are configured to successively carry out the first and second filtrations until all the data from the initial set (DI) have been processed.

15. The device (MO) according to one of claims 9 to 14, wherein the neural network (RN) comprises at least one non-linear layer (CNL) between said at least one convolutional layer (CV) and said at least one pooling layer (CMC).

16. The device (MO) according to one of claims 9 to 15, wherein the parameters of the neural network (RN) are implemented in floating point or fixed precision.

17. A microcontroller (MC) comprising a device (MO) according to one claims 9 to 16.