Method and device for efficiently operating a neural network with convolutions

Abstract

Method for the parallelized calculation of two convolutions of a filter (2) with a first and second receptive field of the filter on input data, wherein the first and second receptive field correspond to a filter shifted by one step on the input data, comprising: initializing a first and second output variable (o 000 ,O 010 ) each with an initial value (b0). Executing a loop for each row r of the filter (2), where the following steps are executed in the loop: Loading a first kernel element (k r0c ) of the filter (2) of row r and the corresponding data values ​​(d r0c , d r1c ) to the first kernel element (k r0c ) of the first and second receptive fields of the input data. Calculate the value of the first output variable (o 000 ) by taking a product of d roc and k r0c to a current value of the first output variable (o 000) and calculating the value of the second output variable (o 010 ) by taking a product of d r1c and k r0c to a current value of the second output variable (o 010 ) is added. Loading the next kernel element (k r1c ) of the filter (2) of line r and recalculating the value of the first output variable (o 000 ) by purchasing a product from d r1c and k r1c to the current value of the first output variable (o 000 ) is added.

Description

[0001] The invention relates to a method for parallelized calculation of two convolution operations of a filter, a device, a computer program and a machine-readable storage medium. State of the art

[0002] Convolutions, or convolutions, are one of the most important layers for neural networks. In convolutional neural networks (CNNs), these layers typically account for over 90% of the total time required to compute an inference. Efficient implementations of these layers are therefore very important and thus a goal of this invention, in particular an implementation of certain convolutions frequently found in modern CNNs that increases data reuse and thus allows for more efficient computation on microcontrollers.

[0003] For more powerful architectures such as GPUs and CPUs, Winograd implementations are often used, which reduce the amount of multiplication but use a large amount of memory. Therefore, these implementations cannot be used for hardware platforms with limited memory. Instead, a partial Im2Col implementation is used. See, for example, K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for document processing. Tenth International Workshop on Frontiers in Handwriting Recognition, Université de Rennes 1, Oct 2006, La Baule (France), 2006. Advantages of the invention

[0004] The current invention addresses a shortcoming of the Im2Col implementation in certain convolution configurations that are very commonly used in modern CNNs. Given that the dilation is 1 and the stride is smaller than the kernel size, consecutive rows of the Im2Col buffer contain duplicate data. At the same time, the data required to compute an output pixel is already stored in memory, row by row. This can now be exploited to compute two convolutional output pixels without double-loading data, thus saving memory bandwidth. Disclosure of the invention

[0005] In a first aspect, the invention relates to a computer-implemented method for the parallelized calculation of two convolutions of a filter with a first and second receptive field of the filter on input data on a data processing system, in particular a microcontroller, with limited computing resources. The filter and the input data are at least two-dimensional (height, width, and a number of input channels). A receptive field of a filter in a mathematical convolution refers to the region of the input data to which the filter is applied. It is the region in which the filter applies the weights to generate the output. The receptive field can be considered a kind of "field of view" of the filter. The receptive field is determined by the size of the filter and the step size (stride).The filter size specifies how many input values ​​the filter considers simultaneously, while the step size specifies how far the filter moves over the input data with each step. Limiting computer resources can relate to the following aspects: CPU, specifically how many calculations and operations can be performed, and memory / space or bandwidth.

[0006] The procedure begins by initializing a first and second output variable (o 000, O 010 ) each with an initial value (b0), where the output variables (o 000 ,O 010 ) each contain a result of the convolution of the filter with the first and second receptive fields. The initial value (b0) is usually a trained parameter of the neural network.

[0007] This is followed by executing a loop for each row r with a value range from 0 to the maximum number of filter coefficients of the filter along the dimension of its rows minus the value 1. This means that the loop is executed over each row of the filter's kernel matrix, performing the following steps within the loop: Loading a first kernel element k r0c of the filter of row r and the corresponding data values ​​d r0c , d r1c to the first kernel element (k r0c ) of the first and second receptive fields of the input data. The corresponding data values ​​d r0c , d r1c to the first kernel element (k r0c ) are those data points of the input data on which the first kernel element (k r0c ) should be applied according to the first and second receptive fields.

[0008] Calculate the value of the first output variable (o 000 ) by taking a product of droc and k r0c to a current value of the first output variable (o 000 ) and calculating the value of the second output variable (o 010 ) by taking a product of d r1c and k r0c to a current value of the second output variable (o 010 ) is added.

[0009] Loading the next kernel element (k r1c ) of the filter from the corresponding line of the current loop index r. The next kernel element k r1c is the first kernel element k r0c immediately following kernel element of the filter.

[0010] Calculate the value of the first output variable (o 000 ) by taking a product of d r1c and k r1c to the current value of the first output variable (o 000 ) is added.

[0011] Loading additional data values ​​d r2c and d r3c . The other data values ​​d r2c and d r3care the already loaded data values ​​d r0c , d r1c following data values ​​d r2c and d r3c of the first and second receptive field, in particular the data values ​​for which the next convolution operation with the two subsequent kernel elements k r1c and k r2c should be executed.

[0012] Calculate the value of the second output variable (o 010 ) by taking the product of d r2c and k r1c to the current value of the second output variable (o 010 ) is added.

[0013] Loading the next kernel element k r2c .

[0014] Calculate the value of the first output variable (o 000 ) by taking the product of d r2c and k r2c to the current value of the first output variable (o 000 ) and calculating the value of the second output variable (o 010 ) by taking the product of d r3c and kr2c to the current value of the second output variable (o 010 ) is added.

[0015] It should be noted that the method was explained using a filter with three kernel elements per row as an example. For a filter with two or more kernel elements per row, the method can be adapted accordingly by alternately loading (and calculating) more or fewer kernel elements and corresponding data values.

[0016] Optionally, the second (and / or first) output variable (o 000 ,O 010 ) as convolution results for the two convolutions.

[0017] After the procedure has been executed, it can be executed again for the next receptive field of the filter. Specifically, the procedure is executed until the filter convolution has been performed on the entire input data. That is, until the entire input data has been effectively covered by the receptive field.

[0018] It is proposed that the filter and the input data have a plurality of channels, and that a further loop be executed for each channel c via the loop over the rows r. The filter and the input data can thus each be given as a tensor, preferably with the number of channels matching. If the number of channels does not match, the inventive method can be repeated several times along the dimension of the channels until the receptive field has effectively covered the entire tensor of the input data.

[0019] Furthermore, it is proposed that the kernel elements are arranged according to an order k 000 ; k 010 ; k 020 ; :k 100 ; k 110 ; k 120 ; : are stored. This means that the kernel elements were read line by line, and the lines were assembled one after the other into the specified sequence. This has the advantage that no pointer is required for the sequence of filter elements, and only two pointers are needed for efficient traversal of the input data.

[0020] In further aspects, the invention relates to a device and a computer program, each of which is configured to carry out the above methods, and to a machine-readable storage medium on which this computer program is stored.

[0021] Embodiments of the invention are explained in more detail below with reference to the accompanying drawings. In the drawings: Fig. Figure 1 shows a schematic representation of a partial Im2Col algorithm for a single output pixel for a convolution with (3x3) kernel without dilation with stride 1; Fig. Figure 2 shows a schematic representation of the partial Im2Col algorithm for two adjacent output pixels in a convolution with a (3x3) kernel without dilation with stride 1; Fig. 3 schematically illustrates the nested implementation of a (3x3) convolution with stride 1; Fig. 4 schematically shows a pseudocode of an embodiment of the invention.

[0022] Partial Im2Col implementations are known from the state of the art. For example, 2D convolutions are implemented in this way in the CMSIS-NN library (L. Lai, N. Suda, and V. Chandra. Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus) from ARM.

[0023] Fig. Figure 1 illustrates the procedure for calculating an output pixel (10) in a convolution (11) with a kernel size (3x3) and stride 1 and dilation 1 according to the Im2Col implementation. The input data (1a, 1d), each preferably present as a 2D input data item, such as a matrix, are to be convolved with one or more filters (2).

[0024] The data required for a pixel in the output channels is copied sequentially from the input data to another memory area (12). The actual convolution is then a simple dot product of the values ​​in the kernel with those from the input data. The kernel is already in the correct order. To reuse data, especially kernel elements, the data for multiple pixels is typically loaded from the input data. Kernel elements for multiple output channels are also loaded (13).

[0025] How many pixels are loaded in parallel is a trade-off between the required RAM and the savings in duplicate loading operations for kernel elements and input data. Furthermore, the amount of data that can be loaded simultaneously is limited by the available registers. For each output pixel to be calculated, an accumulator variable is required in addition to the loaded data, so the register requirement increases very quickly. CMSIS-NN, for example, loads the data for two pixels and two output channels and then only needs to load the kernel data once for these two pixels. This requires four accumulator variables. This number is adapted to the 13 registers of the Cortex-M architecture.

[0026] The advantage of using Im2Col is that for any kernel size and any value for stride and dilation, the convolution is reduced to a matrix multiplication, which is also very efficient on CPUs compared to an implementation with nested loops that has to load the input data multiple times.

[0027] In the following, we will address a shortcoming of the Im2Col implementation for certain convolution configurations that are very commonly used in modern CNNs. Under the assumption that the dilation is 1 and the stride is smaller than the kernel size, consecutive rows of the Im2Col buffer contain duplicate data, see Fig. 2. That is, if the receptive field of the filter (2) after completion of convolution for the next convolution by one step, in Fig. 2 is shifted one position horizontally to the right, then some of the data from the input data point that was already in the receptive field of the filter in the previous step is reloaded into the memory area (12). Since the receptive field has shifted one step to the right, in the next step only the three superimposed data points to the right of the previous data points in the memory area (12) are really new. The schematic first and second row of the memory area (12) thus contain 66% of the data duplicated, without this being exploited in the Im2Col implementation. Instead, this data is loaded multiple times from its original position in the memory into the memory area (12). At the same time, the data required for calculating an output pixel is already located in the memory, row by row, see Fig. 3.

[0028] The index notation d ijk in Fig. 3 denotes the i-th row, the j-th column, and the k-th input channel. This can now be exploited to calculate two output pixels of the convolution without double-loading data and thus saving memory bandwidth. The marked data on the left is used to calculate output pixels 0 and 1. For each output pixel, the data is already stored in memory, row by row. For example, for output pixel o 000 the data d 000 to d c03 in the first line.

[0029] Fig. Figure 4 shows an example of a calculation rule in the form of a pseudo-algorithm to solve this problem. The structure of this calculation takes into account the properties of microcontrollers: Data is only loaded when needed for the next calculation. This takes into account the fact that small microcontrollers have only a few registers in which variables are stored on the processor. Each kernel element is loaded only once for two output pixels. Each data value is loaded only once for two output pixels. The specific implementation can be adapted to the Cortex-M architecture. For other architectures with more registers, three or more pixels can be calculated in parallel, which further increases data reuse.

[0030] This pseudocode represents a procedure called "Interleaved Convolution 3x3, Stride 1." It performs a convolution operation on a 3x3 kernel with a stride of 1. This is an example implementation; for example, the filter dimension, etc., can be changed arbitrarily.

[0031] The procedure according to Fig. 4 begins with an initialization of output variables o 000 and o 010 with the value b0, which is, for example, zero.

[0032] This is followed by starting a loop for each input channel c from 0 up to the total number of input channels. If only one channel is present, this loop can be abandoned.

[0033] This is followed by a nested loop for each row r from 0 to 2, or up to the maximum number of filter coefficients of the filter (2) minus the value 1. The following steps are executed in the nested loop: Loading the data values ​​d r0c , d r1c and the kernel element k r0c . Calculating the value of o 000 by a product of d roc and k r0c to a current value of o 000 is added, and calculating the value of o 010 by a product of d r1c and k r0cto a current value of o 010 is added. Loading the next kernel element k r1c . Calculating the value of o 000 by purchasing a product from d r1c and k r1c to the current value o 000 is added. Loading data values ​​d r2c and d r3c . Calculating the value of o 010 by taking the product of d r2c and k r1c to the current value o 010 is added. Loading the next kernel element k r2c . Calculating the value of o 000 by taking the product of d r2c and k r2c to the current value of o 000 is added, and calculating the value of o 010 by taking the product of d r3c and k r2c to the current value of o 010 is added.

[0034] After the last step within the nested loop, either the nested loop is executed for the next higher value for r, or when r has reached its maximum value, the nested loop is terminated.

[0035] What has just been said regarding the termination of the nested loop applies analogously to the outer loop concerning channels c.

[0036] After the procedure according to Fig. 4 has been executed, it can be executed again for the next receptive fields of the filter.

[0037] The Fig. The embodiment described in Figure 4 does not access the input data sequentially. This does not result in any runtime disadvantages for microcontrollers. However, the machine code requires two pointers to efficiently navigate through the input data, which can limit the available registers. The same applies to the kernel elements. However, since the kernel elements are constant and already known at the time of code generation, they can be rearranged so that they are stored sequentially in memory. Instead of the old order k 000 ; k oo1 ;::: ; k 010 ; k 011 ; :::the data are stored in the order k 000 ; k 010 ; k 020 ; ::: k 100 ; k 110 ; k 120 ; ::: filed.

[0038] Microcontrollers with DSP units, such as the Cortex-M4 and Cortex-M7, can execute two MAC instructions in parallel for quantized networks and also load 4 bytes in parallel. For such architectures, the algorithm is Fig. 4 is modified so that two input channels are processed in parallel. Accordingly, the kernel reordering then also takes place with the elements from two channels one after the other, e.g., k 000 ; k 001 ; k 010 ; k 011 ; k 020 ; k 021 ; :::.

[0039] It should be noted that convolutions are also calculated in int8 networks with int16 variables to avoid overflows, ie two variables correspond to 4 bytes.

[0040] The implementation was described for a (3x3) kernel with stride 1 and dilation 1. Among the kernel configurations commonly used in modern CNNs, this is the configuration that benefits most. However, it can also be implemented analogously for (3x3) kernels with stride 2 and for (2x2) kernels with stride 1, as well as for exotic kernel configurations where data is reused. For each of the aforementioned cases, a different reordering of the kernel elements results. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] K. Chellapilla, S. Puri, and P. Simard. High performance convolutional neural networks for document processing. Tenth International Workshop on Frontiers in Handwriting Recognition, Université de Rennes 1, Oct 2006, La Baule (France), 2006

[0003] L. Lai, N. Suda, and V. Chandra. Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus

[0022]

Claims

[1] Method for the parallelized calculation of two convolutions of a filter (2) with its first and second receptive fields on input data, in particular on a data processing system with limited computer resources, wherein the first and second receptive fields correspond to one another to the filters (2) on the input data shifted by one step, comprising: Initializing a first and second output variable (o 000 ,O 010 ) each with an initial value (b0), where the output variables (o 000 ,O 010 ) each comprise a result of the convolution of the filter with its first and second receptive fields; Execute a loop for each row r of the filter (2), where the following steps are performed in the loop: Loading a first kernel element (k r0c ) of the filter (2) of row r and of corresponding data values ​​d r0c , d r1c to the first kernel element kr0c the first and second receptive fields of the input data; Calculate the value of the first output variable (o 000 ) by taking a product of d roc and k r0c to a current value of the first output variable (o 000 ) and calculating the value of the second output variable (o 010 ) by taking a product of d r1c and k r0c to a current value of the second output variable (o 010 ) is added; Loading a kernel element (k r0c ) subsequent kernel element (k r1c ) of the filter (2) of line r; Calculate the value of the first output variable (o 000 ) by taking a product of d r1c and k r1c to the current value of the first output variable (o 000 ) is added; Loading the data values ​​d r0c , d r1c following data values ​​d r2c and d r3cthe first and second receptive fields; Calculate the value of the second output variable (o 010 ) by taking the product of d r2c and k r1c to the current value of the second output variable (o 010 ) is added; Loading the next kernel element k r2c of the filter (2) of line r; Calculate the value of the first output variable (o 000 ) by taking the product of d r2c and k r2c to the current value of the first output variable (o 000 ) and calculating the value of the second output variable (o 010 ) by taking the product of d r3c and k r2c to the current value of the second output variable (o 010 ) is added. [2] The method of claim 1, wherein the filter and the input data comprise a plurality of channels and a further loop is executed for each channel c over the loop over the rows r. [3] Method according to claim 1 or 2, wherein the kernel elements were read out line by line and the read out lines are stored in sequence as a sequence, wherein the steps of loading the kernel elements are carried out from the sequence. [4] Method according to one of the preceding claims 2 or 3, wherein the data processing system can execute two MAC instructions simultaneously, wherein two loops over the lines r for two channels are executed simultaneously. [5] Method according to one of the preceding claims, wherein the filter (2) has at least one 2x2 or 3x3 kernel. [6] Method according to one of the preceding claims, wherein the filter is a convolutional neural network (CNN) filter. [7] Use of the method according to one of the preceding claims in operating the convolutional neural network. [8] Device which is arranged to carry out the method according to one of the preceding claims. [9] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claims 1 to 6. [10] A machine-readable storage medium on which the computer program according to claim 9 is stored.

Images