Padding methods for convolutional neural networks

Abstract

This invention relates to an improved padding method for convolutional neural networks (particularly for concentrically constructed data). A padding method for a convolutional neural network includes the following steps: receiving concentrically constructed data of an object, wherein the concentrically constructed data is associated with an image concentrically located on the object; unfolding the concentrically constructed data into a data array, the data array including ground truth data on opposite sides of the data array; performing a convolution operation using circular padding, wherein, in the case of circular padding, the ground truth data on the opposite side of the data array is filled using the ground truth data from one side of the data array, and / or, and / or vice versa.

Description

Technical Field

[0001] The invention relates to a padding method for a convolutional neural network, a device, a computer program and a machine-readable storage medium. Background Art

[0002] Convolutional neural networks, also known as folded neural networks (faltende neuronale Netzwerk), are considered to be one of the most efficient tools for image processing or video processing.

[0003] When controlling at least partially autonomous robots (especially at least partially automated vehicles), image data and other physical observations of the robot's surroundings are important sources of information. Artificial neural networks (e.g., convolutional neural networks) are used to classify which objects are present in the robot's surroundings from these very high-dimensional data.

[0004] Modern convolutional neural networks have different layers (so-called convolutional layers). An output data array is generated from an input data array (e.g. image data) of a convolutional layer by discrete convolutions with the aid of filter kernels (Filterkernels) which essentially correspond to the convolution matrix. Since the filter kernels themselves usually have multiple dimensions, the output data array has a lower dimension than the input data array. Thus, when multiple convolutional layers are applied to the input data array, its dimensionality is continuously reduced.

[0005] Therefore, such convolutional neural networks usually have a so-called padding mechanism in order to ensure that the input data array and the output data array of the convolutional neural network are numerically compatible, ie, in particular have the same dimension.

[0006] For this purpose, firstly so-called zero filling is known, in which the input data array is expanded at the edge by exactly the data dimension that is reduced when using the filter kernel. The data used for this purpose are simple zeros.

[0007] However, if the input data array maps annular surrounding information (for example in the case of an at least partially autonomous robot), the known filling methods do not offer the possibility to specifically cope with this type of input data.

[0008] There is therefore a desire to provide improved filling methods for convolutional neural networks, in particular for concentrically structured data. Summary of the invention

[0009] According to one aspect, a padding method for a convolutional neural network comprises the following steps:

[0010] receiving data of a concentric structure of an object, wherein the data of the concentric structure are associated with an image recorded concentrically to the object;

[0011] The concentrically structured data are expanded into a data array that includes the true correlations on its opposite sides. data;

[0012] A convolution operation is performed using circular filling, wherein, in the case of circular filling, real relevant data on one side of the data array is used to fill real relevant data on a side of the data array opposite to the side, and / or real relevant data on the side of the data array opposite to the side is used to fill real relevant data on the side of the data array.

[0013] The term "concentrically structured data" includes data detected concentrically around an object. Preferably, such data is also referred to as annular data or annular data.

[0014] The term “convolutional neural network” is preferably also described by the term “convolutional neural network”.

[0015] The term “reality-relevant data” includes data (in particular surroundings data) which are arranged alongside one another in reality.

[0016] The term "convolutional layer" is also described by the term "convolutional layer". A convolutional neural network has different convolutional layers. Each convolutional layer comprises a data array, to which a convolution operation is applied by means of a filter kernel. The output data of the convolution operation determine another data array, preferably another convolutional layer. Thus, the original data array can be processed by a plurality of successive convolutional layers. Preferably, the unprocessed data array of the convolutional layer is referred to as the input data array, and the data array generated by the convolution operation is referred to as the output data array. The output data array of a convolutional layer preferably corresponds to the input data array of another convolutional layer.

[0017] The characteristic of concentrically structured data is mainly that the concentrically structured data are continuously connected to each other in one direction. For example, if the entire surroundings of an object are detected, as in automated driving, concentrically structured data are generated. For example, it can be imagined as a hollow cylinder. Therefore, in the circumferential direction of the hollow cylinder, all data are surrounded by other data.

[0018] For example, if these data are associated with an image recorded concentrically around an object, data in the form of a hollow cylinder can be imagined. In the case of such hollow cylinder data, the hollow cylinder opens along the longitudinal sides and "flattens" when unfolding. Therefore, the data array formed in this way has real relevant data on two opposite sides of the data array. Although these real relevant data map data that are side by side in reality, these data are located on opposite sides in the data array.

[0019] Obviously, concentrically configured data other than this example can be envisioned.

[0020] In this way, the noise added to the data of the concentric configuration by conventional filling methods is eliminated. Furthermore, the truly relevant data can thus be guided relative to each other.

[0021] The filter kernels are applied to both the real data and the data added by the padding method. In this way, although the dimension of the data array is not reduced in any convolutional layer, noise is introduced into the information flow of the data. Unlike the usual padding method, the convolutional neural network requires significantly fewer convolutional layers in order to eliminate the noise again. In this way, shallower architectures or models of convolutional neural networks can be used.

[0022] In addition, the automated anti-noise process that must be learned by the convolutional neural network to remove noise introduces unnecessary complexity and thus wastes the network capacity of the convolutional neural network. This problem can be eliminated through ring filling, thereby achieving smaller, faster and simpler models of convolutional neural networks.

[0023] According to a preferred embodiment, the data array has real non-correlated data on opposite sides of the data array, wherein the method further comprises:

[0024] The data array is filled with real non-correlated data by conventional filling methods (preferably constant filling, zero filling or reflection filling).

[0025] Preferably, the described ring filling is not required in the case of truly non-correlated data.

[0026] According to a preferred configuration, the data array includes a left end and a right end, and the left end and the right end respectively have real relevant data, wherein the method further includes:

[0027] The real related data at the left end of the data array is filled with the real related data at the right end of the data array, and / or the real related data at the right end of the data array is filled with the real related data at the left end of the data array.

[0028] According to a preferred configuration, the data array comprises an upper end and a lower end, wherein the upper end and the lower end respectively have real non-correlated data, wherein the method further comprises:

[0029] The real non-correlated data at the upper and lower ends of the data array are filled by a conventional filling method (preferably constant filling, zero filling or reflection filling).

[0030] According to a preferred embodiment, the data array has an arbitrary number of dimensions.

[0031] Typically, the data array associated with an image has one or two dimensions. However, other dimensions are also conceivable.

[0032] According to a preferred embodiment, the dimension of the annular filling depends on the dimension of the filter kernel used for the convolution operation.

[0033] The larger the dimension of the filter kernel, the larger the dimension of the annular padding must be. The amount of real relevant data added on one side of the annular padding by the annular padding preferably depends on the dimension of the filter kernel used, and therefore on the reduction of the data array caused by the convolution operation.

[0034] According to a preferred configuration, the filling method further comprises:

[0035] Within the convolutional layers of a convolutional neural network, the relationship between the true correlated data on opposite sides of the data array is learned.

[0036] When the concentrically constructed data is expanded into non-annular data, the two ends of the data are arranged to be far away from each other, although the two ends of the data are actually arranged close to each other side by side. In the usual filling method, the convolutional neural network must learn the relationship between the real related data on the opposite sides of the data array. For this purpose, a deep architecture of the convolutional neural network is required in order to have information for this. Through annular filling, the relationship between the real related data on the opposite sides of the data array can be learned within the convolution layer of the convolutional neural network.

[0037] According to a further aspect, a device is provided for carrying out a filling method as described herein.

[0038] Another aspect relates to a control method for an at least partially autonomous robot, the control method comprising the following steps:

[0039] receiving ambient data of an at least partially autonomous robot, wherein the ambient data includes concentrically constructed data associated with an image captured concentrically to the at least partially autonomous robot;

[0040] In case of performing the filling method, image recognition is applied to the captured image;

[0041] The at least partially autonomous robot is controlled based on image recognition for the recorded images.

[0042] In this way, an at least partially autonomous robot can be controlled in an improved manner based on the received surroundings data.

[0043] In general, in particular, the at least partially autonomous robot may be an at least partially automated vehicle. As an alternative, the at least partially autonomous robot may also be another mobile robot, for example a mobile robot that moves by flying, swimming, diving or walking. The mobile robot may also be, for example, an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot, etc.

[0044] According to another aspect, a computer program comprises instructions for causing a computer to perform a method as described herein when the computer program is executed by the computer.

[0045] According to another aspect, a computer program as described herein is stored on a machine-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other measures to improve the present invention will be further illustrated below in conjunction with the description of preferred embodiments of the present invention based on the accompanying drawings.

[0047] Figure 1a shows data of a two-dimensional concentric structure, which is unfolded into a data array;

[0048] Figure 1b shows data of a one-dimensional concentric structure, which is unfolded into a data array;

[0049] Figure 2a A schematic diagram showing zero padding of a data array;

[0050] Figure 2b A schematic diagram showing a circular filling of a data array;

[0051] Figure 3a Shows the use of zero padding for polygons;

[0052] Figure 3b shows the use of a circular fill for a polygon;

[0053] Figure 4a A schematic diagram showing zero filling in the case of a relationship between real correlation data on opposite sides of a learning data array;

[0054] Figure 4b A schematic diagram showing a ring filling in the case of a relationship between real correlation data on opposite sides of a learning data array;

[0055] Figure 5 Schematic diagram showing the padding method in a convolutional neural network. DETAILED DESCRIPTION

[0056] Figure 1a Two-dimensional concentrically structured data 10 are shown. For example, the concentrically structured data 10 are associated with surroundings data which are recorded concentrically to an object, in particular an at least partially automated vehicle. In order to process the concentrically structured data 10, it must be expanded. Figure 1a As shown, the concentrically constructed data 10 has the shape of a hollow cylinder, and the object is arranged concentrically in the center of the hollow cylinder.

[0057] In order to unfold the concentrically constructed data 10 , these data are separated at the separation point 11 and unfolded into a data array 20 . In this way, the real relevant data 21 , 26 arranged at the separation point 11 are arranged on two opposite sides of the data array 20 .

[0058] Figure 1b The data 10 of the one-dimensional concentric structure is shown. The expansion of the data 10 of the concentric structure corresponds to Figure 1a An implementation scheme of two-dimensional concentrically constructed data 10.

[0059] Figure 2a and Figure 2b Shown in Figure 2a Zero padding and Figure 2b Schematic diagram of the circular filling in the convolution operation for the case of a data array 20, wherein the data array is composed of concentrically structured data 10.

[0060] The data array 20 shown comprises six data sets 21, 22, 23, 24, 25, 26, in particular a first data set 21, a second data set 22, a third data set 23, a fourth data set 24, a fifth data set 25 and a sixth data set 26. A convolution operation is performed on the data array 20 by means of a filter kernel 30. The dimension of the filter kernel 30 is three, that is, three data sets are processed by the filter kernel 30 respectively, and these three data sets are discretely convolved into an output data set of an output array 40. The method is implemented step by step until the entire data array 20 has been processed. Due to the limitation of the data array 20 at the edge area, if the data array 20 is convolved without any type of padding method, it will result in a reduction in the dimensionality of the data array 20. In the case shown, the filter kernel 30 will process the data array 20 in four steps. To this end, the filter kernel 30 will perform discrete convolution based on the fourth data set 24, the fifth data set 25 and the sixth data set 26 in the first step, perform discrete convolution based on the third data set 23, the fourth data set 24 and the fifth data set 25 in the second step, perform discrete convolution based on the second data set 22, the third data set 23 and the fourth data set 24 in the third step, and perform discrete convolution based on the first data set 21, the second data set 22 and the third data set 23 in the fourth step. In this way, the dimension of the data array 20 to the output array 40 will be reduced from six to four.

[0061] Therefore, the data array 20 is usually artificially enlarged by a padding method, in this case from six to eight, so that the output array 40 continues to have a dimension of six after the convolution operation.

[0062] exist Figure 2a In the case of zero padding in , the data array 20 is extended at both ends with a left zero data set 21a and a right zero data set 26a, respectively, and the left zero data set and the right zero data set have zero values, respectively. In step S1, the filter kernel 30 performs discrete convolution based on the fifth data set 25, the sixth data set 26 and the right zero data set 26a. As described above, the filter kernel 30 processes the data array 20 to the left in steps S2 to S5. In step S6, the filter kernel 30 performs discrete convolution based on the left zero data set 21a, the first data set 21 and the second data set 22. In this way, the output array 40 has another dimension six.

[0063] exist Figure 2bIn the case of the circular padding in step S1, the data array 20 is not artificially expanded. Instead, in step S1, the filter kernel 30 performs discrete convolution based on the fifth data set 25, the sixth data set 26 and the first data set 21. Because the data array 20 is composed of concentrically constructed data 10, the first data set 21 and the sixth data set 26 are real related data. In this way, the circular padding uses real data instead of zeros so that the padding method is used for the data array 20.

[0064] As described above, the filter core 30 processes the data array 20 to the left in steps S2 to S5. In step S6, the filter core 30 performs discrete convolution based on the sixth data set 26, the first data set 21, and the second data set 22. In this way, the output array 40 has a dimension of six. In this way, the ring filling uses real data instead of zeros so that the filling method is used for the data array 20.

[0065] Figure 3a and 3b The difference between zero filling and ring filling is illustrated using polygons as an example.

[0066] The results shown are from a convolutional autoencoder that has been given the task of reconstructing polygons that represent a one-dimensional array. Each polygon consists of 360 numbers representing the distance to the origin in a polar coordinate system with a resolution of 360 degrees. The basic shapes 50 represent reality. The output shapes 51 each include the learned values.

[0067] exist Figure 3a The output shape 51 shown in has been established with the help of zero padding, while Figure 3b The output shape 51 shown in has been created by means of a ring fill. Figure 3a It is clear from the diagram what errors zero padding introduces into the system, since the additional data set required for zero padding is described by zeros. Figure 3b The output shape 51 shown in FIG. 5 , which is learned by means of ring filling, is relatively correct.

[0068] Figure 4a and Figure 4b The flow of information of data sets from the left side of data array 20 to the right side of data array 20 and vice versa is shown.

[0069] The basic structure of the data array 20 corresponds to Figure 2a and 2b The data array 20 shown in FIG. Figure 4aThe use of convolution operations on multiple convolutional layers L1, L2, L3, L4, L5, L6 with zero padding is shown. The first data set 21 requires six convolutional layers L1, L2, L3, L4, L5, L6 in order to penetrate from the left side of the data array 20 to the right side of the data array 20. Only then can the relationship between the first data set 21 and the sixth data set 26 be established by the learning algorithm and it can be recognized that this is real relevant data. The opposite applies to the sixth data set 26, which also requires 6 convolutional layers L1, L2, L3, L4, L5, L6 in order to penetrate to the left side of the data array 20.

[0070] In contrast, Figure 4b As shown in FIG. 1 , after the first convolutional layer L1 , a relationship between the first data set 21 and the sixth data set 26 can already be established by a learning algorithm, and it can be recognized that this is actually related data.

[0071] Figure 5 Schematic diagram showing padding methods in convolutional neural networks.

[0072] In step Z1 , data of a concentric structure of an object are received, wherein the data of the concentric structure are associated with an image recorded concentrically to the object.

[0073] In step Z2 , the concentrically structured data are expanded into a data array which includes the actual relevant data on its opposite sides.

[0074] In step Z3, a convolution operation is performed using circular filling, wherein, in the case of circular filling, the real relevant data on one side of the data array is used to fill the real relevant data on the side opposite to it, and / or the real relevant data on the side of the data array opposite to this side is used to fill the real relevant data on this side of the data array.

Claims

1. A filling method for a convolutional neural network, the filling method comprising the following steps: Receiving concentrically constructed data (10) of an object, wherein the concentrically constructed data (10) is associated with an image taken concentrically to the object; Expanding the concentrically structured data (10) into a data array (20), the data array comprising real correlation data (21, 26) on opposite sides of the data array (20); A convolution operation is performed using annular filling, wherein, in the case of the annular filling, real relevant data (21, 26) on one side of the data array (20) is used to fill real relevant data (21, 26) on the side of the data array (20) opposite to the side, and / or, real relevant data on the side of the data array opposite to the side is used to fill real relevant data on the side of the data array.

2. The filling method according to claim 1, wherein: The data array (20) has true non-correlated data on opposite sides of the data array, wherein the method further comprises: The real non-correlated data of the data array (20) is filled by constant filling, zero filling or reflection filling.

3. The filling method according to any one of claims 1 or 2, wherein: The data array (20) has a left end and a right end, and the left end and the right end respectively have the real relevant data (21, 26), wherein the method further comprises: The real relevant data (21, 26) at the left end of the data array (20) is filled with the real relevant data (21, 26) at the right end of the data array (20), and / or the real relevant data at the right end of the data array is filled with the real relevant data at the left end of the data array.

4. The filling method according to claim 2, wherein: The data array (20) comprises an upper end and a lower end, wherein the upper end and the lower end respectively have real non-correlated data, wherein the method further comprises: The real non-correlated data at the upper and lower ends of the data array (20) are filled by constant filling, zero filling or reflection filling.

5. The filling method according to any one of claims 1 or 2, wherein: The data array (20) has an arbitrary number of dimensions.

6. The filling method according to any one of claims 1 or 2, wherein: The dimension of the annular padding depends on the dimension of the filter kernel (30) used for the convolution operation.

7. The filling method according to any one of claims 1 or 2, further comprising: The relationship between the real related data (21, 26) at opposite ends of the data array (20) is learned in a convolutional layer (L1) of the convolutional neural network.

8. A filling device for a convolutional neural network, wherein the filling device is configured to implement a filling method according to any one of claims 1 to 7.

9. A control method for an at least partially autonomous robot, the control method comprising the steps of: Receiving surrounding data of the at least partially autonomous robot, wherein the surrounding data comprises concentrically constructed data (10), the concentrically constructed data being associated with an image taken concentrically to the at least partially autonomous robot; When performing the filling method according to any one of claims 1 to 7, image recognition is used for the recorded image; The at least partially autonomous robot is controlled based on image recognition for the recorded images.

10. A computer program product comprising instructions for causing a computer to implement the method according to any one of claims 1 to 7 when the computer program product is implemented by the computer.

11. A machine-readable storage medium on which the computer program product according to claim 10 is stored.

Images