Methods for running artificial neural networks

Abstract

A method for operating an artificial neural network (10) is provided, the artificial neural network having at least one convolutional layer (12a-c), which is configured to convert an input matrix (I) of the convolutional layer (12a-c) into an output matrix (A) based on a convolution operation and a shift operation. The method comprises: determining at least one first normalization variable and a second normalization variable based on entries of the input matrix (I) and / or based on a training data set; determining a modified filter matrix ( / ) based on an original filter matrix (f) and based on at least one of the determined first normalization variable and the second normalization variable; determining a modified shift matrix (5) based on an original shift matrix (b) and based on at least one of the determined first normalization variable and the second normalization variable. The method further comprises: converting the input matrix (I) into the output matrix (A) by applying the modified filter matrix ( / ) and the modified shift matrix (5). In this way, the normalization operation can be advantageously combined with the convolution operation and the shift operation.

Description

Technical Field

[0001] The present invention generally relates to the field of artificial intelligence. In particular, the present invention relates to a method for operating an artificial neural network and an artificial neural network, which can be used, for example, for object recognition in a vehicle. Background Art

[0002] In order to train an artificial neural network, in particular a multi-layer and / or convolutional neural network, the training data (e.g. training images) provided to the neural network are usually normalized and / or standardized. This can also be used to homogenize the training data of a training data set. By means of standardization, objects and / or features present in the training data can also be highlighted and / or emphasized in some cases.

[0003] After training a neural network and / or during operation of a trained neural network (which is also often referred to as inference), the input data provided to the neural network for interpretation and / or classification is usually normalized and / or standardized similar to the training data. In this case, the normalization of the input data can be computationally intensive and / or time-consuming and requires powerful hardware to perform. Summary of the invention

[0004] By means of embodiments of the present invention, an efficient and fast method for operating an artificial neural network can be provided in an advantageous manner. By means of embodiments of the present invention, an improved artificial neural network can also be provided.

[0005] One aspect of the invention relates to a method for operating an artificial neural network. The artificial neural network has at least one convolutional layer and / or at least one convolutional layer, which is configured to transform an input matrix of the convolutional layer into an output matrix based on a convolution operation and a shift operation, such as a linear shift. The method comprises the following steps: based on the entries of the input matrix of the neural network And / or determining at least one first normalization variable and a second normalization variable based on a training data set of a neural network. The method is particularly characterized by the following steps:

[0006] Determining a modified filter matrix based on an original filter matrix of the at least one convolutional layer and based on at least one of the determined first normalization parameter and the determined second normalization parameter;

[0007] Determine a modified shift matrix based on an original shift matrix of the at least one convolutional layer and based on at least one of the determined first normalized parameter and the determined second normalized parameter;

[0008] The input matrix is ​​transformed into an output matrix by applying and / or based on the modified filter matrix and the modified shift matrix.

[0009] The method may represent a method for training a neural network and / or a method for running a trained neural network. The neural network may, for example, represent a multilayer artificial neural network that may have a plurality of convolutional layers. Alternatively or additionally, the neural network may represent a convolutional neural network. The neural network may, for example, have a plurality of convolutional layers, such as hidden layers, that follow one another. In the context of the present disclosure, at least one convolutional layer may, for example, represent an input layer and / or a first layer of the neural network, a convolutional layer downstream of the input layer, and / or a hidden layer of the neural network.

[0010] The input matrix may generally represent any input data elements of at least one convolutional layer. If the neural network has multiple convolutional layers, the input matrix may represent the input data elements for each of these convolutional layers. For the input layer and / or the first layer of the neural network, the input matrix may represent the input data elements provided to the neural network, such as an input image. For a convolutional layer downstream of the input layer, the input matrix may also represent the input data elements provided to the convolutional layer, which input data elements may, for example, be related to and / or may be equivalent to the output matrix of the upstream convolutional layer. The input matrix may also have any dimension. The input matrix may in particular be one-dimensional or multi-dimensional. In other words, the input matrix may represent a one-dimensional or multi-dimensional input tensor.

[0011] The output matrix may represent any output data element of at least one convolutional layer, which may be generated by the convolutional layer based on the input matrix by applying a convolution operation and a shift operation. If the neural network has multiple convolutional layers, the output matrix may be the output data element of each arbitrary convolutional layer. In the case of multiple convolutional layers following one another, the output matrix of the upstream convolutional layer may also form the input matrix of the convolutional layer downstream of the upstream convolutional layer, be related to the input matrix of the downstream convolutional layer and / or be equivalent to the input matrix of the downstream convolutional layer. The output matrix may also have any dimension. The output matrix may in particular be one-dimensional or multi-dimensional. In other words, the output matrix may represent a one-dimensional or multi-dimensional output tensor. The output matrix may also have the same dimension as the input matrix of the corresponding convolutional layer, or may have a different dimension from the input matrix of the corresponding convolutional layer.

[0012] The original filter matrix may, for example, represent a filter matrix of a convolutional layer as used when training a neural network and / or obtained for the convolutional layer. The modified filter matrix may represent a filter matrix used by the associated convolutional layer during operation of the trained neural network (also referred to as inference). Similarly, the original shift matrix may represent a shift matrix of a convolutional layer as used when training a neural network and / or obtained for the convolutional layer. The modified shift matrix may represent a shift matrix used by the associated convolutional layer during operation of the trained neural network.

[0013] The first normalization parameter and the second normalization parameter can be used for normalization of the input matrix and / or represent parameters for normalization of the input matrix. The first normalization parameter and / or the second normalization parameter can be a scalar, a vector, a matrix and / or a tensor. The first normalization parameter and the second normalization parameter can generally be parameters that can be derived from the input items of the input matrix and / or from the training data set, such as statistical parameters. The first normalization parameter can be related to the standard deviation and / or the second normalization parameter can be related to the mean value. The mean value can also be a weighted mean value, wherein the input items of the input matrix can be weighted differently and / or the individual training images of the training data set can be weighted differently. The first normalization parameter and the second normalization parameter can also be high-order elements of statistics ( Momente), such as skewness (English: Skewness, German: Schiefe) or kurtosis (English: Kurtosis, German: ). In general, within the scope of the method according to the invention, any number of normalization variables can be determined, with the aid of which the input matrix can be normalized by using addition, subtraction, multiplication and / or division.

[0014] The present invention can be considered as being based on the knowledge described below in particular. During the training of an artificial neural network, the training data elements (e.g., training images) can be normalized and / or standardized by subtracting the mean value from the entry of the training data elements and dividing the entry of the training data elements by the standard deviation. The mean value and the standard deviation can be global values ​​here and / or can be obtained based on the entire training data set with multiple training data elements. Then, all training data elements, all input data elements, and / or all input matrices of each convolutional layer can be normalized with the help of the global values ​​of the standard deviation and the mean value. Alternatively, the mean value and the standard deviation can also be local values, which are obtained separately for each training data element provided to the neural network and / or each input data element provided to the neural network and can be used for normalization. In conventional methods for operating neural networks and / or in conventional neural networks, each input data element (e.g., input image) fed into the neural network is usually normalized in this way during the operation of the trained neural network. The input matrices of one or more convolutional layers can also be normalized in this way. Sometimes this requires a lot of computing costs and / or computing time.

[0015] Therefore, according to the present invention, normalization and / or standardization are combined with the convolution operation and / or shift operation performed by the convolution layer. Therefore, in the modified filter matrix and the modified shift matrix, a normalization operation for normalizing the input matrix can be included and / or integrated, especially a linear normalization operation. The normalization operation can usually include the addition, subtraction, multiplication and / or division of the input matrix with the first normalization parameter and / or the second normalization parameter and / or with other normalization parameters. The normalization operation can especially include deducting the mean value from the entry of the input matrix and / or dividing the entry of the input matrix by the standard deviation. Therefore, by considering and / or integrating the first normalization parameter and the second normalization parameter into the modified filter matrix and the modified shift matrix, the convolution operation, the shift operation and the normalization operation can be combined and / or implemented in one step. Therefore, the separate step for performing the normalization operation can be omitted. In other words, by integrating the first and / or second normalization parameters into the modified filter matrix and / or the modified shift matrix, the normalization operation can be integrated into the modified filter matrix and / or the modified shift matrix. The combination of normalization and convolution operation and / or shift operation can be implemented in the input layer and / or the first layer of the neural network, through which the input data elements, such as the input image, can be provided to the neural network. Alternatively or additionally, the convolution operation and / or shift operation applied to the input matrix of the convolution layer by any other convolution layer can also be combined with the corresponding normalization operation of the input matrix. Thus, explicit and / or separate normalization of the input data elements and / or the input matrix can be avoided. Therefore, the computational cost and / or computational time can be reduced overall, so that the neural network can be configured faster and more efficiently. Therefore, the trained neural network can be implemented on almost any hardware and / or transplanted to almost any hardware, with only little impact on the performance of the hardware or no impact on the performance of the hardware. This also makes it possible to implement hardware for running the neural network, in particular a processor and / or a graphics processor, with less power, which can save costs.

[0016] According to one embodiment of the present invention, the first normalized parameter is a parameter related to the standard deviation and / or is the standard deviation. The first normalized parameter may also be, for example, the reciprocal of the standard deviation and / or is related to the reciprocal of the standard deviation. Alternatively or additionally, the second normalized parameter is a parameter related to the mean value and / or is the mean value. The second normalized parameter may also be, for example, the negative ratio of the mean value to the standard deviation and / or is related to the ratio. The first and second normalized parameters may be obtained based on the entries of the input matrix and / or based on the training data set.

[0017] According to one embodiment of the invention, the modified filter matrix is ​​determined based on, in particular only based on, the original filter matrix and the determined first standard deviation. Alternatively or additionally, the modified shift matrix is ​​determined based on the original shift matrix, based on the determined standard deviation and based on the determined mean value. Thus, the standard deviation and mean value for normalizing the input matrix can be integrated into the modified filter matrix and / or shift matrix, so that a separate input matrix normalization to be performed can be omitted.

[0018] According to one embodiment of the present invention, the average value obtained is the average value of the entries of the input matrix. Alternatively or additionally, the standard deviation obtained is the standard deviation of the entries of the input matrix. Therefore, the average value and the standard deviation can be obtained as local values ​​of the input matrix of the convolution layer. In this way, the average value assigned to the input matrix and / or the standard deviation assigned to the input matrix can be obtained for each input matrix. If the corresponding convolution layer is the input layer, the average value obtained can be the average value of the entries of the corresponding input data elements, such as the average value of the pixels of the input image. Similarly, the standard deviation can be the standard deviation of the entries of the corresponding input data elements, such as the standard deviation of the pixels of the input image.

[0019] According to one embodiment of the present invention, the training data set has a plurality of training data elements, in particular training images, wherein the standard deviation and the mean value are calculated based on the training data elements of the training data set. Therefore, the mean value and the standard deviation can be calculated as global values ​​of the input matrix of the convolutional layer. Therefore, the same mean value and the same standard deviation can be used for each arbitrary input matrix and / or each arbitrary input data element. This can further reduce the required computational cost and / or computational time.

[0020] According to one embodiment of the present invention, the step of converting the input matrix into the output matrix comprises: convolving the input matrix with the modified filter matrix and adding the modified shift matrix to the convolved input matrix. As described above, the modified filter matrix and the modified shift matrix contain a normalization operation for normalizing the input matrix. Therefore, the input matrix can be normalized, convolved and shifted at the same time to generate the output matrix. Therefore, the input matrix does not need to be normalized before being provided to the convolutional layer, but all calculation operations applied to the input matrix can be performed in one step.

[0021] According to one embodiment of the present invention, the step of obtaining the modified filter matrix includes: obtaining the ratio of the entries of the original filter matrix to the obtained standard deviation. In other words, the modified filter matrix can be obtained by dividing the entries of the original filter matrix by the obtained standard deviation. This ensures that the input matrix is ​​correctly normalized. Here, the entries of the original filter matrix can represent the parameter values ​​and / or weights of the corresponding convolutional layer obtained during the training of the neural network.

[0022] According to one embodiment of the present invention, the step of obtaining the modified shift matrix comprises: a step of convolving the modified filter matrix with a normalized matrix, wherein all entries of the normalized matrix have the obtained average value; and a step of subtracting the modified filter matrix after the convolution with the normalized matrix from the original shift matrix. In other words, the modified shift matrix can be obtained by obtaining the difference between the convolution results of the original shift matrix and the normalized matrix and the modified filter matrix. This ensures that the input matrix is ​​correctly normalized. Here, the entries of the original shift matrix can represent the parameter values ​​and / or weights of the corresponding convolution layer obtained during the training of the neural network.

[0023] According to one embodiment of the present invention, the method further comprises the following steps: converting the input matrix into a higher dimensional input matrix by adding entries to the input matrix. Here, if the neural network is trained and / or during operation of the trained neural network or during inference, the added entries each have the average value obtained. Alternatively or additionally, during neural network training, the added entries each have a zero value. Advantageously, by adding the entries to the input matrix, the dimension of the output matrix after applying the convolution operation is consistent with the dimension of the input matrix before adding the entries. The addition of entries is generally referred to as "padding". Therefore, if "zeros" are added to the input matrix of the convolution layer during neural network training, entries with average values ​​can be added to the input matrix of the convolution layer after training, i.e., during operation of the trained network.

[0024] According to one embodiment of the present invention, if the neural network is trained, the values ​​of the entries added to the input matrix each have the value zero. Alternatively or additionally, during the training of the neural network, the entries added each have the negative value of the ratio of the calculated mean value to the calculated standard deviation. Therefore, if during the training of the neural network, entries are added to the input matrix of the convolution layer, which entries correspond to the negative ratio of the calculated mean value to the calculated standard deviation, then after the training, that is, during the operation of the trained network, "zeros" can be added to the input matrix of the convolution layer. In this way, in particular, the dimensions of the output matrix can remain unchanged and / or be consistent with the dimensions of the input matrix before the addition of the entries.

[0025] According to one embodiment of the invention, the at least one convolutional layer is an input layer and / or a first layer of a neural network. Alternatively or additionally, the input matrix is ​​an input data element, in particular an input image, which is provided to the neural network for interpretation and / or classification. In this case, the input data element can have any dimension and / or be an input image with any number of pixel arrays. The pixels of the input image can here be entries of the input data element and / or entries of the input matrix of the input layer.

[0026] According to one embodiment of the present invention, the modified filter matrix and the modified shift matrix are used only in the input layer of the neural network. Therefore, the efficiency of the method for running the neural network can be further improved and / or the computational cost or computational time can be further reduced.

[0027] According to one embodiment of the present invention, a neural network has a plurality of convolutional layers, each of which is configured to convert an input matrix of a corresponding convolutional layer into an output matrix of the corresponding convolutional layer, wherein a modified convolution matrix and a modified shift matrix are obtained for each convolutional layer, and each input matrix of each convolutional layer is converted into an output matrix by applying a modified filter matrix obtained for the corresponding convolutional layer and a modified shift matrix obtained for the corresponding convolutional layer. Here, the modified filter matrix and the modified shift matrix can be obtained based on an original filter matrix and an original shift matrix of the corresponding convolutional layer.

[0028] According to one embodiment of the present invention, the output matrix of at least one convolutional layer forms the input matrix of at least one convolutional layer downstream of the convolutional layer, so that the mean value and standard deviation for the downstream convolutional layer are obtained based on the output matrix. In other words, the output matrix of the upstream convolutional layer can be equivalent to the input matrix of the convolutional layer directly downstream of the upstream convolutional layer.

[0029] Another aspect of the invention relates to an artificial neural network. The artificial neural network is provided for carrying out the method for operating a neural network as described above and below. The neural network can be implemented in particular on a processor and / or can have a processor.

[0030] Features, elements and / or steps of a method for operating a neural network may be features and / or elements of the neural network, and vice versa. In other words, what is disclosed about the method also applies to what is disclosed about the neural network, and vice versa.

[0031] The neural network can in particular have at least one convolutional layer, which is configured to: convolve an input matrix with a modified filter matrix and add the convolved input matrix to a modified shift matrix to generate an output matrix of the convolutional layer. Here, the entries of the modified filter matrix can be related to the ratio of the original filter matrix entry and the standard deviation of the convolutional layer. The entries of the modified filter matrix can be given, for example, by the ratio of the entry of the original filter matrix and the standard deviation. The entries of the modified shift matrix can also be related to the difference between the result after the original shift matrix and the modified filter matrix are convolved with a normalized matrix, and the entries of the normalized matrix have the average values ​​obtained. In other words, the entries of the modified shift matrix can be given by the difference between the result after the original shift matrix and the modified filter matrix are convolved with the normalized matrix. The standard deviation and the average value can be obtained based on the entries of the input matrix and / or based on the training data set of the neural network.

[0032] The method and / or neural network for operating a neural network according to the present invention can be advantageously used in many industrial fields. The method and / or neural network can be used in the field of autonomous driving, for example in motor vehicles. The method and / or neural network can be used, for example, for object recognition based on image data. With the help of the method and neural network according to the present invention, it is especially possible to perform recognition of traffic signs, pedestrians, obstacles and / or other objects in image data. Image data and / or images can be recorded, for example, by means of a camera of a motor vehicle and provided to the neural network for object recognition. The neural network can interpret, process and / or process the image data. Here, the neural network can, in particular, classify the objects contained in the image data and output corresponding output items. For example, the neural network can determine at least one probability value and / or probability that the objects contained in the image data can be assigned to a specific object classification and / or object category based on the processing of the image data. Alternatively or additionally, the neural network can determine the object classification and / or object category to which the objects contained in the image data can be assigned. The object category and / or at least one probability value can be output and / or provided by the neural network. The output item can be provided, for example, to a control unit of the motor vehicle, which infers a vehicle maneuver, such as a bypass maneuver and / or a braking maneuver, based on the output item. The control unit also arranges the motor vehicle to execute the inferred or ascertained vehicle maneuver, in particular to execute a bypass maneuver and / or a braking maneuver. Thus, a method for operating a vehicle can be implemented with the aid of a neural network. Alternatively or additionally, a warning device can be actuated based on the output item of the neural network, for example to output an acoustic and / or optical warning signal.

[0033] The method and the neural network can also be advantageously used for segmenting image data. The method and the neural network can also be used for analyzing medical images and / or medical image data, for example for identifying diabetes in a patient. The method and the neural network can also be used and / or applied in robotics, for example for controlling a robot, in particular based on image data. Another possible field of use of the neural network and / or the method is quality control, for example for identifying goods with manufacturing defects based on image data and / or other data.

[0034] Embodiments of the present invention are described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] Figure 1 An artificial neural network according to an embodiment of the present invention is shown;

[0037] Figure 2 A flow chart is shown for illustrating the steps of a method for operating a neural network according to one embodiment of the present invention;

[0038] Figure 3 A flow chart is shown for illustrating the steps of a method for operating a neural network according to one embodiment of the present invention;

[0039] Figure 4 A flow chart is shown for illustrating the steps of a method for operating a neural network according to an exemplary embodiment of the present invention.

[0040] These drawings are only schematic and not drawn to scale. In the drawings, identical, identically functioning or similar elements are provided with the same reference numerals. DETAILED DESCRIPTION

[0041] Figure 1 An artificial neural network 10 according to an embodiment of the present invention is shown. Figure 1 The artificial neural network 10 shown in FIG. 1 is embodied as a multilayer neural network 10 by way of example.

[0042] For example, in the context of object recognition in a motor vehicle, the neural network 10 can be used to interpret image data. The neural network 10 can be coupled to a storage device 11 and / or can have a storage device 11 for storing image data, for example. The image data can be recorded, for example, by means of a camera of the motor vehicle and can be stored in the storage device 11 for further processing and / or interpretation by the neural network 10. The image data can here have one or more images, which can be provided to the neural network 10 and / or fed into the neural network 10.

[0043] The artificial neural network 10 has an input layer 12a and / or a first layer 12a, to which an input matrix I0 is provided. Typically, the input layer 12a of the neural network 10 is a convolutional layer. Therefore, the first layer and / or the input layer 12a can represent the first convolutional layer 12a of the neural network 10. The input matrix I0 provided to the input layer 12a can be, for example, an input data element I0 and / or an input image I0. The input data element, the input image and / or the input matrix I0 of the first layer 12a can be any array with an arbitrary number of pixels, wherein a pixel corresponds to an entry of the input matrix I0. The input data element, the input image and / or the input matrix I0 of the input layer 12a can be, in particular, a single image of image data stored in the storage device 11. Here, the input data element, the input image and / or the input matrix I0 of the input layer 12a can be provided to the neural network 10 without preprocessing, in particular without prior normalization and / or standardization, which is explained in detail below.

[0044] The neural network 10 also has a plurality of convolutional layers 12b, 12c. The neural network 10 can have any number of convolutional layers 12b, 12c. The convolutional layer 12b is located directly downstream of the input layer 12a and / or is coupled to the input layer. Similarly, the convolutional layer 12c is located downstream of the convolutional layer 12b, wherein further convolutional layers and / or further layers of the neural network 10 can optionally be arranged between the convolutional layers 12b, 12c.

[0045] The input layer 12a and the convolutional layers 12b, 12c are generally configured to transform the input matrices I0, I1, I2 provided to the corresponding layers 12a-c based on convolution operations and shift operations, especially linear shifts. n Transformed into the output matrices A0, A1, A of the corresponding layers 12a-c n-1 In other words, each of the layers 12a-c is implemented to perform a convolution operation and a shift operation. Figure 1 In the embodiment shown in , the input layer 12a generates an output matrix A0 from the input matrix I0 of the input layer 12a by using a convolution operation and a shift operation. Based on the output matrix A0 of the input layer 12a, an input matrix I1 can be generated for the convolution layer 12b directly downstream of the input layer 12a. To this end, the output matrix A0 of the input layer 12a can be modified, and / or the output matrix A0 of the input layer 12a can constitute the input matrix I1 of the convolution layer 12b. Similarly, the output matrix A0 of the convolution layer directly upstream of the convolution layer 12c n-1 The input matrix I of the convolution layer 12c can be formed n ,like Figure 1 shown.

[0046] Details of the convolution operations and shift operations performed by each of the convolutional layers 12a-c will be explained in subsequent figures.

[0047] As a result of processing and / or interpreting the input matrix, the input data elements and / or the input image I0, the neural network 10 can determine, provide and / or output an output item 14. The output item 14 can be, for example, an object class to which an object contained in the input image I0 can be assigned. Alternatively or additionally, the output item 14 can have one or more probability values. Each probability value can indicate the probability that an object contained in the input image I0 can be assigned to a determined object class with this probability.

[0048] If neural network 10 is used to detect obstacles in a motor vehicle, output item 14 can also be provided to a control unit of the motor vehicle which, based on output item 14 , for example ascertains bypassing and / or braking maneuvers and can order the motor vehicle to carry out these bypassing and / or braking maneuvers.

[0049] Figure 2 A flow chart is shown for illustrating the steps of a method for operating a neural network 10 according to an exemplary embodiment of the present invention. Figure 2 The neural network 10 described in the reference Figure 1 The neural network 10 described has the same elements and features.

[0050] Reference Figure 2 The depicted neural network 10 typically has at least one convolutional layer 12a-c, which is configured to transform the input matrix I i , i=0…n is transformed into output matrix A i , i=0...n. For simplicity, the indices i=0...n are omitted below. The method described below can be performed for each arbitrary convolutional layer 12a-c. In other words, the method described below can involve one, multiple or all convolutional layers 12a-c.

[0051] In a first step S1, at least one first normalized parameter v and a second normalized parameter w are determined. The first normalized parameter v can be related to the standard deviation σ, can be the inverse of the standard deviation and / or can be the standard deviation σ. The second normalized parameter w can be related to the average value μ, can be the average value μ and / or can be the negative ratio of the average value to the standard deviation. The first normalized parameter v, the second normalized parameter w, the standard deviation σ and / or the average value μ can be obtained based on the input matrix I of the corresponding convolutional layer 12a-c and / or based on the entries of the input matrix I. For this purpose, the average value μ can be obtained as the average value μ of all entries of the input matrix I. Different weightings can also be performed on the various entries of the input matrix to obtain the average value μ. Similarly, the standard deviation σ can be obtained as the standard deviation σ of all entries of the input matrix I. Here, for one or more convolutional layers 12a - c , a standard deviation σ and an average value μ may be determined for the corresponding convolutional layer 12a - c based on an input matrix I provided to the convolutional layer 12a - c.

[0052] Alternatively or additionally, the first normalized parameter v, the second normalized parameter w, the standard deviation σ and / or the mean value μ can be obtained based on the training data set of the neural network 10. The training data set can have multiple training data elements, in particular multiple training images. Therefore, the mean value μ can be the mean value of all entries of all training data elements of the training data set. It is also possible to weight each training image differently to obtain the mean value μ. Similarly, the standard deviation σ can be the standard deviation of all entries of all training data elements of the training data set. The standard deviation σ and the mean value μ obtained in this way can then be used for one, multiple or all convolutional layers 12a-c.

[0053] In step S1 , the first normalization variable v and / or the second normalization variable, in particular the standard deviation and / or the mean value μ, can also be modified, for example by adding a value and / or by multiplying by a factor.

[0054] In a further step S2, a modified filter matrix f is determined based on the original filter matrix f of the convolutional layer 12a-c and based on the determined first normalization variable v (eg the determined standard deviation σ).

[0055] In a further step S3, a modified shift matrix is ​​determined based on the original shift matrix b of the convolutional layer 12a-c, based on the first normalization parameter v and based on the second normalization parameter w. For example, the modified shift matrix b can be determined based on the original shift matrix b of the convolutional layer 12a-c, based on the determined standard deviation σ and based on the determined mean value μ.

[0056] Therefore, in the modified filter matrix and the modified shift matrix The normalization operation for the input matrix I may be included in , so that the convolution operation performed by the convolution layers 12a-c and the shift operation performed by the convolution layers 12a-c may be combined with the normalization operation. Thereby, explicit and / or separate normalization of the input matrix may be avoided.

[0057] In conventional methods for operating a neural network, the input matrix I is first normalized, then convolved with the original filter matrix f and shifted with the original shift matrix b. The input matrix I can generally be normalized and / or standardized with the aid of a first normalization parameter v and a second normalization parameter w. For normalization, in particular, the entries of an input matrix I and / or multiple input matrices I can be multiplied with the first normalization parameter v, and the second normalization parameter w can be added to the multiplication result. In conventional neural networks, the normalized input matrix I is convolved with the original filter matrix f and shifted with the original shift matrix b, as shown in the following equation.

[0058] (vI+w)*f+b

[0059] The modified filter matrix obtained in step S2 and the modified shift matrix obtained in step S3 The above equation can be transformed as follows.

[0060]

[0061] in

[0062]

[0063] is the modified filter matrix, and

[0064]

[0065] is the modified shift matrix.

[0066] Then, in a further step S4, the input matrix I is converted into an output matrix A. To this end, the input matrix I is compared with the modified filter matrix Perform convolution and pass the convolution result through the modified shift matrix The addition is shifted as shown in the following equation.

[0067]

[0068] The method described above is described below for this case: the first normalization parameter v is a parameter related to the standard deviation σ, and the second normalization parameter w is a parameter related to the mean value μ.

[0069] In a conventional neural network 10 and / or a conventional method for operating a neural network 10, in order to normalize the input matrix, the mean value μ is usually subtracted from each entry of the input matrix I, and the result of the difference is divided by the standard deviation σ. Then, the normalized result is convolved with the original filter matrix f and linearly shifted with the original shift matrix b, as shown in the following equation.

[0070]

[0071] in, is a normalized matrix and / or a mean matrix whose entries all have mean values ​​μ and have the same dimensions as the input matrix I. The original shift matrix b has the same dimensions as the result of convolving the normalized input matrix with the original filter matrix f, and all entries of the shift matrix b have constant values ​​b.

[0072] With the aid of the modified filter matrix determined in step S2 and the modified shift matrix obtained in step S3 The above equation can be transformed as follows.

[0073]

[0074] in

[0075]

[0076] is the modified filter matrix, and

[0077]

[0078] is the modified shift matrix.

[0079] Therefore, the first normalization variable v may be the inverse of the standard deviation σ and / or be related to the inverse thereof, and the second normalization variable w may be the negative ratio of the mean value μ to the standard deviation σ and / or be related to the ratio.

[0080] Then, in a further step S4, the input matrix I is transformed into an output matrix A. To this end, the input matrix is ​​combined with the modified filter matrix Perform convolution and pass the convolution result through the modified shift matrix Addition is performed as a shift, as shown in the following equation:

[0081]

[0082] Therefore, as can be seen from the above four equations, through the modified filter matrix and the modified shift matrix The normalization of the input matrix I can be advantageously combined with the input matrix and the modified filter matrix Perform the convolution together and pass the modified shift matrix with the result of the convolution Perform the shift together.

[0083] The method described above can be performed in the input layer 12a. The method described can be performed in particular only in the input layer 12a. Alternatively, the method described can also be performed in any, in particular all, convolutional layers 12a-c.

[0084] The method according to the present invention is compared with a conventional method for running a neural network. In the conventional method, the input matrix I is first normalized, and then the normalized result is convolved with the original filter matrix f. The convolution result is then added to the original shift matrix to generate the output matrix A. Therefore, the conventional method can be expressed by the equation . Therefore, in the conventional method, the normalization operation as well as the convolution and the shift are separated. Here, the first step is normalization, which is performed for each input matrix not only during the training of the neural network 10 but also during the operation of the trained neural network. Here, the normalization includes subtracting the mean value μ and dividing by the standard deviation σ from all entries of the input matrix I. In the second step, in the conventional method, the normalized input matrix is ​​convolved with the original filter matrix f and the result of the convolution is shifted by adding it with the original shift operation.

[0085] In contrast, in the method according to the invention, by applying a modified filter matrix and the modified shift matrix Combine the normalization operation with the convolution operation and the shift operation. In order to obtain the modified filter matrix The entries in the original filter matrix f are divided by the standard deviation σ. Therefore, step S2 may include a sub-step: obtaining the ratio of the entries in the original filter matrix f to the standard deviation σ. In order to obtain the modified shift matrix Also the modified filter matrix With the normalized matrix The convolution is performed and the result of the convolution is subtracted from the original shift matrix b. Thus, the input matrix I can be replaced in an unchanged and / or unnormalized form with the modified filter matrix and the modified shift matrix Convert to output matrix.

[0086] If the standard deviation σ and mean μ are obtained based on the training data set, the modified filter matrix can be obtained at one time: and the modified shift matrix And it is not necessary to modify the input matrix I, the modified filter matrix and / or a modified shift matrix This can significantly reduce the computational effort and / or computational time for processing the input data elements and / or input images I by the neural network 10 compared to conventional methods, since in conventional methods each input matrix is ​​first normalized before being provided to the neural network 10.

[0087] If the standard deviation σ and the mean μ are calculated based on the input matrix I, it may be necessary to match the modified filter matrix based on the standard deviation σ of the input matrix I and the mean μ of the input matrix I for the corresponding convolutional layer 12a-c. and the modified shift matrix However, because the modified filter matrix and the modified shift matrix Typically has a smaller dimension than the input matrix I, so even in this case the computational expense and / or computation time can be significantly reduced compared to conventional methods.

[0088] Figure 3 A flow chart is shown for illustrating the steps of a method for operating a neural network 10 according to an exemplary embodiment of the present invention. Figure 3 The method described in Figure 2 The same elements, features and steps as described in the method. In particular, steps S1 to S4 can be similar to Figure 2 method to configure.

[0089] Figure 3 The method shown in FIG. 1 has an optional step S2′ in which the input matrix I is transformed into a higher-dimensional input matrix by adding entries This process is also often referred to as "padding", and can be used, among other things, to compensate for the dimensionality reduction of the output matrix A compared to the input matrix I caused by the convolution, so that the output matrix A can have the same dimensions as the input matrix I

[0090] exist Figure 2 The equations described in mainly apply when no padding is applied. In this case the input matrix I is equal to the modified filter matrix The corresponding convolution operation performed is also referred to as "effective convolution". The neural network 10 can be trained in the usual way, i.e. similar to conventional methods, wherein the input matrix I can be normalized with a first normalization variable v and a second normalization variable w, such as a standard deviation σ and a mean value. Optionally, however, the input matrix I of a convolutional layer downstream of the convolutional layer that performs the method according to the invention can be converted into an input matrix of higher dimension by adding zero-valued entries, so-called zero padding.

[0091] Therefore, if no entries are added to the convolutional layers 12a-c that perform the method of the present invention and the input matrix I (e.g., training images or training data elements) provided to the convolutional layers 12a-c during the training of the neural network 10, the provided input matrix I can also be provided to the convolutional layers 12a-c during the operation of the trained neural network 10 without adding entries to the input matrix I. Therefore, after the neural network 10 is trained, that is, during the operation of the trained neural network 10, the input matrix I provided can be provided to the convolutional layers 12a-c according to the input matrix I provided in the training process. Figure 2 The method of the present invention described in the embodiment of the present invention performs convolution operation and shift operation.

[0092] If padding is performed during the training of the neural network 10 and / or entries are added to the input matrix I of the convolutional layer 12a-c implementing the method of the present invention during training, it may be necessary to also add elements to the input matrix I of the convolutional layer 12a-c during the operation of the trained neural network 10 in order to convert the input matrix I into an input matrix of higher dimension. If the number of added entries is chosen so that the dimensions are the same after the convolution, this is often also referred to as "same convolution". Depending on the hardware implementing the neural network 10, "zero" entries (so-called zero padding) and / or entries with non-zero constant values ​​(so-called non-zero padding) can be added to the input matrix I.

[0093] During the training of the neural network 10, the input matrix can be normalized in a conventional manner, for example, by multiplying the input matrix by a first normalization parameter v and by adding to a second normalization parameter w. Furthermore, during the training, entries with zero values ​​can also be added to the input matrix I of the convolutional layers 12a-c implementing the method according to the invention (zero padding). After the training, the trained neural network 10 can be implemented on hardware and / or can be ported to hardware, and the modified filter matrix can be used in the corresponding convolutional layers 12a-c. and the modified shift matrix In this case, in step S2 ′, entries are added to the input matrix I during the operation of the trained neural network 10 , which entries can each have a negative ratio of the second normalization variable v to the first normalization variable w (non-zero filling).

[0094] If during the training of the neural network 10 entries with zero values ​​are added to the input matrix I of the convolutional layers 12a-c implementing the method according to the invention in a conventional manner by subtracting the mean value μ from the entries of the input matrix I and normalizing the input matrix by dividing by the standard deviation σ (zero padding), the trained neural network 10 can be implemented on hardware and / or ported to hardware after training, and the modified filter matrix can be used in the corresponding convolutional layers 12a-c and the modified shift matrix In this case, in step S2 ′, entries are added to the input matrix I during the operation of the trained neural network 10 , which entries each have the mean value μ ascertained in step S1 (non-zero filling).

[0095] Alternatively, during the training of the neural network 10, the input matrix can be normalized in a conventional manner by multiplying the input matrix by a first normalization parameter v and by adding to a second normalization parameter w, and during the training, entries can be added to the input matrix I of the convolutional layer 12a-c implementing the method of the present invention, the entries each having the value of the second normalization parameter w (non-zero padding). The trained neural network 10 can be implemented on hardware and / or ported to hardware after training, and the modified filter matrix can be used in the corresponding convolutional layer 12a-c and the modified shift matrix In this case, in step S2 ′, entries are added to the input matrix I during the operation of the trained neural network 10 , which entries each have a zero value (zero filling).

[0096] If during the training of the neural network 10, the input matrix I is normalized in a conventional manner by subtracting the mean value μ from the entries of the input matrix I and by dividing by the standard deviation σ, and during the training, entries are added to the input matrix I of the convolutional layers 12a-c implementing the method of the present invention, wherein the entries each have a value that is a negative ratio of the mean value μ to the standard deviation σ, i.e., a value of -μ / σ (non-zero padding), then the trained neural network 10 can be implemented on hardware and / or ported to hardware after training, and the modified filter matrix can be used in the corresponding convolutional layers 12a-c and the modified shift matrix In this case, in step S2 ′, entries are added to the input matrix I during the operation of the trained neural network 10 , which entries each have a zero value (zero filling).

[0097] For example, the “addition of entries to the input matrix I” described above may be performed only in the input layer 12a of the neural network. For all other convolutional layers 12b, 12c, the same padding as during training may be used during the operation of the trained neural network 10. Alternatively, however, the “addition of entries to the input matrix I” described above may be performed in each of the convolutional layers 12a-c.

[0098] Figure 4 A flow chart is shown for illustrating the steps of a method for operating a neural network 10 according to an exemplary embodiment of the present invention. Figure 4 The method shown in the embodiment includes, in particular, training a neural network 10 and running the trained neural network 10. The method of the invention is exemplarily performed only in the input layer 12a below, but optionally, the method of the invention can also be implemented in any convolutional layer 12a-c. If not otherwise specified, Figure 4 The method described in has the same elements, features and steps as the method described in the previous figures.

[0099] In step S0 upstream of "training the neural network 10", the mean value μ and the quasi-deviation σ are determined based on the training data set, as in Figure 2 Therefore, step S0 may also include a sub-step of providing the training data set in the storage device 11, for example. Alternatively, as in Figure 2 As described in , the mean value μ and the quasi-deviation σ can also be calculated based on the input matrix I.

[0100] During the training of the neural network 10, training data elements, for example training images, are provided to the neural network 10 or the input layer 12a in step S1. These training data elements can be normalized in step S1 by subtracting the mean value μ from the entry of each training data element and dividing the subtraction result by the standard deviation σ. As described in the above figures, these training data elements can be like an input matrix I for the input layer 12a.

[0101] In an optional step S2, the input matrix I or the training data elements can be transformed into a higher dimensional input matrix by adding entries with zero values ​​or by adding entries with values ​​-μ / σ or higher dimensional training data elements, such as in Figure 3If no entries are added to the input matrix I of the input layer 12a, entries having, for example, zero values ​​may be added to the output matrix A of the input layer 12a to compensate for the dimensionality loss caused by the convolution after the input layer 12a.

[0102] In step S3, the actual training of the neural network 10 can then be carried out. As a result of the training, entries of the original filter matrix f and the original shift matrix b can be determined, which can represent the trained weights, parameters and / or parameter values ​​of the neural network 10 as it were.

[0103] The trained neural network 10 can be implemented on hardware and / or ported to hardware to run the trained neural network 10. In step S4, the unnormalized input data elements I, input matrix I and / or input image I are provided to the neural network 10 and / or input layer 12a for interpretation and / or classification.

[0104] If in optional step S2 an element with the value zero is added to the training data element, then in step S4 an entry with the value μ is added to the input matrix I. If in optional step S2 an entry with the value −μ / σ is added to the training data element, then in step S4 an entry with the value zero is added to the input matrix I. If in optional step S2 no entry is added to the input matrix I.

[0105] In step S5, the modified filter matrix is ​​applied and the modified shift matrix The input matrix I of the input layer 12a is converted into the output matrix A, as shown in Figure 2 and Figure 3 As detailed in .

[0106] The neural network 10 may further propagate the output matrix A through other layers 12b, 12c and output an output item 14. The output item 14 may be, for example, an expected object category, a semantic segmentation, and / or any other parameter.

[0107] In addition, it should be pointed out that "comprising" does not exclude other elements and "a" does not exclude a plurality. It should also be pointed out that the features described with reference to one of the above embodiments can also be used in combination with other features of other embodiments described above. The reference signs in the claims should not be regarded as limiting.

Claims

1. A computer-implemented method for running an artificial neural network (10), wherein: An image is provided to the artificial neural network, wherein the artificial neural network outputs a probability and / or a probability value that can assign an object contained in the image to a specific object classification and / or object category, or wherein the artificial neural network outputs a semantic segmentation of the image, wherein the artificial neural network (10) has at least one convolutional layer (12a-c), which is configured to transform an input matrix (I) of the convolutional layer (12a-c) into an output matrix (A) based on a convolution operation and a shift operation, and the method comprises the following steps: determining at least one first normalization variable and a second normalization variable based on the entries of the input matrix (I) and / or based on the training data set of the artificial neural network (10), Characterized in that the method further comprises the following steps: Get the modified filter matrix The modified filter matrix is ​​obtained by dividing the original filter matrix (f) of at least one convolutional layer (12a-c) by the first normalization parameter. Get the modified shift matrix The modified shift matrix is ​​obtained by multiplying the original filter matrix (f) of at least one convolutional layer (12a-c) by the negative ratio of the second normalized parameter to the first normalized parameter, and then adding it to the original shift matrix (b). Based on the modified filter matrix and the modified shift matrix The input matrix (I) is transformed into the output matrix (A), wherein the input matrix (I) is combined with the modified filter matrix Convolution and then the modified shift matrix Add it to the result of the convolution to obtain the output matrix (A).

2. The method according to claim 1, wherein: The first normalized parameter is a parameter related to the standard deviation (σ) and / or is the standard deviation; and / or The second normalization variable is a variable related to the mean value (μ) and / or is the mean value (μ).

3. The method according to claim 2, wherein: The second normalization parameter is the mean value (μ) of the entries of the input matrix (I); and / or, The first normalization parameter is the standard deviation (σ) of the entries of the input matrix (I).

4. The method according to claim 2, wherein: The first normalization variable is a standard deviation (σ), and the second normalization variable is a mean value (μ), wherein the standard deviation (σ) and the mean value (μ) are determined based on training images of a training data set of the artificial neural network (10).

5. The method according to any one of claims 2 to 4, further comprising: By adding entries to the input matrix (I), the input matrix (I) is converted into a higher dimensional input matrix in, During the inference of the artificial neural network (10), the added entries each have a second normalization variable, wherein the second normalization variable is a mean value (μ); and / or During the training of the artificial neural network, the entries added each have the value zero.

6. The method according to any one of claims 2 to 4, further comprising: By adding entries to the input matrix (I), the input matrix (I) is converted into a higher dimensional input matrix in, If the artificial neural network is trained, the added entries each have a value of zero; and / or During the training of the artificial neural network, the entries added each have the negative value of the ratio of the second normalization variable to the first normalization variable, wherein the second normalization variable is the mean value (μ) and the first normalization variable is the standard deviation (σ).

7. A method according to any one of claims 1 to 4, wherein The at least one convolutional layer (12a-c) is an input layer (12a) of the artificial neural network (10).

8. The method according to claim 7, wherein: The modified filter matrix is ​​used only in the input layer (12a) of the artificial neural network (10). and the modified shift matrix 9. The method according to any one of claims 1 to 4, wherein: The artificial neural network (10) has a plurality of convolutional layers (12a-c), each of which is configured to convert an input matrix (I) of each convolutional layer (12a-c) into an output matrix (A) of the corresponding convolutional layer (12a-c); wherein: For each convolutional layer (12a-c), obtain the modified convolution matrix and the modified shift matrix and, Each input matrix (I) of each convolutional layer (12a-c) is modified by applying the modified filter matrix obtained for the corresponding convolutional layer (12a-c) and the modified shift matrix obtained for the corresponding convolutional layer (12a-c) Transformed into the output matrix (A).

10. A computer-implemented artificial neural network (10) arranged to operate according to the method according to any one of the preceding claims, wherein: The artificial neural network has at least one convolutional layer (12a-c), which is configured to transform an input matrix (I) of the convolutional layer (12a-c) into an output matrix (A) based on a convolution operation and a shift operation.

Images