Method for generating convolutional neural network with invariant integral layers for object classification

Abstract

A method for generating a trained convolutional neural network with at least one invariant integral layer to classify objects of a digital image of an environment of a mobile platform using a plurality of training cycles is proposed, wherein each training cycle has the steps of providing a digital image of an environment of a mobile platform with at least one object; providing a reference image assigned to the digital image, wherein the at least one object is labeled in the reference image; providing the digital image as an input signal to the convolutional neural network with at least one invariant integral layer; and adapting the convolutional neural network with at least one invariant integral layer so as to minimize a deviation of a classification of the at least one object of the digital image from the corresponding assigned reference image.

Description

Technical Field

[0001] The present invention relates to a method for generating a trained convolutional neural network with at least one invariant integral layer to classify objects in digital images of a mobile platform environment. Background Technology

[0002] To control at least a partially automated mobility platform, deep neural networks (DNNs) can be used for various tasks, including object classification, such as identifying and classifying surrounding road users, i.e., object recognition. The results of object recognition can then be used to control the partially automated mobility platform.

[0003] DNNs must be trained using large amounts of labeled data. However, labeling data is intensive, laborious, and time-consuming. Therefore, it is important to improve data efficiency during DNN training to reduce the amount of labeled data required in order to achieve the desired performance. Summary of the Invention

[0004] According to aspects of the invention, methods for generating trained convolutional neural networks with at least one invariant integral layer for classifying objects, methods for classifying objects in two-dimensional digital images, applications of convolutional neural networks for object classification, applications of convolutional neural networks for providing control signals, convolutional neural networks, apparatus, computer program products, and machine-readable storage media are proposed, as described in the independent claims. Advantageous designs are the subject of the dependent claims and the description that follows.

[0005] A method is proposed for generating a trained convolutional neural network with at least one invariant integral layer using multiple training epochs to classify objects in digital images of a mobile platform environment, wherein each training epoch has the following steps:

[0006] In one step of this training cycle, a digital image of the mobile platform's environment, containing at least one object, is provided. In the next step of this training cycle, a reference image is provided, assigned to the digital image, in which the at least one object is labeled. In the next step of this training cycle, the digital image is provided as input to a convolutional neural network with at least one invariant integral layer. In the next step of this training cycle, the convolutional neural network with at least one invariant integral layer is matched to minimize the deviation between the classification and the corresponding assigned reference image when classifying the at least one object in the digital image.

[0007] To classify objects in digital images, convolutional neural networks can be used. To train such a network, reference images, i.e., images in which these objects are labeled, are used.

[0008] Neural networks provide a framework for a variety of different algorithms used in machine learning, collaboration, and processing complex data inputs. These neural networks learn to perform tasks based on examples, and are typically not programmed with task-specific rules.

[0009] This type of neural network is based on the aggregation of different units or nodes, which are called artificial neurons. Each connection can transmit a signal from one artificial neuron to another. The artificial neuron that receives the signal can process it and then activate other artificial neurons connected to it.

[0010] In conventional implementations of neural networks, the signals at the connections of artificial neurons are real numbers, and the output of an artificial neuron is calculated as a nonlinear function of the sum of its inputs. These connections typically have weights that adapt as learning progresses. These weights increase or decrease the signal strength at the connection. Artificial neurons may have thresholds, such that an output signal is only generated when the total signal exceeds the threshold. Typically, multiple artificial neurons are layered. Different layers may perform different types of transformations on their inputs. The signal may travel from the first layer (the input layer) to the last layer (the output layer) after multiple passes through these layers.

[0011] This artificial neural network architecture can be based on a multi-layer perceptron (MLP) network. The MLP belongs to the family of artificial feed-forward neural networks. In principle, an MLP consists of at least three neuron layers: an input layer, intermediate layers (hidden layers), and an output layer. This means that all neurons in the network are layered, with each neuron in one layer always connected to all neurons in the next layer. There are no connections to the previous layer and no connections that skip a layer. In addition to the input layer, there are different neuron layers with non-linear activation functions that are connected to neurons in the next layer. Deep neural networks can have four such intermediate layers.

[0012] Such convolutional neural networks must be trained for their specific tasks. Here, each neuron in the corresponding architecture of the neural network receives, for example, random initial weights. Then, input data is fed into the network, and each neuron weights the input signal according to its weights and passes the result to the next layer of neurons. The output layer then provides the final result. The magnitude of the error can be calculated, as well as the share of that error held by each neuron, and then the weights of each neuron are changed in a direction that minimizes the error. This process is then recursively repeated, remeasuring the error and adjusting the weights until the error falls below a predetermined limit.

[0013] An invariant integral layer is a layer in a neural network that determines the group mean matrix by integrating the transformation group of the output feature map of the convolutional layer with an invariant function. This invariant integral layer is described in more detail below. Here, when matching a convolutional neural network with this invariant integral layer, the parameters of this invariant integral layer are also matched.

[0014] By using a convolutional neural network with novel convolutional neural network layers that perform invariant integrals, geometric prior knowledge is incorporated by explicitly ensuring invariant properties of the objects to be classified, such as rotation invariance and / or translation invariance and / or scaling invariance and / or affine transformation invariance. This improves data efficiency because less data is needed to train the neural network to achieve the desired accuracy, and / or accuracy is improved with the same amount of data.

[0015] Therefore, this method can reduce the time and cost of labeling training data because it reduces the amount of labeled data required to achieve the desired performance of the neural network constructed in this way.

[0016] Here, the network layer with invariant integrals described herein is differentiable so that matching of the convolutional neural network can be achieved by means of backpropagation.

[0017] In this general description of the invention, the order of the method steps is now described to make the method easy to understand. However, those skilled in the art will recognize that multiple method steps in these steps can also be traversed in another order and yield the same or corresponding results. In this sense, the order of these method steps can be changed accordingly. Some features are equipped with numbers to improve readability or make the assignment more explicit, but this does not imply the existence of specific features.

[0018] According to one aspect of the method, it is proposed that a convolutional neural network with at least one invariant integral layer has at least one convolutional layer and at least one invariant integral layer, and the invariant integral layer determines the group mean matrix by means of the integral of the transformation group of the output feature map of the convolutional layer through the invariant function of the convolutional layer.

[0019] Here, the group mean matrix can be determined according to the following formula:

[0020]

[0021] Where x is a 3D tensor that includes a feature map with dimensions of height x width x channels; f is a vector consisting of at least one function; g is a transformation of the feature map, and g describes a single transformation from the transformation group G.

[0022] The group mean A is defined by the integral of the invariant function f over the transformation g ∈ G acting on the input space x.

[0023] That is, the invariant integral layer is used on the feature space of the group equivariant neural network, which is computed by means of the equivariant convolutional layer.

[0024] Therefore, expertise on invariants such as rotation invariance and / or translation invariance and / or scaling invariance and / or affine transformation invariance can be introduced through the transformation group G with respect to the existing problem.

[0025] Group theory forms the basis of invariant integrals. A group G is a mathematical abstraction consisting of a set of elements, and operations on these elements are performed under the axioms of closure, associativity, neutrality, and invertibility.

[0026] If the exact relationship between the transformation g ∈ G of the input x (feature map is multiplicative; feature space) of a function f and the corresponding transformation g' ∈ G of the function's output can be determined, then the function is considered equivariant with respect to the transformation group G. The apostrophe is intended to indicate that the transformation of the output is not exactly the same as the transformation of the input x;

[0027] Here, the mathematical definition of isovariance is derived as follows:

[0028]

[0029] Where f: a vector consisting of at least one invariant or equivariant function; f(x): an equivariant feature space; x: a 3D tensor comprising a feature map (height x width x channels); g: a transformation of the feature map; g': a transformation of the function's output value.

[0030] For equivariant functions, the output transformation is predetermined; for invariant functions, the output transformation is an identity (e.g., 1 in multiplication). Therefore, the mathematical definition of invariance is:

[0031]

[0032] An example of an equivariant function is a convolutional layer, which is equivariant to translation. In practice, there is also interest in enumerating the invariance of other transformation groups besides translation, such as rotation, scaling, or affine transformations. Invariant integrals are algorithms used to construct a complete feature space with respect to symmetric transformations.

[0033] The feature space is defined as complete if all equivalent modes with respect to the transform group G are mapped to the same point in the feature space, while all different modes are mapped to different points. This means that the entire feature space is invariant with respect to the transform g ∈ G of the input signal.

[0034] This can be achieved by normalizing the input features, using the channel-wise minimum determined during training.

[0035] x i = max (ε, x i - x min + 1), where 0 < ε ≪ 1

[0036] This prevents the vanishing of the exponent and gradient of the input in the invariant integral layer, thus enabling the use of the backpropagation algorithm.

[0037] According to one aspect of the present invention, it is proposed that the invariant function f is a monomial of the output feature map of the convolutional layer.

[0038] Here, a monomial is a special form of a polynomial that contains only one term and is defined as follows:

[0039]

[0040] Where x: a 3D tensor containing a feature map, with dimensions of height x width x channels; f: a vector consisting of at least one function; m: a monomial; g: a transformation of the feature map, and g describes a single transformation from the transformation group G; K: the order of the monomial, i.e., the size of the input feature. : A monomial index that is adapted to make the neural network match.

[0041] For an invariant function f, the set of all possible monomials m(x) is a good choice for generating the complete feature space.

[0042] Group theory shows that when using monomials in invariant integrals, transformation-invariant features are suppressed by the invariant integral, and the differences between transformation-invariant features and other features are amplified.

[0043] The upper limit of the number of all possible monomials is However, not all of them are calculated because it is a very high value. The coefficient bi is trained during the training of the neural network.

[0044] According to one aspect of this method, it is proposed that the transformation is rotation and / or translation.

[0045] Here, the group mean matrix or group mean vector can be determined according to the following formula:

[0046]

[0047] Where x: a 3D tensor comprising a feature map, with dimensions of height x width x channels; f: a vector consisting of at least one function; g: a transformation of the feature map, and g describes a single transformation from the transformation group G; K: the order of the monomial; u, v: pixel positions on the feature map; : Unimetic distance; : A monomial exponent that is adapted to make the neural network match; and : Rotation angle.

[0048] According to one aspect of this method, it is proposed that the transformation is rotation and / or translation and / or scaling.

[0049] Here, the group mean matrix or group mean vector can be determined according to the following formula:

[0050]

[0051] Where x: a 3D tensor comprising a feature map, with dimensions of height x width x channels; f: a vector consisting of at least one function; g: a transformation of the feature map, and g describes a single transformation from the transformation group G; K: the order of the monomial; u, v: pixel positions on the feature map; : Unimetic distance; : A monomial exponent that is adapted to make the neural network match; : Rotation angle; and s: Scaling factor.

[0052] The invariant integral layer computes the group mean to explicitly constrain the invariant feature space using a set of monomials m(x), for example, by integrating the group over two-dimensional translations (u, v), scaling (s), and rotations φ. Here, the group mean can be computed using monomials of an arbitrary transformation group G.

[0053] Advantageously, it can be deduced from this invariant integral layer that object recognition is invariant to the rotation, scaling, and translation of the relevant objects. Thus, from a bird's-eye view perspective, for example, when classifying vehicles as objects, their orientation in the digital image is irrelevant.

[0054] According to one aspect of this method, the following is proposed: invariant integral It is determined by the relationship described in Formula 1:

[0055] (1)

[0056] The parameters are as described above: K: order of the monomial; u, v: pixel positions on the feature map; : Unimetic distance; : Monomial exponent; x: 3D tensor, which includes feature maps; and : Rotation angle.

[0057] According to one aspect of this method, the following is proposed: invariant integral It is determined by the relationship described in Formula 2:

[0058] (2)

[0059] The parameters are as described above: K: order of the monomial; u, v: pixel positions on the feature map; : Unimetic distance; : Unimetic exponent; x: 3D tensor, which includes a feature map; : Rotation angle; and s: Scaling factor.

[0060] According to one aspect of this method, it is proposed to parameterize the monomial, randomly generate the parameters of the monomial, and select optimized parameters from the randomly generated parameters.

[0061] The parameters of the monomial, namely the order, distance, and exponent, are selected using an iterative method.

[0062] If M monomials are needed (e.g., M = 5), then M+1 (e.g., 6) parameter combinations are randomly selected. For each of the M+1 possible combinations of the M parameters, a linear classifier and its validation accuracy are computed. Parameter combinations that were not used in the best-performing classifier are discarded and replaced with new, randomly selected combinations. This process is repeated a certain number of times (e.g., 100 times) to find the optimal M parameter combinations.

[0063] This method of selecting the parameters of a monomial, namely the order, distance, and exponent, is simple and fast.

[0064] Alternatively, pruning methods are used to select the parameters of the monomials: order, distance, and exponent. If M monomials are needed (e.g., M = 5), N > M (e.g., 100) parameter combinations are randomly selected. The neural network is pre-trained using N monomials, and the pruning method directly or iteratively determines the N parameter combinations that contribute most to the network's correct classification. The contribution to classification can be determined, for example, by summing the absolute values ​​of the weights at the output of the monomials. These methods for selecting the parameters of the monomials—order, distance, and exponent—are simple, fast, and enable direct end-to-end training of the network.

[0065] According to one aspect of this method, it is proposed that a convolutional neural network has multiple convolutional layers, which highlight features important for object classification before the invariant integral layer determines the group mean matrix. Thus, features important for object classification are highlighted.

[0066] This method of extracting features important to the classification reduces the computational cost of calculating invariant integrals. For example, information about object translation and / or color that is not important to the classification is discarded.

[0067] A method for classifying objects in digital images, especially two-dimensional digital images, in mobile platform environments is proposed. This method has the following steps:

[0068] In the first step, a digital image of the mobile platform's environment is provided. In the next step, using a trained convolutional neural network with at least one invariant integral layer, the digital image is classified according to multiple object categories based on one of the methods described above and the digital image as the input signal to the convolutional neural network with at least one invariant integral layer. The convolutional neural network with at least one invariant integral layer has been trained to classify objects in the digital reference images according to the multiple object categories using multiple digital reference images and the classified objects in the corresponding digital reference images; and the objects in the digital image are also classified.

[0069] This object classification method is particularly suitable for at least partially automated mobile platforms, such as autonomous vehicles.

[0070] An application of a convolutional neural network with at least one invariant integral layer for classifying objects in digital images in a mobile platform environment is proposed, the convolutional neural network being trained according to one of the methods described above.

[0071] Since the invariant integral layer is part of a convolutional neural network, it offers the advantage of higher accuracy, as explained above, or the need for less labeled data to meet the specific accuracy requirements of a classification task.

[0072] An application is proposed for classifying objects in digital images of a mobile platform environment using a convolutional neural network with at least one invariant integral layer trained according to one of the above methods, wherein control signals for manipulating at least a partially automated mobile platform and / or alarm signals for alerting occupants of at least a partially automated mobile platform are provided based on the classification results.

[0073] The term "based on" should be understood broadly regarding the characteristic of "providing control signals based on the results of this classification." This term should be understood as such that: based on the results of this classification, any determination or calculation of the control signal is used, where this does not exclude the use of other input parameters for such determination of the control signal. This correspondingly applies to the provision of alarm signals.

[0074] Advantageously, from the classification of objects in the environment of at least part of the automated mobile platform, the possibility of controlling the mobile platform based on the classified and detected objects can be derived, for example, in order to avoid accidents.

[0075] According to one aspect, a convolutional neural network with at least one invariant integral layer is described, which is trained according to one of the methods described above.

[0076] Using this convolutional neural network, the classification task described can be easily integrated into different systems.

[0077] According to one aspect, an apparatus is described having a convolutional neural network with at least one invariant integral layer, the convolutional neural network being trained according to one of the methods described above.

[0078] Using this device, the corresponding methods can be easily integrated into different systems.

[0079] According to one aspect, a computer program is described that includes instructions, which, when executed by a computer, cause the computer to perform one of the methods described above. This computer program enables the described methods to be used in different systems.

[0080] A machine-readable storage medium is described, on which the aforementioned computer program is stored. This machine-readable storage medium allows for the transfer of the aforementioned computer program.

[0081] A mobile platform can be understood as a mobile system and / or driver assistance system that is at least partially automated. Examples can be partially automated vehicles or vehicles with driver assistance systems. That is, in this context, a system that is at least partially automated includes a mobile platform with at least partial automation functionality, but a mobile platform also includes vehicles and other mobile machinery, including driver assistance systems. Other examples of mobile platforms can be: driver assistance systems with multiple sensors; mobile multi-sensor robots, such as robotic vacuum cleaners or lawnmowers; multi-sensor monitoring systems; manufacturing machines; personal assistants; or access control systems. Each of these systems can be fully or partially automated.

[0082] An apparatus is described, which is configured to perform one of the methods described above.

[0083] With the aid of such a device, it is possible to make the above method, for example, be used to embed into a mobile platform.

[0084] A computer program is described, comprising instructions that, when executed by a computer, cause the computer to perform one of the methods described above.

[0085] With the help of such a computer program, the above methods can be used on mobile platforms in a simple way.

[0086] This describes a machine-readable storage medium on which the aforementioned computer program is stored.

[0087] The aforementioned computer program product can be transferred using this machine-readable storage medium. Attached Figure Description

[0088] Reference to embodiments of the present invention Figure 1 This will be shown and elaborated upon in more detail below. Among them:

[0089] Figure 1 A method is shown for classifying objects in digital images of a mobile platform environment using a trained convolutional neural network with at least one invariant integral layer. Detailed Implementation

[0090] Figure 1 The illustration schematically shows how image 110 is transformed into feature map 120 using multiple convolutional layers of a neural network with invariant integral layers in step S1. In step S2, invariant integration is performed on feature map 120, and a group mean vector 130 is created. Then, subsequent fully connected layers of the convolutional neural network with invariant integral layers utilize step S3 to derive a classification vector 140, from which the corresponding classification result of the object can be read.

[0091] The architecture of invariant neural networks used for classification typically consists of two parts. The first part is a sequence of equivariant layers, which are used to downsample the signal to a smaller resolution, with the aim of obtaining the desired data and discarding redundant information. Due to the equivariance of the layers, the output of the convolutional layers will change predictably when the input of the layers is transformed in a predetermined way (e.g., rotation).

[0092] In the second part, the equivariant feature space is transformed into an invariant feature space that remains unchanged during input transformations. For this purpose, an invariant integral layer is used. Based on the features obtained in this way, multiple fully connected layers are used to obtain the classification result.

[0093] Table 1 lists examples of invariant neural network architectures. The part used to obtain isovariant features consists of d consecutive isovariant convolutional layers (e.g., d=6) with kernel sizes from N1 to N_d (e.g., N1=24, N2=32, ...). ReLU, batch normalization, and max pooling are used between these layers.

[0094] The invariant integral layer uses five polynomials to transform the equivariant feature space into a one-dimensional invariant feature vector. Building upon this, k fully connected blocks with neurons M1 to M_k (e.g., M1 = 90, M2 = 90) are used to obtain classification scores from the invariant features.

[0095] Table 1: Structure of Invariant Networks Used for Classification Input signal: 28x28x1 image 24 equivariant conv 9x9. ReLU. BatchNorm (batch normalization). 32 isovariant transformations (7x7). ReLU. BatchNorm. MaxPool (maximum pool), stride (Sride) 2. 36 isovariant transformations (7x7). ReLU. BatchNorm. 36 isovariant transformations (7x7). ReLU. BatchNorm. MaxPool, stride 2. 64 isovariant transformations (7x7). ReLU. BatchNorm. 96 isovariant transformations (5x5). ReLU. BatchNorm. ReLU. Invariant integral layer, 5 monomials. Fully connected layer, 90 neurons. ReLU. BatchNorm. Fully connected layer, 90 neurons. ReLU. BatchNorm. Fully connected layer, 10 neurons. Softmax.

[0096] Supervised training of a neural network uses a training set of input signals from a convolutional neural network with invariant integral layers (CNN) and a desired output signal (commonly known as "Ground Truth"). The desired output signal can be, for example, a classification label. Based on the actual output values ​​of the CNN and the desired output signal, a so-called loss function to be optimized is calculated; that is, the parameters or weights of the CNN are calculated to optimize this loss function. This optimization is performed by calculating the gradient of the loss function using these weights, which can be achieved by iteratively applying the chain rule. This method is also known as backpropagation.

[0097] In order to use invariant integral layers within deep neural networks and to make preceding layers optimizable using backpropagation, the monomials must be differentiable with respect to their input data. This differentiation is calculated as follows:

[0098]

[0099] As you can see: the value must be This prevents the gradient from "vanishing" (i.e. becoming 0), which would hinder the training of the preceding layers.

[0100] It can also indicate: for monomial exponents The derivative is also defined. Therefore, these monomial exponents can be optimized during the training process of the neural network.

[0101]

[0102] Importantly, all values ​​are... The reason is that otherwise, the logarithm would not be defined, and the gradient of the product would vanish again.

[0103] Therefore, the feature map shifting mentioned above is used:

[0104] .

Claims

1. A method for generating a trained convolutional neural network with at least one invariant integral layer using multiple training epochs to classify objects in digital images of a mobile platform environment, wherein each training epoch comprises the following steps: A digital image (110) having at least one object in a mobile platform environment; A reference image is provided to the digital image, wherein at least one object is marked in the reference image; The digital image (110) is provided as the input signal to a convolutional neural network with at least one invariant integral layer; and A convolutional neural network with at least one invariant integral layer is matched to minimize the deviation of the classification from the corresponding assigned reference image when classifying at least one object in the digital image. wherein A convolutional neural network with at least one invariant integral layer (130) has at least one convolutional layer and at least one invariant integral layer (130), and the invariant integral layer (130) determines the group mean matrix by means of the integral of the transformation group of the output feature map of the convolutional layer through the invariant function of the convolutional layer. In this process, geometric prior knowledge is incorporated by using a convolutional neural network with an invariant integral layer (130) that performs invariant integrals, in order to explicitly ensure invariant properties of the objects to be classified, wherein the invariant properties are rotation invariance and / or translation invariance and / or scaling invariance and / or affine transformation invariance.

2. The method according to claim 1, wherein the invariant function is a monomial of the output feature map (120) of the convolutional layer.

3. The method according to claim 1 or 2, wherein the transformation is rotation and / or translation.

4. The method of claim 1 or 2, wherein the invariant integral is determined by the relationship described in Equation 1 : （1） Where K: order of the monomial; u, v: pixel positions on the feature map; : Unimetic distance; : Univariate index; x: A 3D tensor that includes a feature map; and : Rotation angle.

5. The method of claim 2, wherein the monomial is parameterized, and parameters of the monomial are randomly generated, and optimized parameters are selected from the randomly generated parameters.

6. The method of claim 1 or 2, wherein the convolutional neural network has a plurality of convolutional layers, which highlight features important for the classification of the object before the invariant integral layer determines the group mean matrix.

7. A method for classifying objects in a two-dimensional digital image in a mobile platform environment, the method comprising the following steps: Provides a digital image (110) of the environment of the mobile platform; Using a trained convolutional neural network with at least one invariant integral layer (130), according to any one of claims 1 to 6 and the digital image (110) as the input signal of the convolutional neural network with at least one invariant integral layer (130), the digital image is classified according to a plurality of object categories, wherein the convolutional neural network with at least one invariant integral layer (130) has been trained with the aid of a plurality of digital reference images and the classified objects of the corresponding digital reference images to classify the objects of the digital reference images with respect to the plurality of object categories; and to classify the objects of the digital image.

8. An application of a convolutional neural network with at least one invariant integral layer (130) for classifying objects in digital images in a mobile platform environment, said convolutional neural network being trained by the method according to any one of claims 1 to 6.

9. An application of a convolutional neural network with at least one invariant integral layer (130) trained according to any one of claims 1 to 6 for classifying objects in a digital image (110) of a mobile platform environment, wherein control signals for manipulating at least a partially automated mobile platform and / or alarm signals for alerting occupants of at least a partially automated mobile platform are provided based on the results of the classification.

10. An apparatus having a convolutional neural network with at least one invariant integral layer (130), the convolutional neural network being trained by the method according to any one of claims 1 to 6.

11. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

12. A machine-readable storage medium having a computer program stored thereon, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images