Method for training at least one neural network for a system to detect an object

Abstract

The disclosure includes a method for training at least one neural network for a system for detecting an object, comprising the following steps: Providing initial measurement data from a data source Generating a second set of measurement data by adding a direction-dependent input parameter to the first set of measurement data. Defining at least one feature for the at least one neural network, wherein the at least one feature is assigned either to a first feature class or to at least a second feature class. Feeding the first measurement data and the second measurement data into the at least one neural network Assigning at least one feature of the at least one neural network, taking into account the direction-dependent input parameter, where at least one feature of the output corresponds to either the first feature class or the second feature class Check whether the assignment of at least one feature of at least one neural network corresponds to a predefined quality criterion, using a loss function that determines whether there is a deviation of the determined assignment from the predefined quality criterion. Training the at least one neural network if the detected deviation does not meet the specified quality criterion and to minimize loss in the detected deviation from the determined assignment.

Description

[0001] The present invention relates to a method for training at least one neural network for a system for detecting an object. State of the art

[0002] Deep neural networks are currently used in a variety of environmental perception algorithms, such as multimodal object detection. For 3D object detection in particular, it is crucial that the network output is invariant or equivariant with respect to translation or rotation. In physics, such invariances or equivariances are also referred to as symmetries. Since the objects to be detected in the automotive sector are usually upright on the road, the group of 2D rotation matrices (SO(2)) is primarily relevant, as translations are already covered by the architecture of convolutional neural networks.

[0003] The Fig. Figures 1a to 1d show an example of outputs that are invariant or equivariant with respect to SO(2), also called scalars or vectors, respectively. Fig. 1a and Fig. 1b Scalars are moved to a different location by applying a rotation, but retain their value. In the Fig. 1c and Fig. In 1d, vectors are moved to a different location and change their values ​​(orientation).

[0004] These two types of outputs or features (scalars and vectors) are precisely the two relevant representations of SO(2). The feature maps of a neural network should change—invariantly or equivariantly—depending on the encoded feature, as in the example above. To ensure this, the current state of the art employs the approach of augmentation. This method typically involves augmenting the input and output (e.g., object boxes) using global transformations. By repeatedly augmenting the input data with rotations and simultaneously transforming the labels, stochastic training forces equivariant or invariant behavior in the network. This requires no changes to standard Convolutional Neural Network (CNN) architectures and is used in virtually all common 3D object detection methods. One advantage of this method is that it optimally utilizes the capacity of the neural network.The disadvantage of this approach, however, is that not all augmentation is possible, depending on the sensors used. While augmentation is generally possible for lidar or radar data, for example, this method cannot be so easily applied to camera images or data from a fusion architecture. The neural network then often quickly loses its equivariant or invariant behavior.

[0005] Another use case is when the feature map needs to be corrected for self-motion. Furthermore, while invariant or equivariant outputs are obtained, practically no statement can be made about the invariance or equivariance of the features in the respective layers of the neural network. In other words, it is unknown whether a feature is a scalar or, together with another feature, roughly represents a vector.

[0006] The object of the present invention is therefore to provide a solution by which a neural network can be trained efficiently and reliably for a system in order to detect an object in the environment of that system. Disclosure of the invention

[0007] This task is solved by a method for classifying an object to be detected with at least one sensor having the features of the independent claim.

[0008] According to a first aspect, the disclosure relates to a method for training at least one neural network for a system for detecting an object, which comprises the following steps: The first step involves providing initial measurement data from a data source.

[0009] In a second step, second measurement data is generated by providing the first measurement data with a direction-dependent input parameter.

[0010] In a third step, at least one feature is defined for the at least one neural network, whereby the at least one feature is assigned either to a first feature class or to at least a second feature class.

[0011] In a fourth step, the first measurement data and the second measurement data are fed into the at least one neural network.

[0012] In a fifth step, at least one output of the at least one neural network is assigned, taking into account the direction-dependent input parameter, whereby the at least one output corresponds to either the first feature class or the second feature class.

[0013] In a sixth step, a check is performed to see if the assignment of the at least one output of the at least one neural network corresponds to a predefined quality criterion, using a loss function which determines whether there is a deviation of the determined assignment with the predefined quality criterion.

[0014] In a seventh step, the at least one neural network is trained if the detected deviation does not meet the specified quality criterion and in order to minimize a loss in the detected deviation from the determined assignment.

[0015] A fundamental idea of ​​the present invention is that, by means of the inventive method for training a neural network and the use of suitable loss functions, it is possible to force a reliable prediction about the features in specific layers of the neural network and thus ultimately to make predictions with near certainty. With the inventive method, a neural network can therefore be trained in such a way that a feature map of a neural network can be partitioned into scalars and vectors, so that these can be represented as in the Fig. The representations shown in Figures 1a to 1d behave under rotations from SO(2). A further fundamental idea of ​​the present invention is to provide an architecture for a system in which the known approach of augmentation is combined with an equivariant architecture.

[0016] The present invention thus advantageously enables the fusion of information from different sensor types, such as radar, lidar, and camera sensors, and furthermore allows for the correction of features from the neural network's feature map to account for self-motion. In this way, a neural network can be efficiently trained and its quality improved to reliably detect an object using various sensor information.

[0017] One possible implementation of the method involves using the direction-dependent input parameter to represent the orientation of the second measurement data. This can improve the training of the neural network.

[0018] One possible implementation of the method involves deriving the data source for the initial measurements from sensor monitoring of the system's environment. This offers the advantage of improved results from the trained neural network.

[0019] One possible implementation of the method involves the first feature class describing an immutable property of the object to be detected, and the second feature class describing a mutable property. This offers the advantage of allowing a clear statement about the invariance or equivariance of the neural network's features, thus contributing to improved performance of the trained neural network.

[0020] One possible implementation of the method involves representing a feature of the first feature class as a scalar quantity and a feature of the second feature class as a vector quantity. This offers the advantage of allowing a clear statement about the invariance or equivariance of the neural network's features, thus contributing to improved performance of the trained neural network.

[0021] One possible implementation of the method involves replacing the loss function with a network that builds upon the features and generates a desired output. This approach utilizes a network architecture for the neural network that already assumes a correct assignment of the features into a first feature class and at least a second feature class. This simplifies the training of the neural network.

[0022] According to a second aspect, the disclosure relates to a computer program containing machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the computer(s) or computer instances to execute the method according to the invention.

[0023] According to a third aspect, the disclosure concerns a machine-readable data carrier and / or download product containing the computer program.

[0024] According to a fourth aspect, the disclosure relates to one or more computers and / or computer instances with the computer program, and / or with the machine-readable data carrier and / or the download product.

[0025] Further measures improving the invention are described in more detail below, together with a description of preferred embodiments of the invention, with reference to figures. Examples of implementation

[0026] It shows: Fig. 1a, Fig. 1b, Fig. 1c, Fig. 1d Schematic representation of the behavior of representations of an SO(2) group according to the state of the art, Fig. 2 Schematic representation of different feature maps with respective input point clouds according to an embodiment of the present invention, Fig. 3 Schematic flow diagram of method 100 for training a neural network according to an embodiment of the present invention, and Fig. 4 Schematic representation of the method according to the invention in one embodiment of the present invention.

[0027] Fig. 1a, Fig. 1b, Fig. 1c, Fig. Figure 1d shows a schematic representation of the behavior of representations of an SO(2) group according to the state of the art. Fig. 1a, Fig. 1b, Fig. 1c and Fig. Figure 1d shows the behavior of representations of an SO(2) group that are moved from one location to another by applying a rotation, but retain their values. In the Fig. 1c and Fig. In 1d, vectors are moved from one location to another and change their values ​​or orientation.

[0028] Regarding the Fig. 1a, Fig. 1b of the Fig. 1 can be explained as follows: While the vehicle is being driven by Fig. When point 1a is rotated to point 1b, for example, triggered by movement of the ego vehicle, the positions of parts of the object (points P1, P1*, P2, P2*) change, but the relative distances between the gray points remain constant. The values ​​of points P1, P1*, P2, P2* correspond to features of the object that do not change. For example, a point P1, P2 represents the fact that a mirror or a headlight of a vehicle is located at that point. The same applies to the length and width of the vehicle, as well as the object class (car, bus, pedestrian, etc.), which may be encoded in the features of a point P1, P1*, P2, P2*. Another example would be the distance between subcomponents of the vehicle, such as the distance between the two side mirrors. This is also referred to as invariant behavior.

[0029] The Fig. 1c, Fig. 1d of the Fig. Figure 1 shows an example of equivariance using vectors as outputs: When the vehicle rotates, not only does the position of the parts (D1, D1*) change, but also the direction of the vectors attached to points D1 and D1*. In the case of a vehicle's mirrors described above, the vehicle's orientation in space would be encoded by the feature, and not just its property of being a vehicle. This also applies, for example, to the relative positions of a vehicle's components. Thus, while the distance between the side mirrors does not change, the path from the left to the right side mirror changes for the observer.

[0030] Fig. 2 Schematic representation of different feature maps with respective input point clouds according to an embodiment of the present invention.

[0031] Between the Fig. 2a) and Fig. 2 b) The input data was rotated by 90 degrees. The outputs 40 are also rotated by 90 degrees. Although the network is (practically) equivariant due to augmentation in the output, since the detections also work on rotated input ( Fig. 2a,b). Between the Fig. In 2b) and c), the input data was not changed, but the feature map was rotated by 90 degrees. This leads to the assumption that all features are scalars. Since the detections no longer work, this is apparently not entirely true. The present invention solves this problem, so that, as in the Fig. As shown in 2c, the output is again approximately the same as in 2a).

[0032] Fig. Figure 3 shows a schematic flowchart of the method 100 for training a neural network 2 for a system 150 for detecting an object 200 according to an embodiment of the present invention.

[0033] In step 102, initial measurement data 10 is provided from a data source 30. The initial measurement data 10 can represent 2D or 3D data. The data source 30 can, for example, represent a drone taking aerial images of an object, or a point cloud.

[0034] The data source 30 for the first measurement data 10 can be derived from sensor monitoring of an environment 155 of the system 150. The system 150 can be an autonomous or semi-autonomous system; it can be a driver assistance system in a vehicle or, more generally, a car, aircraft, robot, etc. The object 200 can be a vehicle, a person, a post, an object, or a living being.

[0035] In step 104, a second set of measurement data 20 is generated by adding a direction-dependent input parameter 12 to the first set of measurement data 10. The direction-dependent input parameter 12 represents the orientation of the object 200, which could be, for example, a vehicle or a vehicle component such as a side mirror.

[0036] In step 106, at least one feature 105 is defined for the at least one neural network 2, whereby the at least one feature 105 is assigned either to a first feature class 14 or to at least a second feature class 16.

[0037] In step 108, the first measurement data 10 and the second measurement data 20 are fed into the at least one neural network 2. The neural network 2 can be a convolutional neural network or a fully connected neural network.

[0038] In step 110, at least one feature from the at least one neural network 2 is assigned, taking into account the direction-dependent input parameter 12, whereby the at least one feature 22 of the output corresponds to either the first feature class 14 or the second feature class 16. Thus, one feature map is obtained as output for each measurement data set.

[0039] The first feature class 14 describes a non-changing property of the object 200 to be detected, and the second feature class 16 describes a changing property of the object 200 to be detected.

[0040] A feature of the first feature class 14 is implemented as a scalar quantity and a feature of the second feature class 16 is implemented as a vector quantity.

[0041] In step 112, a check is performed to see if the assignment of the at least one feature 22 of the at least one neural network 2 corresponds to a given quality criterion 25, using a loss function, which determines whether there is a deviation 28 of the determined assignment with the given quality criterion 25.

[0042] In an optional embodiment, the loss function can be implemented by a network that builds upon the features and generates a desired output (e.g., objects that one wants to detect), using a network architecture (e.g., vector perceptron) for the neural network 2 that already assumes a correct assignment of the features into a first feature class 14 and at least a second feature class 16.

[0043] In step 114, the at least one neural network 2 is trained if the detected deviation 28 does not correspond to the specified quality criterion 25 and in order to minimize a loss in the detected deviation 28 from the determined assignment.

[0044] Fig. Figure 4 shows a schematic representation of the inventive method 100 according to an embodiment of the present invention.

[0045] In the state of the art, a network is normally designed end-to-end with the loss L H the heads trained. The present invention now provides that a feature map is conceptually divided into scalars s and vectors V of dimension 2. If a feature map has, for example, 512 channels, it can be divided, for example, into 300 scalars and 106 vectors. So that scalars s and vectors V are in Fig. If the conditions for invariance or equivariance shown in point 1 are met, additional losses L are added. s and LV inserted.

[0046] To define a loss at all, an augmented pair must be generated from each batch. For this, two different rotations from SO(2) are applied to the same data and stored, e.g., 0 degrees and 45 degrees, as shown. Fig. Figure 4 shows how this transforms a batch with N elements into a batch with 2N elements. After forward-passing the batch through the neural network, the losses L can now be calculated for all desired feature maps. H and L s can be defined. Since the applied augmentation is known, suitable scalars and v vectors can be found (see Fig. 4) are assigned to each other. Furthermore, it is known how scalars (invariant) and vectors (equivariant) should behave under the known augmentation. The losses L H and L sare now defined precisely in such a way that such behavior produces a lower loss. At the same time, however, the loss of the heads L is also reduced. H It is used further because otherwise the features are not useful for the desired learning task. L H This can only be applied to the first half of the batch or the entire batch (pairs). The losses are then trained using known and standard methods, such as backpropagation and stochastic gradient descent.

Claims

[1] Method (100) for training at least one neural network (2) for a system (150) for detecting an object (200), comprising the following steps: - providing (102) first measurement data (10) from a data source (30); - generating (104) second measurement data (20) by providing the first measurement data (10) with a direction-dependent input parameter (12); - defining (106) at least one feature (105) for the at least one neural network (2), wherein the at least one feature (105) is assigned either to a first feature class (14) or to at least a second feature class (16); - feeding (108) the first measurement data (10) and the second measurement data (20) into the at least one neural network (2); - Assigning (110) at least one feature (22) of the at least one neural network (2) taking into account the direction-dependent input parameter (12), wherein the at least one feature of the output (22) corresponds either to the first feature class (14) or to the second feature class (16); and - checking (112) whether the assignment of the at least one feature (22) of the at least one neural network (2) corresponds to a predetermined quality criterion (25) by means of a loss function which determines whether there is a deviation (28) of the determined assignment with the predetermined quality criterion (25); - training (114) the at least one neural network (2) if the detected deviation (28) does not correspond to the predetermined quality criterion (25) and in order to minimize a loss in the detected deviation (28) from the determined assignment. [2] Method (100) according to claim 1, wherein the direction-dependent input parameter (12) represents an orientation of the second measurement data (10). [3] Method (100) according to one of the preceding claims, wherein the data source (30) for the first measurement data (10) is formed from a sensory monitoring of an environment (155) of the system (150). [4] Method (100) according to one of the preceding claims, wherein the first feature class (14) describes a non-variable property of the object (200) to be detected and the second feature class (16) describes a variable property of the object (200) to be detected. [5] Method (100) according to one of the preceding claims, wherein a feature of the first feature class (14) is formed as a scalar quantity and a feature of the second feature class (16) is formed as a vector quantity. [6] Method (100) according to one of the preceding claims, wherein the loss function is replaced by a network that is based on the features and generates a desired output, using a network architecture for the neural network (2) that already assumes a correct assignment of the features into a first feature class (14) and at least one second feature class (16). [7] A computer program containing machine-readable instructions which, when executed on one or more computers and / or computer instances, cause the computer or computer instances to carry out the method according to any one of claims 1 to 6. [8] Machine-readable data carrier and / or download product with the computer program according to claim 7. [9] One or more computers and / or computer instances with the computer program according to claim 7, and / or with the machine-readable data carrier and / or download product according to claim 8.

Images