Training neural networks for better robustness and generalization

Abstract

A neural network is trained for better robustness and generalization. A method of training a task neural network for a given task to be executed on an input comprising at least an image comprises: providing a 2D training image tagged with a base truth value relative to the given task; expanding at least a subset of the training images into a 3D representation of respective training image content; the to-be-trained task neural network processes the 2D training image into a task and outputs the task; processing the 3D representation into an auxiliary output by an auxiliary neural network trained for auxiliary tasks; rating, by a task loss function, the deviation of the task output of each training image from the base truth with which the training image is tagged; rating, by an auxiliary loss function, the likelihood of the results of the task neural network generated from the at least one training image with the results of the auxiliary neural network generated from the corresponding 3D representation; summing up the task loss function value and the auxiliary loss function value into total loss; parameters characterizing task neural network behavior are optimized towards a target of improving total loss.

Description

Technical Field

[0001] The invention relates to the training of a neural network which can be used in particular for evaluating images during the operation of a vehicle and / or a robot. Background Art

[0002] In road traffic and also in industrial settings, at least partially autonomous maneuvering of vehicles and / or robots requires constant monitoring of the environment of the vehicle and / or robot. For the purpose of such monitoring, it is customary to acquire images of the environment and to analyze these images with trained neural networks for a given task. Examples of such tasks are classification of the image as a whole and / or classification of objects shown in the image.

[0003] This process is supposed to mimic driving lessons for human drivers, as training based on a limited number of training situations will enable the neural network to perform well in many unseen situations. However, this generalization capability is primarily limited to the domain and / or distribution to which the training images used to train the neural network belong. Summary of the invention

[0004] The present invention provides a method for training a task neural network for a given task. The given task will be performed on an input comprising at least an image. That is, in addition to images, the neural network can also accept other types of inputs, such as radar or lidar point clouds. In particular, the image can assign one or more intensity values ​​to each pixel in a regular grid in the image plane. For example, the image can be an RGB image, which provides intensity values ​​of the red, green and blue primary colors for each pixel.

[0005] In the process of the method, 2D training images labeled with ground truth with respect to a given task are provided. If the neural network accepts additional inputs in addition to images, the ground truth may be applied to the complete record of training data including both the training images and the additional inputs.

[0006] At least a subset of the training images is expanded into a 3D representation of the content of the corresponding training image. That is, while the training images can be viewed as a function of assigning pixel values ​​to a pair of coordinates in a plane (such as Cartesian coordinates x and y), the 3D representation can be viewed as a function of assigning values ​​to three coordinates in the image plane (such as Cartesian coordinates x, y, and z).

[0007] The task neural network to be trained processes the 2D training images into the task output. If the task neural network additionally accepts further inputs, the complete record of training data including both the training images and the further inputs is processed into the task output.

[0008] The already trained auxiliary network processes the 3D representation into an auxiliary output. The auxiliary network has been trained for an auxiliary task. The auxiliary task can be the same as the given task, or at least similar to the given task. But this is not required. Even if the auxiliary task is not apparently related to the given task, the output of the auxiliary network can be useful feedback for training. The reason is that many neural networks (especially convolutional networks) first extract basic features from the input image, then combine them into more complex features, and finally solve the given task based on these more complex features. The same extracted features can be used for many different tasks.

[0009] The task loss function rates the deviation of the task output for each training image (or training image plus additional input) from the ground truth with which the training image (or combination of training image and additional input) is labeled.

[0010] In addition, an auxiliary loss function rates the plausibility of the results of the task neural network generated from at least one training image with the results of the auxiliary neural network generated from the corresponding 3D representation. In particular, this plausibility can measure

[0011] How likely the outcome of the task neural network is given the outcome of the auxiliary neural network, and / or

[0012] How likely is the result of the auxiliary neural network given the result of the task neural network.

[0013] In particular, the result of the auxiliary neural network may be of the same type as the result of the task neural network, or it may be of at least a similar type. However, this is not required. On the contrary, even very different types of results may be plausible or implausible relative to each other, depending on whether the training state of the task neural network to be trained is good.

[0014] The value of the task loss function and the value of the auxiliary loss function are summed up to form a total loss. The parameters that characterize the behavior of the task neural network are optimized towards the goal of improving the total loss.

[0015] The inventors have found that training performed at least in part on 3D knowledge enables the task neural network to better generalize input images from other domains and / or distributions (than the domain and / or distribution to which the training images belong). At the same time, the task neural network becomes more robust to noise in the input data.

[0016] In particular, during normal operation (inference) of the task neural network, the task neural network can better handle domain shifts between the domain of the training images and the domain of subsequent input images. For example, the training of the task neural network for analyzing driving scenes may have been performed using images acquired during daytime, but the vehicle is also expected to analyze driving scenes during nighttime. The use of auxiliary neural networks during training encourages the task neural network to learn generalizable features that are relatively invariant across different domains. For example, in the use case of detecting and / or classifying objects in images, the shape of the objects is well suited to distinguish between different types of objects regardless of lighting conditions.

[0017] The use of 3D knowledge is particularly suitable for this, since the increment of 3D knowledge relative to 2D knowledge is mainly geometric information, which is invariant to changes in lighting, color and other imaging properties.

[0018] From this perspective, it would be better to solve a given task in 3D rather than completely in 2D. But solving a given task in 3D will consume more resources at multiple stages in the process. First, 3D input images must be provided instead of 2D input images. This will require a means to directly obtain the input image in 3D, or to expand each 2D input image calculation into a 3D image. Second, 3D images have a much larger amount of data than 2D images. This means that more communication bandwidth is required to transmit 3D images to the task neural network for processing, and more processing power and memory are required to solve a given task within a given amount of time. But in many safety-critical real-time applications (such as the manipulation of vehicles and / or robots), memory and processing power are very valuable. Mobile applications in vehicles and / or robots set limits on the size, power consumption and / or heat generation of the hardware used. In addition, hardware with a high safety integrity level SIL for safety-critical applications is very expensive.

[0019] Therefore, it is advantageous to perform training only in 3D in order to get the benefits of 3D processing, and then solve a given task in 2D in order to minimize resource consumption. That is, the effort of 3D processing is spent only once during training, but the inference, which will be performed again and again, will be kept as efficient as possible.

[0020] In particular, because the use of 3D knowledge is directly integrated into the main training with an additional loss term, rather than being restricted to a pre-training step, there is no “catastrophic forgetting” of 3D knowledge during optimization towards the 2D task.

[0021] In a particularly advantageous embodiment, the auxiliary task is chosen to correspond to the given task. In this context, "corresponding" is understood to mean that the two tasks are at least of the same type, such as classification or segmentation. In this way, both the value of the task loss function and the value of the auxiliary loss function can be expected to have similar dependencies on the training state of the task neural network. The additional consideration of the auxiliary loss function is then more of a regularization of the original given task than a completely different training objective.

[0022] In a further particularly advantageous embodiment, the extending of at least one training image into a 3D representation comprises computing a depth map which contains, for each pixel of the 2D training image, the distance from the camera with which the training image was recorded. For the generation of the depth map, many tried and tested methods are available.

[0023] In particular, the depth map can then be further processed into a 3D image that assigns pixel values ​​of the 2D training image to locations outside the plane of the 2D training image. For example, such a 3D image can be in the form of a point cloud that assigns values ​​to some points in the volume, but not all points. For this purpose, standard libraries such as Open3D are available.

[0024] In one example, before processing into a 3D image, the depth map is cropped to include only a set of desired objects or other semantic subunits of the training image. The resulting point cloud can then be provided, for example, to an auxiliary neural network that is trained to classify or otherwise analyze individual object instances.

[0025] In a further advantageous embodiment, the given task comprises:

[0026] a mapping of an input image or any portion thereof to classification scores for one or more classes relative to a given classification, and / or

[0027] Segmentation of input images into semantic sub-units.

[0028] Solving these problems in 2D is not very different from solving them in 3D. In particular, regardless of whether the task is solved in 2D or 3D, the final result is always of the same type, which means that it is easier to evaluate the plausibility of the results of the auxiliary neural network with the results of the task neural network. Moreover, classification tasks and segmentation tasks are among the most frequently required tasks in the context of monitoring the environment of a vehicle and / or robot.

[0029] In particular, the task neural network can be selected to output bounding boxes of object instances and classification scores associated with these object instances. In one example, the bounding boxes can then be used to crop the depth map to generate a 3D representation. The plausibility between the results of the task neural network and the auxiliary neural network can then measure some kind of "cycle consistency": a certain result of the task neural network is used to determine the input to the auxiliary neural network, and then the result of the auxiliary neural network is compared in some way to the original result of the task neural network. In another example, the distribution of points in 3D space classified as a particular class by the auxiliary neural network can be checked with respect to whether it corresponds to a bounding box determined by the task neural network.

[0030] In a further particularly advantageous embodiment, the result of the task neural network and / or the result of the auxiliary neural network entering the auxiliary loss function comprises the output of an intermediate layer before the respective final output layer. In particular, if the task of the auxiliary neural network is different from the given task, the outputs of the respective intermediate layers may still be very similar. As discussed previously, the solution of both tasks may be based on similar intermediate features. For example, a given classification task on the one hand regarding the type of object instances in the input image and an auxiliary task on the other hand classifying the danger level of a traffic situation as a whole may utilize the same or similar features extracted from the 2D training images, respectively from the 3D representation.

[0031] In another particularly advantageous embodiment, the total loss comprises a weighted sum of the task loss and the auxiliary loss. In this way, the relative weighting of the two loss contributions can be used as a hyperparameter to adjust the strength of the regularization.

[0032] In a further particularly advantageous embodiment, the auxiliary neural network is chosen to be trained on a domain that is a superdomain of the domain of the training images. In this way, better generalization of the task neural network can be steered towards this superdomain. In this document, the term "superdomain" means that members of the domain of the training images also belong to this superdomain, just as all elements of a set will also be elements of a superset of the set. For example, if the training images include images of traffic signs as objects, the superdomain may include images with object instances of many possible types including traffic signs. An auxiliary network that has been trained on this superdomain will have seen far fewer training images from the traffic sign domain than are present in the specific set of training images at hand, but it will have seen at least some traffic signs, as well as at least some object instances from many other domains. In particular, given a specific application with a particular set of training images, auxiliary networks that have been generally trained on the superdomain are likely to be readily available.

[0033] As discussed previously, the point of training using 3D knowledge is to improve the generalization of the task neural network to the domain of 2D images beyond the domain that the 2D training images belong to. This generalization will improve the accuracy of the task neural network when processing input images beyond the domain of the training images.

[0034] Therefore, in a further particularly advantageous embodiment, 2D input images are provided which have been acquired with at least one sensor. These input images are processed by the trained task neural network into task outputs. The task outputs will then benefit from the improved accuracy.

[0035] As discussed previously, the acquisition of 2D input images, the transmission of these input images from the sensor to the task neural network, and the processing of the input images by the task neural network can all occur in the 2D world, with a much lower amount of data than if all processing were done in 3D. But the final accuracy will still benefit from the 3D knowledge that has gone into the training.

[0036] In a further particularly advantageous embodiment, at least one sensor is carried by the vehicle and / or the robot. Furthermore, during the method, an actuation signal is calculated. The vehicle and / or the robot is actuated with the actuation signal. In this way, the reaction performed by the vehicle and / or the robot in response to the actuation signal is more likely to be appropriate in a given situation conveyed by the input image. Furthermore, as discussed previously, mobile applications in vehicles and / or robots benefit from the relatively small data volume of 2D images relative to 3D images or point clouds. For example, the CAN bus commonly used in vehicles has a maximum bandwidth of only 1 Mbit / s.

[0037] The method may be computer-implemented in whole or in part and embodied in software. Therefore, the present invention also relates to a computer program having machine-readable instructions, which, when executed by one or more computers and / or computing instances, causes one or more computers and / or computing instances to perform the above method. In this document, control units for vehicles or robots and other embedded systems capable of executing machine-readable instructions should also be considered computers. Computing instances include virtual machines, containers, or other execution environments that allow machine-readable instructions to be executed in the cloud.

[0038] The non-transitory storage medium and / or download product may include a computer program. A download product is an electronic product that can be sold online and transmitted over a network for immediate fulfillment. One or more computers and / or computing instances may be equipped with the computer program and / or equipped with the non-transitory storage medium and / or download product.

[0039] Hereinafter, the present invention will be described using the accompanying drawings without any intention of limiting the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings show:

[0041] Figure 1 An exemplary embodiment of a method 100 for training a task neural network 1;

[0042] Figure 2 Visualization of the effect of method 100 . DETAILED DESCRIPTION

[0043] Figure 1 is a schematic flow chart of an embodiment of a method 100 for training a task neural network 1. The training enables the task neural network 1 to be used for a given task to be performed on an input. The input comprises at least an image 2.

[0044] In step 110, 2D training images 2a are provided which are labeled with ground truth 3a with respect to a given task.

[0045] According to block 111, a given task may include:

[0046] a mapping of the input image 2 or any part thereof to classification scores for one or more classes with respect to a given classification, and / or

[0047] Segmentation of input image 2 into semantic sub-units.

[0048] According to block 112 , a task neural network 1 may be selected to output bounding boxes for object instances and classification scores associated with those object instances.

[0049] In step 120 , at least a subset of the training images 2 a is expanded into a 3D representation 4 a of the content of the respective training image 2 a .

[0050] According to box 121, expanding at least one training image into a 3D representation may include computing a depth map containing, for each pixel of the 2D training image 2a, a distance from a camera with which the training image 2a was recorded. According to box 121a, the depth map may be cropped 121a to include only a set of desired objects or other semantic subunits of the training image 2a. According to box 121b, it may be further processed into a 3D image that assigns pixel values ​​of the 2D training image 2a to locations outside the plane of the 2D training image 2a.

[0051] In step 130 , the task neural network 1 to be trained processes the 2D training image 2 a into a task output 3 .

[0052] In step 140 , the auxiliary neural network 5 trained for the auxiliary task processes the 3D representation 4 a into an auxiliary output 6 .

[0053] According to block 141 , a secondary task may be selected to correspond to a given task.

[0054] According to box 142, the auxiliary neural network 5 can be selected to be trained on a domain that is a superdomain of the domain of the training images 2a.

[0055] In step 150 , the task loss function 7 rates the deviation of the task output 3 for each training image 2 a from the ground truth 3 a with which the training image 2 a was labeled.

[0056] In step 160, the auxiliary loss function 8 rates the likelihood of the result of the task neural network 1 generated from the at least one training image 2a with the result of the auxiliary neural network 5 generated from the corresponding 3D representation 4a. In particular, the result of the task neural network 1 may be the task output 3 generated from the training image 2a, and / or an intermediate work product 3' generated when computing the task output 3. Similarly, the result of the auxiliary neural network 5 may be the auxiliary output 6 generated from the 3D representation 4a, and / or an intermediate work product 6' generated when computing the auxiliary output 6.

[0057] Therefore, according to box 161, the result of the task neural network 1 and / or the result of the auxiliary neural network 5 entering the auxiliary loss function may include the output 3', 6' of the intermediate layer before the corresponding final output layer.

[0058] In step 170 , the value of the task loss function 7 and the value of the auxiliary loss function 8 are summed to form a total loss 9 .

[0059] According to box 171 , the total loss 9 may include a weighted sum of the task loss 7 and the auxiliary loss 8 .

[0060] In step 180, the parameters 1a characterizing the behavior of the task neural network 1 are optimized towards the goal of improving the total loss 9. In step 130, the task neural network 1 feeds these parameters 1a back to the processing of the training images 2a. The final optimized state of the parameters is labeled with the reference numeral 1a*. This set of parameters also defines the final training state 1* of the task neural network 1.

[0061] In step 190 , a 2D input image 2 that has been acquired with at least one sensor 10 is provided.

[0062] According to box 191 , at least one sensor 10 may be carried by the vehicle 50 and / or the robot 51 .

[0063] In step 200 , the trained task neural network 1 processes the input image 2 into the task output 3 .

[0064] In step 210 , an actuation signal 210 a is calculated from the task output 3 .

[0065] In step 220 , the vehicle 50 and / or the robot 51 is actuated using the actuation signal 210 a .

[0066] Figure 2 An example is given of how the auxiliary network 5 can be used to help train the task neural network 1 to be trained.

[0067] exist Figure 2 In the example shown in , a training image 2a shows a traffic scene 20 with a car 21 on a road 22. In the background, a tree line 23 is visible, and the sun 24 is shining through the tree line 23. The car 21 has been identified as an object instance of an unknown type and is marked with a bounding box B.

[0068] According to step 130 of method 100, the content of bounding box B is provided to the task neural network 1 to be trained. The trained task neural network 1 generates a task output 3 from the content of bounding box B. During this calculation, the three layers 11-13 of the task neural network 1 generate an intermediate work product 3'.

[0069] The training images 2a are labeled with ground truth 3a with respect to a given task, here: classification of an object instance. In step 150 of the method 100, a task loss function 7 rates the deviation of the task output 3 from the ground truth 3a.

[0070] The content of the bounding box B is also expanded into a 3D representation 4a, here: a point cloud. This 3D representation 4a is provided to the auxiliary neural network 5 and processed into an auxiliary output 6. During this calculation, the three layers 51-53 of the auxiliary neural network 5 produce an intermediate work product 6'.

[0071] In step 160 of method 100 , the auxiliary loss function 8 rates the plausibility of the intermediate work product 3 ′ from the task neural network 1 and the intermediate work product 6 ′ from the auxiliary neural network 5 .

[0072] In step 170 of method 100 , a total loss 9 is summed up from the values ​​of the task loss function 7 and the values ​​of the auxiliary loss function 8 .

[0073] As discussed previously, in this way, the task neural network 1 is trained to base its decisions more on easily generalizable features (such as geometric features) than on lighting conditions, textures, or other attributes that may be less suitable for distinguishing object instances under certain conditions. If the task neural network 1 bases its decisions on those other attributes, it can achieve a better task loss 7, but the intermediate work product 3' will become very different from the intermediate work product 6' generated from the 3D representation 4a. This will be penalized by the auxiliary loss 8.

Claims

1. A method (100) for training a task neural network (1) for a given task to be performed on an input comprising at least an image (2), the method comprising the following steps: providing (110) 2D training images (2a) labeled with ground truth (3a) with respect to a given task; Expanding (120) at least a subset of the training images (2a) into a 3D representation (4a) of the content of the corresponding training image (2a); The task neural network to be trained (1) processes (130) the 2D training image (2a) into a task output (3); The 3D representation (4a) is processed (140) into an auxiliary output (6) by an auxiliary neural network (5) trained for the auxiliary task; The deviation of the task output (3) of each training image (2a) from the ground truth (3a) with which the training image (2a) is labeled is rated (150) by the task loss function (7); The results (3, 3') of the task neural network (1) generated from the at least one training image (2a) are rated (160) by an auxiliary loss function (8) for plausibility with the results (6, 6') of the auxiliary neural network (5) generated from the corresponding 3D representation (4a); · Adding (170) the value of the task loss function (7) and the value of the auxiliary loss function (8) to form a total loss (9); and Optimizing (180) the parameters (1a) characterizing the behavior of the task neural network (1) towards the goal of improving the total loss (9).

2. The method (100) of claim 1, wherein an auxiliary task is selected (141) to correspond to a given task.

3. The method (100) according to any one of claims 1 or 2, wherein extending at least one training image into a 3D representation comprises (121) calculating a depth map, which contains, for each pixel of a 2D training image (2a), a distance from a camera, the training image (2a) being recorded with the camera.

4. The method (100) according to claim 3, wherein the depth map is further processed (121b) into a 3D image, which assigns pixel values ​​of the 2D training image (2a) to locations outside the plane of the 2D training image (2a).

5. The method (100) according to claim 4, wherein: Before processing into a 3D image, the depth map is cropped (121a) to include only a set of desired objects or other semantic sub-units of the training image (2a).

6. The method (100) according to any one of claims 1 to 5, wherein the given task comprises (111): a mapping of the input image (2) or any part thereof to classification scores for one or more classes with respect to a given classification, and / or Segmentation of the input image (2) into semantic sub-units.

7. The method (100) of claim 6, wherein the task neural network (1) is selected (112) to output bounding boxes of object instances and classification scores associated with these object instances.

8. A method (100) according to any one of claims 1 to 7, wherein the results of the task neural network (1) and / or the results of the auxiliary neural network (5) entering the auxiliary loss function (8) include (161) the output (3', 6') of the intermediate layer before the corresponding final output layer.

9. The method (100) according to any one of claims 1 to 8, wherein the total loss (9) comprises (171) a weighted sum of a task loss (7) and an auxiliary loss (8).

10. The method (100) according to any one of claims 1 to 9, wherein the auxiliary neural network (5) is selected (142) to be trained on a domain that is a superdomain of the domain of the training images (2a).

11. The method (100) according to any one of claims 1 to 10, further comprising: providing (190) a 2D input image (2) that has been acquired with at least one sensor (10), and The trained task neural network (1) processes (200) the input image (2) into a task output (3).

12. The method (100) according to claim 11, wherein The at least one sensor (10) is carried (191) by a vehicle (50) and / or a robot (51), and the method further comprises: Calculating (210) an actuation signal (210a) from the task output (3), and Actuating (220) the vehicle (50) and / or the robot (51) with an actuation signal (210a).

13. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100) according to any one of claims 1 to 12.

14. A non-transitory machine-readable data carrier and / or download product having a computer program according to claim 13.

15. One or more computers and / or computing instances having a computer program according to claim 13 and / or having a non-transitory machine-readable data carrier and / or a download product according to claim 14.

Images