Method for training a neural network

Abstract

According to various embodiments, a method is provided for training a neural network, comprising: determining an output of the neural network for a training data set by applying the neural network to training data elements of a training data set; calculating a loss for the output of the neural network in relation to a data processing task for which the neural network is to be trained; supplementing the loss by an additional loss which, for each of the training data elements and for each transformation of a number of transformations, contains the gradient of the difference between an output of the neural network for the training data element and the output of the neural network for the training data element transformed by means of the transformation; and adjusting the neural network to reduce to loss supplemented by the additional loss.

Description

[0001]R. 407227 Description Title Method for training a neural network The present disclosure relates to methods for training a neural network. Neural networks nowadays have a wide range of applications and are used, for example, to recognize objects in image data or to control robots and self-driving vehicles. Thanks to their large number of parameters, they can process very complex data sets and are generally trained with the aim of making good predictions for later unknown input data, e.g. to classify the objects in an image as accurately as possible. When training a neural network, however, it can happen that the network adapts too strongly to the specific characteristics of the training data. This undesirable overfitting (overfitting)Overfitting of the training dataset results in the neural network providing very good predictions for the training data, but not always good predictions for later, unknown input data. One case of this is that a neural network can learn invariance to a set of transformations from the training data during training, which it does not carry over well beyond the training data. In other words, if a neural network is trained on a training dataset A and it learns a certain degree of invariance to a transformation group G (or more generally to a set of transformations), it can lose this invariance when applied (or tested) to a dataset B. Therefore, methods for training a neural network are desirable in which the neural network demonstrates invariance to a set of R.407227 transformations in such a way that it retains them even when applied to input data that goes beyond the training data set. According to various embodiments, a method for training a neural network is provided, comprising determining an output of the neural network for a training data set by applying the neural network to training data elements of a training data set, calculating a loss for the output of the neural network with regard to a data processing task for which the neural network is to be trained, supplementing the loss with an additional loss which, for each of the training data elements and for each transformation of a set of (one or more) transformations, the gradient (with respect to the training data element, ie at the location of the training data element and with the training data element as a variable, ie with respect tothe dependence on the training data item) of the difference between an output of the neural network for the training data item and the output of the neural network for the training data item transformed by means of the transformation (i.e., the higher the deviation, the higher the additional loss, i.e., the additional loss penalizes an increase in the gradient) and adapting the neural network to reduce the loss supplemented by the additional loss. In other words, according to various embodiments, regularization (of the loss function) is carried out during training so that the neural network learns reliable invariance to a set of (one or more) transformations, i.e., an invariance that is maintained beyond the training data.Invariance under a transformation means that the neural network produces the same output for an input and the input transformed according to the transformation. Invariance under a transformation can also be understood as a symmetry of the neural network with respect to the transformations (e.g., the neural network is symmetric with respect to a swap of two inputs, i.e., it produces the same output in both cases). This allows a given neural network to be trained for reliable invariance under a set of transformations in the input data without requiring architectural changes to the neural network. The method described above is particularly effective when prior knowledge about the geometry of a data processing task downstream (of the processing of the neural network) is available, i.e.,For example, if it is known that the neural network should ideally be invariant with respect to a specific transformation group (or set of transformations), the set of transformations can then be chosen accordingly. In medical image analysis, for example, a neural network is often trained to be invariant with respect to rotations of the data samples. Another example is an image processing system for autonomous driving, where it is often desirable to train a network to be invariant with respect to permutations of the color channels of the image sensors (color invariance). Various embodiments are given below. Embodiment 1 is a method for training a neural network as described above. The additional loss includes, for example, averaging over the training data elements and the transformations.Embodiment 2 is a method according to embodiment 1, wherein the set of one or more transformations contains a set of one or more similarity transformations. A similarity transformation can be understood as a similarity mapping with a mapping equation of the form X' = m AX + b, where m > 0 is a real number, b is a vector, and A is an orthogonal matrix. If the neural network processes images, then, for example, in the similarity mapping, for each pixel belonging to a displayed object, the position X changes to the position X' according to the mapping equation (i.e., the object is rotated, translated, mirrored, and / or scaled). If, for example, the input to the neural network is a point cloud, the similarity transformation is a mapping according to the mapping equation in three-dimensional space (where X', X, and b are three-dimensional vectors).Embodiment 3 is a method according to embodiment 1 or 2, wherein the set of one or more transformations contains one or more color transformations. This can, for example, ensure that objects (such as vehicles) are recognized regardless of their color, i.e., the neural network focuses on shapes, for example. Embodiment 4 is a method according to one of embodiments 1 or 3, wherein the data processing task is processing sensor data. For example, the output of the (trained) neural network can be used to control a robot device. For example, other vehicles are detected in the vicinity of a vehicle by processing camera image data or radar data, and the vehicle is controlled depending on the detection result (e.g., for autonomous driving or driver assistance functions).Embodiment 5 is a method according to one of embodiments 1 or 4, wherein the data processing task is classification or regression. For example, the neural network extracts (e.g., through one or more convolutional network layers) features (such as low-level image features like pixels or edges) and performs classification (e.g., as part of segmentation and / or object detection) or regression based on the extracted features. For a data processing task such as object detection, invariance against transformations such as reflection is typically desirable. Embodiment 6 is a method according to one of embodiments 1 to 5, comprising selecting the set of one or more transformations as transformations against which the data processing task requires invariance.For example, detection of a vehicle should be independent of the direction in which the vehicle is traveling (e.g., oriented in input sensor data) or how far away it is (or, for example, scaled in input sensor data). Embodiment 7 is a data processing device configured to perform a method according to any one of embodiments 1 to 6. Embodiment 8 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6. Embodiment 9 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6. In the drawings, like reference numerals generally refer to the same parts throughout the several views.The drawings are not necessarily to scale, emphasis instead being placed generally upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Figure 1 shows a vehicle. Figure 2 illustrates the training of a neural network according to one embodiment. R. 407227 Figure 3 shows a flowchart illustrating a method for training a neural network according to one embodiment. The following detailed description refers to the accompanying drawings that, for the purpose of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention.The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects. Various examples are described in more detail below. In machine learning, a function is learned that maps input data to output data. During learning (for example, training a neural network and / or model), from an input data set (also referred to as training data set), which specifies a desired output (for example, a desired classification of the input data or a regression result) for each input, the function is determined in such a way that it maps this mapping of inputs to outputs as best as possible. An example of an application of such a machine-learned function is object classification for autonomous driving, as illustrated in Figure 1.It should be noted that in the following, images or image data are understood very generally as a collection of data representing one or more objects or patterns. The image data can be provided by sensors that measure visible or invisible light, such as infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals. R. 407227 In the example for object recognition in autonomous driving 100 in Figure 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device 102. The vehicle control device 102 has data processing components, e.g. a processor (e.g. a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates, and data processed by the processor 103.For example, the stored control software (computer program) includes instructions which, when executed by the processor, cause the processor 103 to implement a neural network 107. The data stored in memory 104 may, for example, include image data captured by one or more cameras 105. The one or more cameras 105 may, for example, take one or more grayscale or color photos of the surroundings of the vehicle 101. The vehicle control device 102 may (by processing by the neural network 107), for example, based on the image data, determine whether and which objects, e.g., fixed objects such as traffic signs or road markings or moving objects such as pedestrians, animals, and other vehicles, are present in the surroundings of the vehicle 101.For this purpose, the neural network 107 is trained accordingly, for example on an external data processing device, from which it is loaded into the vehicle control device 102 after training. The vehicle 101 can then be controlled by the vehicle control device 102 according to the results of the object determination. For example, the vehicle control device 102 can control an actuator 106 (e.g., a brake) to control the speed of the vehicle, e.g., to brake the vehicle. R. 407227 In the above example, the neural network performs a classification task, but it can also perform a regression task, such as predicting the position of another road user or determining control variables for an anti-lock braking system (e.g., based on information about the surface (such asa camera image of the ground on which the vehicle is moving). For many tasks of a neural network, it is desirable for the neural network to exhibit invariance under a set of transformations. For example, it should recognize another vehicle as such, regardless of whether the other vehicle passes from the right or left of the vehicle 101 in which it is implemented. Such invariance can be learned by the neural network with suitable training data, but the neural network may lose this invariance when applied to other data. Equivariant neural networks aim to incorporate symmetries as an inductive bias into a network architecture. Starting with convolutional networks, which exhibit translational symmetry, the concept of an equivariant network can be generalized to rotations, permutations, scaling, and arbitrary matrix groups.A neural network can also be designed to be equivariant with respect to transformations learned directly from the training data. This can endow a neural network with reliable invariance (or equivariance) with respect to transformations, but achieving this requires non-trivial architectural changes to the respective model, which often involve high memory and inference time overhead. Non-equivariant neural networks can also learn invariance when trained with data augmentation. However, the properties of such learned invariances are still largely unexplored. This complicates the practical use of non-equivariant invariance learning, as it is not clear how reliable these learned invariances are when the operating conditions change. This is a crucial disadvantage when using models with learned invariances in practice.According to various embodiments, a procedure is therefore provided in which, by means of regularization during training of an arbitrary (i.e., not necessarily equivariant) neural network, the neural network is made invariant with respect to (at least) a specific set of (e.g., similarity) transformations (e.g., a transformation group). The procedure enables reliable invariance, i.e., the invariance is maintained across the training data. The provided regularization does not require any change to the architecture of the respective neural network and merely consists of adding an additional term to the loss function used to train the neural network (i.e., to reduce the loss function, the neural network (i.e., its weights) are adapted, typically by means of backpropagation).A similarity mapping or similarity transformation can be understood as a transformation of an object that leaves the object unchanged (where scaling, i.e., changes in distances, are included, but angles and distance ratios are retained). The set of all such transformations of an object with a composition as a link forms a (similarity) transformation group G. A (similarity) transformation group can, for example, be a group of rotations, translations, or scalings. A neural network f parameterized by the set of parameters θ (typically a set of weights) θ : X → Y is considered invariant with respect to the group G if f θ (g ◦ x) = f θ(x) for ∀g ∈ G, where g ◦ x stands for the operation of a group element g on a data element x. R. 407227 One method for learning invariance is data augmentation. This means, for example, that at each training iteration , where B is the size of a respective training (mini-)batch, samples are taken from the group G and applied to the training batch before the output of the neural network is determined for it and from this the value of the loss function is calculated (to reduce which the neural network is then adapted). In the following, the evaluation of the neural network f θ learned invariance (separately from the performance for the respective data processing task), the invariance error used. The invariance error where the respective training data set and is a distance measure between the outputs of the neural network for x and g ◦ x. For classification tasks, this distance function d corresponds, for example, to the Kullback-Leibler divergence between the output logits. Intuitively, a smaller invariance error indicates that f θ has learned a higher degree of invariance with respect to G on the dataset. Training with stronger data augmentation, for example, would lead to a lower invariance error. Besides data augmentation, another approach to learning invariance is to explicitly minimize . In practice, this amounts to regularizing the loss function (which contains a loss for the respective data processing task) with the invariance error objective of equation (1). Given a batch of training data, the invariance error can be estimated as follows: R. 407227 G is a discrete transformation group, but can also be any set of transformations. With this explicit regularization with the invariance error according to equation (2), invariance with respect to a set of transformations can be learned better than by data augmentation. However, invariance learned in this way still extends relatively poorly beyond the training data. According to various embodiments, a regularization is therefore provided, the basis of which can be seen in an analysis of the invariance error from the perspective of Lipschitz continuity. The invariance error is Lipschitz-continuous with respect to the data set if there exists a constant L such that: for ∀x, x′ ∈ . Intuitively, the Lipschitz constant in equation (3) is the upper bound for the change in the invariance error when measured for two different data elements. In other words, a small L guarantees that the invariance error does not change drastically between different data elements. A small L can thus be seen as a hallmark of reliable invariance. Accordingly, various methods use a method for learning invariance that yields an invariance error with a small Lipschitz constant. The Lipschitz constant cannot be accessed directly. However, consider a data sample x ∈ and its neighborhood Ω(x) (within the data set, i.e., surrounding data elements). According to the mean value theorem, for x′ ∈ Ω(x), R. 407227 where . According to the Cauchy-Schwarz inequality, (5) Note that the Lipschitz constant of approximates when x′ → x. Therefore, according to various embodiments, during training, in addition to the loss related to the data processing task (e.g., classification loss or regression loss) for which the neural network is trained, is minimized so that the neural network learns reliable invariance. Formally, given a (mini-)batch of training data elements and a transformation group G (a discrete transformation group is assumed here, but embodiments are not limited to this), the loss function (in addition to the loss related to the data processing task for which the neural network is to be trained) is used to regularize the training, the loss where |G| denotes the cardinality of the transformation group G. Through this regularization, the learning of an invariance is guided towards shallow minima by penalizing a high norm of the gradient of the invariance objective. Figure 2 illustrates the training of a neural network 201 according to the procedure described above. R. 407227 The loss 202 with respect to a data processing task to be performed by the neural network 201 (e.g., classification or regression) 203 for a batch of training data 204 (e.g., determined using ground truth data such as labels or also by a loss for self-supervised learning) is supplemented by the regularization loss 205 according to equation (6) for a respective set of (one or more) transformations (e.g., transformation group, e.g., similarity transformation group) 206 (i.e., added to the loss 202, if necessary).with a weighting relative to the loss 202) and the neural network is adapted (e.g., by means of backpropagation) to reduce the resulting total loss (i.e., loss function value). This leads (e.g., with a sequence of such adaptations across multiple batches of training data, e.g., until a convergence criterion is met) to a trained version of the neural network 207 that is invariant to the set of transformations 206. In summary, according to various embodiments, a method is provided as illustrated in Figure 3. Figure 3 shows a flowchart 300 illustrating a method for training a neural network according to one embodiment. In 301, an output of the neural network for a training data set is obtained by applying the neural network to training data elements of a training data set.In 302, a loss is calculated for the output of the neural network with respect to a data processing task for which the neural network is to be trained. In 303, the loss is supplemented by an additional loss which, for each of the training data elements and for each transformation of a set of (one or more) transformations, includes the gradient (with respect to the training data element, i.e., at the location of the training data element and with the training data element as a variable, i.e., with respect to the dependence on the training data element) of the difference between an output of the neural network for the training data element and the output of the neural network for the training data element transformed by means of the transformation (i.e., the additional loss is higher the higher the deviation, i.e., the additional loss penalizes an increase in the gradient).At 304, the neural network is adapted to reduce the loss supplemented by the additional loss. The method of Figure 3 may be performed by one or more computers having one or more data processing units. The term "data processing unit" may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be processed according to at least one (i.e., one or more than one) specific function performed by the data processing unit.A data processing unit may comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an integrated circuit, a programmable gate array (FPGA), or any combination thereof. Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail here may be carried out (e.g., implemented) by a data processing unit through one or more special functions performed by the data processing unit. Thus, according to various embodiments, the method is in particular computer-implemented.After training, the neural network can be applied to sensor data determined by at least one sensor. The output of the neural network thus provides a result about a physical state of an environment of the at least one sensor and / or of the at least one sensor itself, or the method can comprise using the output of the trained neural network, which it provides in response to an input of sensor data, as such a result. In other words, the method can comprise deriving or predicting the physical state of an existing real object from measurements of physical properties (i.e., sensor data relating to the object) by means of the trained neural network (i.e., its output in response to the sensor data). The approach of Figure 3 is used, for example, to generate a control signal for a robotic device.The term “robot device” can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule is learned for the technical system, and the technical system is then controlled accordingly. For example, after training, the neural network is used to select a control action and generate a control signal for the vehicle. A control action for which one or more control signals are generated is, for example, a distance (e.g., moving one meter to the right), a speed, an acceleration, etc. Various embodiments can use (e.g., time series of) sensor data from various sensors, such as, for example,Receive and use video, radar, LiDAR, ultrasound, motion, thermal imaging, etc.

Claims

R. 407227 Claims 1. A method for training a neural network (107, 201), comprising: determining an output of the neural network (107, 201) for a training data set (204) by applying the neural network (107, 201) to training data elements of a training data set (204); calculating a loss (202) for the output of the neural network (107, 201) with respect to a data processing task (203) for which the neural network (107, 201) is to be trained; Supplementing the loss (202) by an additional loss (205) which, for each of the training data elements and for each transformation of a set of transformations (206), includes the gradient of the difference between an output of the neural network (107, 201) for the training data element and the output of the neural network (107, 201) for the training data element transformed by means of the transformation;and adapting the neural network (107, 201) to reduce the loss (202) supplemented by the additional loss (205).

2. The method of claim 1, wherein the set of one or more transformations (206) includes a set of one or more similarity transformations.

3. The method of claim 1 or 2, wherein the set of one or more transformations (206) includes one or more color transformations. R. 407227 4. The method according to any one of claims 1 or 3, wherein the data processing task (203) is a processing of sensor data.

5. The method according to any one of claims 1 or 4, wherein the data processing task (203) is a classification or a regression.

6. The method according to any one of claims 1 to 5, comprising selecting the set of one or more transformations (206) as transformations to which the data processing task (203) requires invariance.

7. A data processing device configured to perform a method according to any one of claims 1 to 6.

8. A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.

9. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.

Images