Methods for training a neural network

Abstract

According to various embodiments, a method for training a neural network is provided, comprising determining an output of the neural network for a training data set by applying the neural network to training data elements of a training data set, calculating a loss for the output of the neural network with respect to a data processing task for which the neural network is to be trained, supplementing the loss with an additional loss which, for each of the training data elements and for each transformation of a set of transformations, includes the gradient of the difference between an output of the neural network for the training data element and the output of the neural network for the training data element transformed by the transformation, and adjusting the neural network to reduce the loss supplemented by the additional loss.

Description

[0001] The present disclosure relates to methods for training a neural network.

[0002] Neural networks have a wide range of applications today and are used, for example, to detect objects in image data or to control robots and self-driving vehicles. Thanks to their large number of parameters, they can process very complex data sets and are usually trained with the goal of making good predictions for future unknown input data, e.g., to classify the objects in an image as accurately as possible.

[0003] When training a neural network, however, it can happen that the network adapts too strongly to the characteristics of the training data. This undesirable overfitting of the training dataset results in the neural network producing very good predictions for the training data, but not always good predictions for later, unknown input data.

[0004] One case of this is that a neural network, during training, may learn invariance to a set of transformations from the training data that it does not propagate well beyond the training data. In other words, if a neural network is trained on a training dataset A and learns a certain degree of invariance to a transformation group G (or a set of transformations in general), it may lose this invariance when applied (or tested) to a dataset B.

[0005] Therefore, methods for training a neural network are desirable in which the neural network learns invariance to a set of transformations in such a way that it retains this invariance even when applied to input data that goes beyond the training data set.

[0006] According to various embodiments, a method for training a neural network is provided, comprising determining an output of the neural network for a training data set by applying the neural network to training data elements of a training data set, calculating a loss for the output of the neural network with respect to a data processing task for which the neural network is to be trained, supplementing the loss with an additional loss which, for each of the training data elements and for each transformation of a set of (one or more) transformations, determines the gradient (with respect to the training data element, ie at the location of the training data element and with the training data element as a variable, ie with respect tothe dependence on the training data item) of the difference between an output of the neural network for the training data item and the output of the neural network for the training data item transformed by the transformation (ie, the higher the deviation, the higher the additional loss, ie, the additional loss penalizes an increase in the gradient) and adapting the neural network to reduce the loss supplemented by the additional loss.

[0007] In other words, according to various embodiments, regularization (of the loss function) is performed during training so that the neural network learns reliable invariance under a set of (one or more) transformations, i.e., invariance that is maintained beyond the training data. Invariance under a transformation means that the neural network produces the same output for an input and the input transformed according to the transformation. Invariance under a transformation can also be understood as a symmetry of the neural network with respect to the transformations (e.g., the neural network is symmetric with respect to a permutation of two inputs, i.e., it produces the same output in both cases).This allows a given neural network to be trained for reliable invariance under a set of transformations in the input data without requiring any architectural changes to the neural network. The method described above is particularly effective when prior knowledge about the geometry of a downstream data processing task (that of the neural network) is available, i.e., when it is known, for example, that the neural network should ideally be invariant under a certain transformation group (or set of transformations). Then the set of transformations can be chosen accordingly. In medical image analysis, for example, a neural network is often trained to be invariant under rotations of the data samples.Another example is an image processing system for autonomous driving, where it is often desirable to train a network so that it is invariant to permutations of the color channels of the image sensors (color invariance).

[0008] Various examples of implementation are given below.

[0009] Embodiment 1 is a method for training a neural network as described above.

[0010] The additional loss includes, for example, an averaging over the training data elements and the transformations.

[0011] Embodiment 2 is a method according to embodiment 1, wherein the set of one or more transformations includes a set of one or more similarity transformations.

[0012] A similarity transformation can be understood as a similarity mapping with a mapping equation of the form X' = m AX + b, where m > 0 is a real number, b is a vector, and A is an orthogonal matrix. If the neural network processes images, for example, in the similarity mapping, each pixel belonging to a represented object changes from position X to position X' according to the mapping equation (i.e., the object is rotated, translated, mirrored, and / or scaled). If, for example, the input to the neural network is a point cloud, the similarity transformation is a mapping according to the mapping equation in three-dimensional space (where X', X, and b are three-dimensional vectors).

[0013] Embodiment 3 is a method according to embodiment 1 or 2, wherein the set of one or more transformations includes one or more color transformations.

[0014] This can, for example, ensure that objects (such as vehicles) are recognized regardless of their color, so that the neural network concentrates on shapes, for example.

[0015] Embodiment 4 is a method according to one of embodiments 1 or 3, wherein the data processing task is a processing of sensor data.

[0016] For example, the output of the (trained) neural network can be used to control a robotic device. For example, other vehicles in the vicinity of a vehicle can be detected by processing camera image data or radar data, and the vehicle can be controlled based on the detection result (e.g., for autonomous driving or driver assistance functions).

[0017] Embodiment 5 is a method according to one of embodiments 1 or 4, wherein the data processing task is a classification or a regression.

[0018] For example, the neural network (e.g., through one or more convolutional network layers) extracts features (such as low-level image features such as pixels or edges) and performs classification (e.g., as part of segmentation and / or object detection) or regression based on the extracted features. For a data processing task such as object detection, invariance to transformations such as reflection is typically desirable.

[0019] Embodiment 6 is a method according to any one of embodiments 1 to 5, comprising selecting the set of one or more transformations as transformations to which the data processing task requires invariance.

[0020] For example, the detection of a vehicle should be independent of the direction in which the vehicle is traveling (e.g., oriented in input sensor data) or how far away it is (or, e.g., scaled in input sensor data).

[0021] Embodiment 7 is a data processing device configured to perform a method according to any one of embodiments 1 to 6.

[0022] Embodiment 8 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6.

[0023] Embodiment 9 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 6.

[0024] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. 2 illustrates the training of a neural network according to one embodiment. Fig. 3 shows a flowchart illustrating a method for training a neural network according to one embodiment.

[0025] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0026] Various examples are described in more detail below.

[0027] In machine learning, a function is learned that maps input data to output data. During learning (e.g., training a neural network and / or model), the function is determined from an input data set (also called the training data set) that specifies a desired output (e.g., a desired classification of the input data or a regression result) for each input.

[0028] An example of an application of such a machine-learned function is object classification for autonomous driving, as described in Fig. 1 is illustrated.

[0029] It should be noted that, in the following, images or image data are understood very generally as a collection of data representing one or more objects or patterns. The image data can be provided by sensors that measure visible or invisible light, such as infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals.

[0030] In the example of object recognition in autonomous driving 100 of Fig. 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device 102.

[0031] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software according to which the vehicle control device 102 operates and data processed by the processor 103.

[0032] For example, the stored control software (computer program) includes instructions that, when executed by the processor, cause the processor 103 to implement a neural network 107.

[0033] The data stored in memory 104 may include, for example, image data captured by one or more cameras 105. The one or more cameras 105 may, for example, capture one or more grayscale or color photos of the surroundings of the vehicle 101.

[0034] The vehicle control device 102 can (by processing using the neural network 107), for example, based on the image data, determine whether and which objects, e.g., fixed objects such as traffic signs or road markings, or moving objects such as pedestrians, animals, and other vehicles, are present in the surroundings of the vehicle 101. For this purpose, the neural network 107 is trained accordingly, for example, on an external data processing device, from which it is loaded into the vehicle control device 102 after training.

[0035] The vehicle 101 can then be controlled by the vehicle control device 102 according to the results of the object determination. For example, the vehicle control device 102 can control an actuator 106 (e.g., a brake) to control the speed of the vehicle, e.g., to decelerate the vehicle.

[0036] In the example above, the neural network performs a classification task, but it can also perform a regression task, particularly in the context of a driver assistance system or autonomous driving, such as predicting the position of another road user or determining control variables for an anti-lock braking system (e.g. based on information about the surface (such as a camera image of the surface) on which the vehicle is moving).

[0037] For many neural network tasks, it is desirable for the neural network to be invariant under a set of transformations. For example, it should be able to recognize another vehicle as such, regardless of whether the other vehicle passes from the right or left of the vehicle 101 in which it is implemented.

[0038] Such invariance can be learned by the neural network given suitable training data, but the neural network may lose this invariance when applied to other data.

[0039] Equivariant neural networks aim to incorporate symmetries as an inductive bias into a network architecture. Starting with convolutional networks, which exhibit translational symmetry, the concept of an equivariant network can be generalized to rotations, permutations, scaling, and arbitrary matrix groups. A neural network can also be designed to be equivariant with respect to transformations learned directly from the training data. This can endow a neural network with reliable invariance (or equivariance) with respect to transformations. However, achieving this requires non-trivial architectural changes to the respective model, which often involve high memory and inference time overhead.

[0040] Non-equivariant neural networks can also learn invariance when trained with data augmentation. However, the properties of such learned invariances are still largely unexplored. This complicates the practical use of non-equivariant invariance learning, as it is unclear how reliable these learned invariances are when operating conditions change. This is a critical disadvantage of using models with learned invariances in practice.

[0041] According to various embodiments, a procedure is therefore provided in which, by means of regularization during training of an arbitrary (i.e., not necessarily equivariant) neural network, the neural network is made invariant with respect to (at least) a specific set of (e.g., similarity) transformations (e.g., a transformation group). The procedure enables reliable invariance, i.e., the invariance is maintained across the training data. The provided regularization does not require any change to the architecture of the respective neural network and merely consists of adding an additional term to the loss function used to train the neural network (i.e., to reduce the loss function, the neural network (i.e., its weights) are adapted, typically by means of backpropagation).

[0042] A similarity mapping or similarity transformation can be understood as a transformation of an object that leaves the object unchanged (including scaling, i.e., changes in distances, but maintaining angles and ratios). The set of all such transformations of an object with a composition as a link forms a (similarity) transformation group G. A (similarity) transformation group can, for example, be a group of rotations, translations, or scalings.

[0043] A neural network f parameterized by the set of parameters θ (typically a set of weights) θ : X → Y is considered invariant with respect to the group G, if f θ (g ◦ x) = f θ (x) for ∀g ∈ G, where g ◦ x represents the operation of a group element g on a data element x.

[0044] One method for learning invariance is data augmentation, which means that, for example, in each training iteration {gi}i=1,B where B is the size of a respective training (mini-) batch, from which G is sampled and applied to the training batch before the output of the neural network is determined for it and from this the value of the loss function is calculated (to reduce which the neural network is then adapted).

[0045] In the following, the evaluation of neural network f θ learned invariance (separately from the performance for the respective data processing task), the invariance error If used. The invariance error I:X×G→ℝ is given by: If(G,D)=Ex∼DEg∼G(d[fθ(x),fθ(gox)]) where the respective training data set and d:Y×Y→ℝ is a distance measure between the outputs of the neural network for x and g ◦ x. For classification tasks, this distance function d corresponds, for example, to the Kullback-Leibler divergence between the output logits.

[0046] Intuitively, a smaller invariance error If that f θ a higher degree of invariance with respect to G on the data set has learned. Training with stronger data augmentation would, for example, lead to a lower invariance error.

[0047] In addition to data augmentation, another approach to learning invariance is If to be explicitly minimized. In practice, this amounts to regularizing the loss function (which contains a loss for the respective data processing task) with the invariance error objective of equation (1). For a batch of training data, the invariance error can be estimated as follows: If(G,{xj}j=1B)≃∑iB∑g∈Gd[fθ(xi),fθ(goxi)]

[0048] G is a discrete transformation group, but can also be any set of transformations.

[0049] With this explicit regularization using the invariance error according to Equation (2), invariance to a set of transformations can be learned better than with data augmentation. However, invariance learned in this way still extends relatively poorly beyond the training data.

[0050] According to various embodiments, a regularization is therefore provided, the basis of which can be seen in an analysis of the invariance error from the perspective of Lipschitz continuity.

[0051] The invariance error If is Lipschitz-continuous with respect to the data set , if a constant L exists such that: ‖If(G,x)−If(G,x′)‖2≤|L⋅‖x−x′‖2 for ∀x,x' ∈D. Intuitively, the Lipschitz constant in equation (3) is the upper bound for the change in the invariance error when measured for two different data items. In other words, a small L guarantees that the invariance error does not change drastically between different data items. A small L can thus be seen as a hallmark of reliable invariance. Accordingly, various methods use a method for learning invariance that yields an invariance error with a small Lipschitz constant.

[0052] The Lipschitz constant cannot be accessed directly. However, consider a data sample x∈D and its neighborhood Ω(x) (within the data set, i.e., surrounding data elements). According to the mean value theorem, for x' ∈ Ω(x) ‖If(G,x′)−If(G,x)‖2=‖∇If(γ)(x′−x)‖2 where γ = αx - (1 - α)x'. According to the Cauchy-Schwarz inequality: ‖If(G,x′)−If(G,x)‖2≤‖∇If(γ)‖2‖(x′−x)‖2

[0053] Please note that ‖∇If(x)‖2 the Lipschitz constant of If as x' → x. Therefore, according to various embodiments ‖∇If(x)‖2 during training in addition to the loss related to the data processing task (e.g. classification loss or regression loss) for which the neural network is trained, so that the neural network learns reliable invariance.

[0054] Formally, given a (mini-)batch of training data elements and a transformation group G (a discrete transformation group is assumed here, but embodiments are not limited to this), the following loss is included in the loss function (in addition to the loss related to the data processing task for which the neural network is to be trained) to regularize the training: Rf(G,{xi}i=1B)=1B1|G|∑iB∑g∈G‖∇xd[fθ(xi),fθ(g∘xi)]‖2 where |G| denotes the cardinality of the transformation group G.

[0055] This regularization guides the learning of an invariance to shallow minima by penalizing a high norm of the gradient of the invariance target.

[0056] Fig. 2 illustrates the training of a neural network 201 according to the procedure described above.

[0057] The loss 202 with respect to a data processing task (e.g. classification or regression) 203 to be performed by the neural network 201 for a batch of training data 204 (e.g. determined using ground truth data such as labels or also by a loss for self-supervised learning) is supplemented by the regularization loss 205 according to equation (6) for a respective set of (one or more) transformations (e.g. transformation group, e.g. similarity transformation group) 206 (i.e. added to the loss 202, if necessary with a weighting compared to the loss 202) and the neural network is adapted to reduce the resulting total loss (i.e. loss function value) (e.g. by means of back propagation). This leads (e.g., in a sequence of such adaptations over several batches of training data, e.g., until a convergence criterion is met) to a trained version of the neural network 207 that is invariant with respect to the set of transformations 206.

[0058] In summary, according to various embodiments, a method is provided as described in Fig. 3 shown.

[0059] Fig. 3 shows a flowchart 300 illustrating a method for training a neural network according to one embodiment.

[0060] In 301, an output of the neural network for a training data set is obtained by applying the neural network to training data items of a training data set.

[0061] In 302, a loss is calculated for the output of the neural network with respect to a data processing task for which the neural network is to be trained.

[0062] In 303, the loss is supplemented by an additional loss which, for each of the training data elements and for each transformation of a set of (one or more) transformations, includes the gradient (with respect to the training data element, ie at the location of the training data element and with the training data element as a variable, ie with respect to the dependence on the training data element) of the difference between an output of the neural network for the training data element and the output of the neural network for the training data element transformed by means of the transformation (ie the additional loss is higher the higher the deviation, ie the additional loss penalizes an increase in the gradient).

[0063] In 304, the neural network is adapted to reduce the loss added by the additional loss.

[0064] The procedure of Fig. 3 can be performed by one or more computers having one or more data processing units. The term “data processing unit” can be understood as any type of entity that enables the processing of data or signals. The data or signals can, for example, be treated according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit can comprise or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0065] According to various embodiments, the method is therefore particularly computer-implemented.

[0066] After training, the neural network can be applied to sensor data acquired by at least one sensor. The output of the neural network thus provides a result about a physical state of an environment of the at least one sensor and / or the at least one sensor itself, or the method can include using the output of the trained neural network, which it provides in response to an input of sensor data, as such a result.

[0067] In other words, the method may comprise inferring or predicting the physical state of an existing real object from measurements of physical properties (i.e., sensor data concerning the object) by means of the trained neural network (i.e., its output in response to the sensor data).

[0068] The approach of Fig.3 serves, for example, to generate a control signal for a robotic device. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the technical system is learned, and the technical system is then controlled accordingly.

[0069] For example, after training, the neural network is used to select a control action and generate a control signal for the vehicle. A control action for which one or more control signals are generated could be, for example, a distance (e.g., moving one meter to the right), a speed, an acceleration, etc.

[0070] Various embodiments may receive and use (e.g., time series of) sensor data from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc.

Claims

[1] A method for training a neural network (107, 201), comprising: Determining an output of the neural network (107, 201) for a training data set (204) by applying the neural network (107, 201) to training data elements of a training data set (204); Calculating a loss (202) for the output of the neural network (107, 201) with respect to a data processing task (203) for which the neural network (107, 201) is to be trained; Supplementing the loss (202) by an additional loss (205) which, for each of the training data elements and for each transformation of a set of transformations (206), includes the gradient of the difference between an output of the neural network (107, 201) for the training data element and the output of the neural network (107, 201) for the training data element transformed by means of the transformation; and Adapting the neural network (107, 201) to reduce the loss (202) supplemented by the additional loss (205). [2] The method of claim 1, wherein the set of one or more transformations (206) includes a set of one or more similarity transformations. [3] The method of claim 1 or 2, wherein the set of one or more transformations (206) includes one or more color transformations. [4] Method according to one of claims 1 or 3, wherein the data processing task (203) is a processing of sensor data. [5] Method according to one of claims 1 or 4, wherein the data processing task (203) is a classification or a regression. [6] Method according to one of claims 1 to 5, comprising selecting the set of one or more transformations (206) as transformations to which the data processing task (203) requires invariance. [7] Data processing device arranged to carry out a method according to one of claims 1 to 6. [8] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6. [9] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 6.

Images