Methods used to train neural networks

Abstract

A computer-implemented method for training a neural network (60), said neural network being in particular set up for classifying physical measurement parameters, wherein, in the case of supplying an input signal (x) and a respective desired output signal (y T ), an adaptation of the parameters (θ) of the neural network (60) is carried out on the basis of the output signal (y) of the neural network (60), wherein the adaptation of these parameters (θ) is carried out on the basis of a determined gradient (g), characterized in that the components (#imgabs0#) of the determined gradient (g) are scaled on the basis of which layer (S1,..., S5) of the neural network the parameters (θ) corresponding to these components belong to.

Description

Technical Field

[0001] The present invention relates to a method for training a neural network, a training system, an application of a neural network trained in this way, a computer program and a machine-readable storage medium. Background Art

[0002] From "Improving neural networks by preventing co-adaptation of feature detectors", arXiv preprint arXiv:1207.0580v1, Geoffrey E. Hinton, NitishSrivastava, Alex Krizhevsky, llya Sutskever, Ruslan R. Salakhutdinov (2012), a method for training neural networks is known, in which feature detectors are randomly omitted during training. This method is also known by the name "Dropout".

[0003] From “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, arXiv preprint arXiv:1502.03167v3, Sergeyloffe, Christian Szegedy (2015) a method for training a neural network is known, in which the input parameters to a layer are normalized to a mini-batch of training examples. Summary of the invention

[0004] Advantages of the present invention

[0005] In contrast, the method having the features of independent claim 1 has the advantage that overfitting of the parameters of the neural network can be prevented particularly well.

[0006] Advantageous embodiments are the subject matter of the dependent claims.

[0007] Disclosure of the Invention

[0008] If there is enough training data, so-called "Deep Learning" methods, i.e. (deep) artificial neural networks, can be used to efficiently determine the input space V0 and the output space V kThis can be, for example, the classification of sensor data, especially image data, that is, the mapping from sensor data or image data to categories. It also specifies k mappings between these spaces These mappings Each mapping in is usually called a layer. By choosing the space appropriately The weight of the case to be parameterized. k layers Weight Also collectively referred to as weight , and from the input space V0 to the output space V k The mapping is , which is mapped according to the (Where weights are explicitly indicated as subscripts ) is .

[0009] In being defined Given a probability distribution D, the task of training a neural network is to set the weights Determine the expected value Φ of the cost function L

[0010] (1)

[0011] is minimized. In this case, the cost function L represents the cost of The input parameter x is determined D To the output space V k Parameters in The mapping of k The actual output parameter y in D A measure of the distance between .

[0012] A "deep neural network" can be understood as a neural network with at least two hidden layers.

[0013] To minimize this expected value Φ, gradient-based methods can be used. These gradient-based methods determine the gradient with respect to the weight w The gradient Usually with the help of training data is approximated by is approximated by , where the index j is chosen from a so-called epoch. In this case, an epoch is a permutation of the labels {1, ..., N} of the available training data points.

[0014] In order to expand the training dataset, so-called data augmentation (also called augmentation) can be used. In this case, for each index j in the epoch, Position selection enhancement , where the input signal x j Enhanced input value In this case, The input signal x can be j The set of typical variations (including the input signal x j itself), these changes make the input signal x j The classification, that is, the output signal of the neural network remains unchanged.

[0015] However, this epoch-based sampling is not completely consistent with the definition according to equation (1), because each data point is selected exactly once during the epoch. The definition according to equation (1) is based on independently extracted data points. In other words, the premise of equation (1) is the extraction of data points "with replacement", while epoch-based sampling implements the extraction of data points "without replacement". This may lead to the fact that the premise of the mathematical convergence proof does not exist (because if N examples are extracted from a set of N data points, the probability of extracting each of these data points exactly once is less than (for N > 2), while this probability is always equal to 1 in the case of epoch-based sampling.

[0016] If data augmentation is used, this statistical effect can be further enhanced because there is a collection of The elements of the visual enhancement function However, it cannot be excluded that: ,in In this case, it is difficult to use the collection A statistically correct mapping of these augmentations is performed because for each input data x j The effect need not be equally significant for the rotation. Thus, for example, rotation may have no effect on a circular object, but may affect a general object very strongly. The size of the input data x j , which may be problematic for inverse training methods.

[0017] Finally, the number of training data points N is a parameter that is usually difficult to set. If N is chosen too large, the running time of the training method may be excessively extended; if N is chosen too small, convergence cannot be ensured, because the mathematical proof of convergence is generally based on assumptions that are not then satisfied. In addition, it is unclear at which point in time the training should be reliably ended. If in this case, part of the data points is used as an evaluation data set and the convergence quality is determined with the aid of this evaluation data set, this may result in: overfitting (English: "over-fitting") of the weights w occurs with respect to the data points of the evaluation data set, which not only reduces data efficiency, but also may deteriorate the performance of the network when the network is applied to training data different from these training data. This may lead to a reduction in the so-called "generalizability" (English: "generalizability").

[0018] In order to reduce overfitting, the “Dropout” method mentioned at the beginning can be used to make the information stored in the hidden layer randomly sparse.

[0019] In order to improve the stochasticity of the training process, statistical parameters μ and σ for so-called mini-batches can be introduced by using so-called batch normalization layers (English: "batchnormalization layer"), which are updated probabilistically during the training process. In the case of inference, the values ​​of these parameters μ and σ are selected as fixedly predeterminable values, for example as estimated values ​​from training by extrapolation of exponential decay behavior.

[0020] If the layer with index i is a batch normalization layer, the weights In the case of gradient descent, these weights are not updated. So with the weights of the remaining layers k This increases the complexity of the implementation.

[0021] Furthermore, the size of the mini-batch is a parameter which often influences the training result and must therefore be set as well as possible as a further hyperparameter, for example within the framework of a (possibly complex) architecture search.

[0022] Thus, in a first aspect, the invention relates to a method for training a neural network, which is set up in particular for classifying physical measurement variables, wherein adaptation of parameters of the neural network is carried out as a function of an output signal of the neural network when an input signal and an associated desired output signal are supplied, wherein the adaptation of these parameters depends on determined gradients, characterized in that components of the determined gradients are scaled as a function of the layer of the neural network to which the parameters corresponding to these components belong.

[0023] In this context, “scaling” may be understood such that a component of the determined gradient is multiplied by a factor that depends on the layer.

[0024] The scaling may depend in particular on the position, ie the depth, of the layer within the neural network.

[0025] The depth can be characterized, in particular specified, for example, by the number of layers through which a signal supplied to an input layer of a neural network must propagate before it is applied to this layer for the first time as input signal.

[0026] In one embodiment, it can be provided that the scaling also depends on which feature in the feature map the corresponding component of the determined gradient belongs to.

[0027] In particular, it may be provided that the scaling depends on the size of the receptive field of the feature.

[0028] It has been recognized that, especially in convolutional networks, the weights of the feature maps are multiplied by the information accumulation of the features of the receptive field, which may be the reason for overfitting for these weights. This is effectively prevented with the proposed method.

[0029] In a particularly simple and efficient alternative, it can be provided that the scaling depends on the resolution of the layer. In particular, the scaling depends on the quotient of the resolution of the layer and the resolution of the input layer.

[0030] It has been realized that through this, the size of the receptive field can be approximated very simply and efficiently.

[0031] In a further aspect, the invention relates to a method in which a neural network is trained with the aid of a training data set, wherein for the training pairs consisting of an input signal and an associated desired output signal are (randomly) extracted from the training data set in order to adapt parameters of the neural network as a function of the output signal of the neural network when the input signal and the desired output signal are supplied, wherein the extraction of pairs is always carried out from the entire training data set.

[0032] In a preferred embodiment of this aspect, it is provided that the pairs are extracted independently of which pairs have already been extracted previously during the training process.

[0033] In other words, the sampling of pairs, i.e. data points, from the training data set corresponds to "extraction with replacement". This breaks the current paradigm where training examples of the training data set are extracted by "extraction without replacement". Said "extraction with replacement" seems disadvantageous at first, because it cannot be ensured that every data point from the training data set is actually used within a given number of training examples.

[0034] However, the reliability of the trained system can be ensured, which is essential in particular for safety-critical applications. Surprisingly, this advantage is achieved without having to accept a decrease in the performance that can be achieved at the end of the training (for example in image classification). In addition, the interface with other sub-blocks of the training system that can be used to train the neural network is greatly simplified.

[0035] Optionally, the extracted pairs can also be further enhanced. That is, for some or all input signals contained in the training data set (as components of pairs), a set of enhancement functions can be specified to which the input signals can be subjected. The selection of the corresponding enhancement function can also be carried out randomly, preferably independently of which pairs and / or which enhancement functions have been extracted previously in the training process.

[0036] In one embodiment, it can be provided that the decimated input signal of the pair is enhanced using an enhancement function is enhanced, that is to say the input signal is replaced by the image of the input signal in the case of the increment function.

[0037] Preferably, in this case it is provided that: from the possible enhancement functions In particular, randomly select the enhancement function from the set , where the set depends on the input signal.

[0038] In this case, it can be provided that, when randomly selecting a pair from the training data set, the probability of selecting a predeterminable pair depends on a possible enhancement function of the input signal of the predeterminable pair The number of.

[0039] For example, the probability may be a predetermined parameter. In particular, the probability is advantageously selected to be proportional to the number of possible enhancement functions. This allows for full consideration of the fact that some enhancement functions leave the input signal unchanged, so that the cardinality of the set of enhancement functions (that is, the number of elements of the set) may vary greatly between input signals. By taking this full consideration into account, possible problems in the inverse training method can be avoided.

[0040] In another aspect of these extended schemes, it can be provided that the parameters are adapted as a function of the determined gradient and that, in order to determine the gradient, the estimate m1 of the gradient is refined by taking into account a gradually increasing number of pairs extracted from the training data set until a predefinable interruption criterion is met, which depends on the estimate m1 of the gradient.

[0041] This means in particular that the parameters are not adapted until a predefinable termination condition has been met.

[0042] This differs from conventional methods in the prior art, such as stochastic gradient descent, in which the gradient is always averaged over a predeterminable mini-batch. The mini-batch has a predeterminable size that can be set as a hyperparameter. By continuously adding pairs from the training data set, the proposed method allows this determination to be carried out until the gradient reliably points in the ascending direction.

[0043] Furthermore, the mini-batch size is a hyperparameter to be optimized. Since this optimization can be omitted, the method is more efficient and more reliable, since overfitting can be suppressed more effectively and the batch size as a hyperparameter is eliminated.

[0044] In particular, the predefinable termination criterion can also depend on the covariance matrix C of the estimated value m1 of the gradient.

[0045] This makes it particularly easy to ensure that the gradient is reliably oriented in the rising direction.

[0046] For example, the predeterminable interruption condition may include the following condition: whether the estimated value m1 and the covariance matrix C satisfy the condition for the predeterminable confidence value λ .

[0047] This condition is therefore used to introduce a probabilistic interruption criterion. Thus, it is also possible to use a predeterminable confidence value to ensure that the gradient is in an increasing direction at the confidence value λ.

[0048] In another aspect of these extended schemes, it can be provided that the neural network includes a scaling layer, wherein the scaling layer maps an input signal attached to an input of the scaling layer to an output signal attached to an output of the scaling layer, so that the output signal attached to the output is a rescaled signal of the input signal, wherein parameters characterizing the rescaling can be fixedly predetermined.

[0049] Preferably, it can be provided that the scaling layer maps an input signal attached to the input of the scaling layer to an output signal attached to the output of the scaling layer, so that the mapping corresponds to a projection on a sphere, wherein the center c and / or the radius p of the sphere can be fixedly predetermined. Alternatively, it is also possible that these parameters are adapted during the training process like other parameters of the neural network.

[0050] In this case, the mapping can be obtained by the equation with the first norm (N1) and the second norm (N2) In this context, the term "norm" is to be understood in a mathematical sense.

[0051] In a development that is to be calculated particularly efficiently, it can be provided that the first norm (N1) and the second norm (N2) are selected to be identical. Alternatively or additionally, the first norm (N1) can be L ∞ In particular, the norm can also be calculated particularly efficiently even if the first norm (N1) and the second norm (N2) are selected differently.

[0052] Alternatively, it can be specified that: where the first norm (N1) is L 1 Norm. The choice of the first norm is advantageous for the sparsity of the output signal of the scaling layer. This is advantageous, for example, for the compression of neural networks, since weights with the value 0 do not contribute to the output value of the layer of these weights.

[0053] Thus, a neural network having such a layer can be used in a particularly memory-efficient manner, in particular in conjunction with compression methods.

[0054] In the described variant of the first norm (N1), it can be advantageously provided that the second norm (N2) is L 2 The method can thereby be implemented particularly easily.

[0055] In this case, it is particularly advantageous to use the deterministic Newton method to solve the equation To solve.

[0056] Surprisingly, it has been found that the method is particularly effective if an input signal having a plurality of important, ie heavily weighted, features is present at the input of the scaling layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Subsequently, embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In the drawings:

[0058] Figure 1 Schematically illustrates the construction of an embodiment of a control system;

[0059] Figure 2 Schematically illustrates an embodiment for controlling an at least partially autonomous robot;

[0060] Figure 3 An embodiment for controlling a production system is schematically shown;

[0061] Figure 4 An embodiment for controlling a personal assistant is schematically shown;

[0062] Figure 5 An embodiment of a system for controlling access is schematically shown;

[0063] Figure 6 An embodiment for controlling a monitoring system is schematically shown;

[0064] Figure 7 An embodiment for controlling a medical imaging system is schematically shown;

[0065] Figure 8 A training system is schematically shown;

[0066] Fig. 9 The structure of the neural network is schematically shown;

[0067] Fig.10 Schematic diagram showing information forwarding within a neural network;

[0068] Fig.11 The implementation method of the training method is shown in a flow chart;

[0069] Fig.12 An embodiment of a method for estimating a gradient is shown in a flow chart;

[0070] Fig.13 An alternative embodiment of a method for estimating a gradient is shown in a flow chart;

[0071] Fig.14 An embodiment of a method for scaling an estimated gradient is shown in a flow chart;

[0072] Fig.15 An embodiment for implementing a scaling layer within a neural network is shown as a flow chart;

[0073] Fig.16 A method for operating a trained neural network is shown in a flow chart. DETAILED DESCRIPTION

[0074] Figure 1An actuator 10 is shown interacting with a control system 40 in its surroundings 20. The actuator 10 and the surroundings 20 are also collectively referred to as an actuator system. The state of the actuator system is detected at preferably uniform time intervals by means of a sensor 30, which can also be provided by a plurality of sensors. The sensor signal S of the sensor 30 or each sensor signal S in the case of a plurality of sensors is transmitted to the control system 40. The control system 40 thus receives a sequence of sensor signals S. The control system 40 determines therefrom a control signal A, which is transmitted to the actuator 10.

[0075] The sensor 30 is any sensor that detects the state of the surroundings 20 and transmits it as a sensor signal S. For example, it can be an imaging sensor, in particular an optical sensor such as an image sensor or a video sensor, or a radar sensor, or an ultrasonic sensor, or a laser radar (LiDAR) sensor. It can also be a sound sensor that receives structure-borne sound or voice signals, for example. The sensor can also be a position sensor (such as a GPS) or a motion sensor (such as a single-axis or multi-axis acceleration sensor). Sensors that characterize the orientation of the actuator 10 in the surroundings 20 (such as a compass) are also possible. Sensors that detect the chemical composition of the surroundings 20, such as oxygen sensors, are also possible. Alternatively or in addition, the sensor 30 can also include an information system that determines information about the state of the actuator system, such as a weather information system, which determines the current or future state of the weather in the surroundings 20.

[0076] The control system 40 receives a sequence of sensor signals S of the sensor 30 in an optional receiving unit 50, which converts the sequence of sensor signals S into a sequence of input signals x (alternatively, the sensor signals S can also be used directly as input signals x). The input signals x can be, for example, fragments of the sensor signals S or further processing of the sensor signals S. The input signals x can, for example, include image data or images, or include individual frames of a video recording. In other words, the input signals x are determined from the sensor signals S. The input signals x are fed to the neural network 60.

[0077] The neural network 60 is preferably parameterized by parameters θ, including weights w, for example, which are stored in a parameter memory P and provided by the latter.

[0078] The neural network 60 determines the output signal y as a function of the input signal x. Typically, the output signal y encodes information about the classification of the input signal x. The output signal y is supplied to an optional modification unit 80, which determines control signals A therefrom, which are supplied to the actuator 10 in order to control the actuator 10 accordingly.

[0079] The neural network 60 can be set up, for example, to detect persons and / or road signs and / or traffic lights and / or vehicles in the input signal (i.e. to classify whether these persons and / or road signs and / or traffic lights and / or vehicles are present) and / or to classify them according to their type (this can be done in the form of semantic segmentation, region by region, in particular pixel by pixel).

[0080] The actuator 10 receives the control signal A, is controlled accordingly and performs a corresponding action. In this case, the actuator 10 may include a control logic (not necessarily structurally integrated) which determines a second control signal based on the control signal A and then uses it to control the actuator 10 .

[0081] In other embodiments, the control system 40 includes the sensor 30. In still other embodiments, the control system 40 also includes the actuator 10, alternatively or additionally.

[0082] In other preferred embodiments, the control system 40 includes one or more processors 45 and at least one machine-readable storage medium 46 on which commands are stored, which, when executed on the processor 45, cause the control system 40 to implement a method for operating the control system 40.

[0083] In an alternative embodiment, a display unit 10 a is provided as an alternative to the actuator 10 or in addition to the actuator 10 .

[0084] Figure 2 An exemplary embodiment is shown in which a control system 40 is used to control an at least partially autonomous robot, here an at least partially automated motor vehicle 100 .

[0085] The sensor 30 may be a combination of Figure 1 The mentioned sensors are preferably one or more video sensors and / or one or more radar sensors and / or one or more ultrasound sensors and / or one or more lidar sensors and / or one or more position sensors (eg GPS), which are preferably arranged in motor vehicle 100 .

[0086] The neural network 60 may detect, for example, an object in the surroundings of the at least partially autonomous robot based on the input data x. The output signal y may be information characterizing the location of the object in the surroundings of the at least partially autonomous robot. The output signal A may then be determined based on and / or corresponding to this information.

[0087] The actuator 10 preferably arranged in the motor vehicle 100 may be, for example, a brake device, a drive device or a steering device of the motor vehicle 100. Subsequently, the control signal A may be determined such that the actuator or actuators 10 are controlled such that, in particular when certain classes of objects are involved, such as pedestrians, the motor vehicle 100, for example, avoids a collision with an object identified by the neural network 60. In other words, the control signal A may be determined according to and / or in correspondence with the determined class.

[0088] Alternatively, the at least partially autonomous robot may also be another mobile robot (not shown), for example a robot that moves by flying, floating, diving or walking. The mobile robot may also be, for example, an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In these cases, the control signal A may also be determined so that the drive device and / or the steering device of the mobile robot are controlled so that the at least partially autonomous robot, for example, avoids a collision with an object identified by the neural network 60.

[0089] In another alternative, the at least partially autonomous robot can also be a garden robot (not shown) which uses an imaging sensor 30 and a neural network 60 to determine the type or state of plants in the surroundings 20. The actuator 10 can then be, for example, a feeder of chemicals. The control signal A can be determined as a function of the determined type of plant or the determined state of the plant, so that an amount of chemical corresponding to the determined type or the determined state is applied.

[0090] In other alternatives, the at least partially autonomous robot can also be a household appliance (not depicted), in particular a washing machine, a stove, an oven, a microwave oven or a dishwasher. The state of an object processed by a household appliance can be detected by a sensor 30, such as an optical sensor, for example, the state of the laundry in the washing machine can be detected in the case of a washing machine. Then, the type or state of the object can be determined using a neural network 60 and the type or state of the object can be characterized by an output signal y. Then, the control signal A can be determined to make the household appliance control according to the determined type or determined state of the object. For example, in the case of a washing machine, the washing machine can be controlled according to what material the laundry is made of. Then, the control signal A can be selected according to which material the laundry is determined.

[0091] Figure 3 The following exemplary embodiment is shown in which a control system 40 is used to control a production machine 11 of a production system 200 by controlling an actuator 10 which controls the production machine 11. The production machine 11 can be, for example, a machine for punching, sawing, drilling and / or cutting.

[0092] The sensor 30 may be a combination of Figure 1 The mentioned sensor is preferably, for example, an optical sensor which detects properties of the finished product 12. It is possible that the actuator 10 controlling the production machine 11 is actuated as a function of the determined properties of the finished product 12, so that the production machine 11 accordingly carries out the subsequent processing steps of the finished product 12. It is also possible that the sensor 30 determines the properties of the finished product 12 processed by the production machine 11 and, as a function of this, adapts the actuation of the production machine 11 for the next finished product.

[0093] Figure 4 The following embodiment is shown, in which the control system 40 is used to operate the personal assistant 250. The sensor 30 can be combined with Figure 1 The mentioned sensors. Preferably, the sensor 30 is an acoustic sensor which receives a speech signal of the user 249. Alternatively or additionally, the sensor 30 can also be configured to receive an optical signal, for example a video image of the gesture of the user 249.

[0094] Based on the signal of the sensor 30, the control system 40 determines an actuation signal A for the personal assistant 250, for example by means of a neural network performing gesture recognition. The determined actuation signal A is then transmitted to the personal assistant 250 and the personal assistant is thus controlled accordingly. The determined actuation signal A can in particular be selected such that it corresponds to a desired actuation envisaged by the user 249. The envisaged desired actuation can be determined based on the gesture recognized by the neural network 60. The control system 40 can then select an actuation signal A for transmission to the personal assistant 250 based on the envisaged desired actuation and / or select an actuation signal A for transmission to the personal assistant 250 corresponding to the envisaged desired actuation.

[0095] The corresponding control may include, for example, personal assistant 250 calling up information from a database and reproducing this information in a form that is acceptable to user 249 .

[0096] Instead of personal assistant 250 , a domestic appliance (not shown), in particular a washing machine, a stove, an oven, a microwave or a dishwasher, can also be provided in order to be controlled accordingly.

[0097] Figure 5 The following embodiment is shown, in which the control system 40 is used to operate the access system 300. The access system 300 may include, for example, a physical access control device, such as a door 401. The sensor 30 may be combined with Figure 1The mentioned sensor is preferably an optical sensor (e.g. for detecting images or video data) which is set up to detect faces. The detected image can be interpreted by means of the neural network 60. For example, the identity of a person can be determined. The actuator 10 can be a lock which activates an access control device or deactivates the access control device according to the control signal A, for example, opens the door 401 or does not open the door 401. For this purpose, the control signal A can be selected according to the interpretation of the neural network 60, for example, according to the determined identity of the person. Instead of a physical access control device, a logical access control device can also be provided.

[0098] Figure 6 The following embodiment is shown, in which the control system 40 is used to control the monitoring system 400. Figure 5 The embodiment shown in FIG. 1 differs from this embodiment in that a display unit 10a is provided instead of the actuator 10, and the display unit is controlled by the control system 40. For example, the neural network 60 can determine whether an object photographed by the optical sensor is suspicious, and the control signal A can then be selected so that the object is highlighted in color by the display unit 10a.

[0099] Figure 7 The following embodiment is shown, in which the control system 40 is used to control a medical imaging system 500, such as an MRT apparatus, an X-ray apparatus or an ultrasound apparatus. The sensor 30 can be provided, for example, by an imaging sensor, and the display unit 10a is controlled by the control system 40. For example, the neural network 60 can determine whether the area recorded by the imaging sensor is abnormal, and the control signal A can then be selected so that the area is highlighted by the display unit 10a with a color.

[0100] Figure 8 An exemplary embodiment of a training system 140 for training a neural network 60 by means of a training method is schematically shown. A training data unit 150 determines suitable input signals x, which are supplied to the neural network 60. For example, the training data unit 150 trains a computer-implemented database in which training data records are stored and, for example, randomly selects input signals x from the training data records. Optionally, the training data unit 150 also determines the desired or “actual” output signal y assigned to the input signal x. T , these output signals are supplied to an evaluation unit 180 .

[0101] The artificial neural network x is designed to determine associated output signals y as a function of the input signals x supplied to it. These output signals y are supplied to an evaluation unit 180 .

[0102] The evaluation unit 180 can, for example, use a method that depends on the output signal y and the desired output signal y. T The cost function To characterize the performance of the machine learning system 60. The parameter θ can be calculated based on the cost function to be optimized.

[0103] In other preferred embodiments, the training system 140 includes one or more processors 145 and at least one machine-readable storage medium 146 on which commands are stored, which, when executed on the processor 145, cause the control system 140 to implement the training method.

[0104] Fig. 9 A possible configuration of a neural network 60 is shown by way of example, which is provided as a neural network in this embodiment. The neural network comprises a plurality of layers S1, S2, S3, S4, S5 in order to determine an output signal y attached to an output of an output S5 as a function of an input signal x supplied to an input of an input layer S1. In this case, each of these layers S1, S2, S3, S4, S5 is configured to determine a (possibly multidimensional) output signal z1, z2, z4, z5, y attached to an output of the corresponding layer S1, S2, S3, S4, S5 as a function of a (possibly multidimensional) input signal x, z1, z3, z4, z6 attached to an input of the corresponding layer S1, S2, S3, S4, S5. Such output signals are also referred to as feature maps (English: “feature maps”), in particular in image processing. In this case, it is not necessary for these layers S1, S2, S3, S4, S5 to be arranged so that all output signals entering other layers as input signals enter the immediately following layer from the previous layer. As an alternative, skip connections (English: "Skip Connections") or loop connections are also possible. Of course, it is also possible that the input signal x enters multiple layers of these layers; or the output signal y of the neural network 60 is composed of output signals of multiple layers.

[0105] The output layer S5 can be given, for example, by an Argmax layer (i.e., a layer that selects the name of the input terminal whose assigned input value is the maximum input value among multiple input terminals with respectively assigned input values), and one or more of the layers S1, S2, and S3 can be given, for example, by a convolutional layer.

[0106] Advantageously, the layer S4 is designed as a scaling layer, which is designed such that an input signal (x) attached to the input of the scaling layer (S4) is mapped to an output signal (y) attached to the output of the scaling layer (S4), so that the output signal (y) attached to the output is a rescaling of the input signal (x), wherein parameters characterizing the rescaling can be fixedly predetermined. Fig.15 An embodiment of a method that can be performed by the scaling layer S4 is described.

[0107] Fig.10 The schematic diagram illustrates the information forwarding within the neural network 60. Here, three multidimensional signals within the neural network 60 are schematically shown, namely the input signal x and the later feature maps z1, z2. In this embodiment, the input signal x has The spatial resolution of pixels, The first feature map z1 of pixels, In this embodiment, the resolution of the second feature map z2 is lower than the resolution of the input signal x, but this is not mandatory.

[0108] The features of the second feature map z2 are also shown If the function for determining the second feature map z2 from the first feature map z1 is shown, for example, by a convolutional layer or a fully connected layer (English: "fully connected layer"), it is also possible that: multiple features of the first feature map z1 are added to the feature map z2 However, it is also possible that only the unique feature of the first feature map z1 is added to the feature Determination of the value of .

[0109] In this case, “addition” can advantageously be understood as: deriving a combination of the values ​​of the parameters characterizing the function used to determine the second characteristic map z2 from the first characteristic map z1 and the values ​​of the first characteristic map z1, such that the characteristic map z2 is The value of depends on the value of the added features. The sum of these added features is Fig.10 The area is called Be.

[0110] One or more features of the input signal x are added to each feature of the region Be The addition of the input signal x affects the characteristics of the region Be. The set of all features that determine at least one feature in is called a feature In other words, the receptive field rF of feature The receptive field rF includes the direct or indirect (in other words: at least indirect) addition of the input signal x to the feature The determination of All those characteristics of the value of .

[0111] Fig.11 The sequence of a method for training neural network 60 according to one specific embodiment is shown in a flow chart.

[0112] First (1000), a training data set X is provided, which includes an input signal x i and the output signal y i The pair The learning rate η is initialized, for example, to η = 1.

[0113] Furthermore, optionally, for example, if in step 1100 this part of the method is used Fig.12 In the embodiment illustrated in FIG. 1 , the first set G and the second set N are initialized. If the method of this part should be used in step 1100 Fig.13 In the embodiment illustrated in FIG, the initialization of the first set G and the second set N can be omitted.

[0114] Initialization of the first set G and the second set N can be performed as follows: include those pairs of the training data set X that have been extracted during the current epoch of the training method The first set G is initialized to an empty set. It includes those pairs of the training dataset X that have not been extracted during the current epoch. The second set N is obtained by assigning all pairs of the training data set X to the second set. to be initialized.

[0115] Now (1100), with the input signal x from the training dataset X i and the output signal y i The pair , estimate the characteristic parameters The gradient g with respect to the function θ is . Combined Fig.12 or 13 to describe an embodiment of the method.

[0116] Then (1200), optionally scaling the gradient g is performed. Fig.14 An embodiment of the method is described below.

[0117] Following (1300), an adaptation of the learning rate η is optionally performed. In this case, the learning rate η can be reduced by a predeterminable learning rate reduction factor, for example, as long as the number of traversed epochs is divisible by a predeterminable number of epochs, for example 5. (For example )(Right now ).

[0118] Then (1400), the parameters θ are updated with the aid of the determined and possibly scaled gradient g and learning rate η. For example, the parameters θ are Alternative.

[0119] Now (1500), it is checked whether the method has converged by means of a predeterminable convergence criterion. For example, it can be determined based on the absolute change in the parameter θ (e.g. between the last two epochs) whether the convergence criterion has been met. For example, it is just when L with respect to the change in all parameters θ between the last two epochs is 2 The convergence criterion can be met when the norm is smaller than a predeterminable convergence threshold.

[0120] If it is determined that the convergence criterion is met, the parameter θ is used as the learned parameter and the method ends. If it is not determined that the convergence criterion is met, the method branches back to step 1100 .

[0121] Fig.12 An exemplary method for determining the gradient g in step 1100 is illustrated in a flow chart.

[0122] First (1110), a predeterminable number bs of pairs of the training data set X (without replacement) should be extracted, i.e. selected. And distribute them to batches B. The predefined number bs is also called batch size. Batch B is initialized to an empty set.

[0123] To this end, it is checked (1120) whether the batch size bs is larger than the number of pairs present in the second set N The number of.

[0124] If the batch size bs is not larger than the number of pairs present in the second set N The number of pairs is randomly selected from the second set N (1130), that is, bs pairs are selected. And add it to batch B.

[0125] If bs is greater than the number of pairs present in the second set N If the number of pairs is s, then all pairs of the second set N whose number is s are extracted (1140), i.e. selected, and added to the batch B, and the remaining pairs, i.e. bs - s, are extracted from the first set G, i.e. selected.

[0126] Following step (1150) (1130) or (1140), for all parameters θ, it is optionally determined whether these parameters θ should be ignored in this training round. For this purpose, for example, the probability that the parameters θ of this layer are ignored is specified separately for each layer (S1, S2, ..., S6). For example, this probability can be 50% for the first layer (S1) and is reduced by 10% for each subsequent layer.

[0127] Then, with the help of these prescribed corresponding probabilities, it can be determined for each of the parameters θ whether the parameter is to be ignored.

[0128] Now (1155), for each pair in batch B , optionally determine the corresponding input signal x i For each corresponding input signal x that should be enhanced i , preferably randomly selecting an enhancement function and applying the enhancement function to the input signal x i Then, the enhanced input signal x i Replace the original input signal x i If the input signal x i If φ is an image signal, the enhancement function can be given, for example, by rotating it by a predeterminable angle.

[0129] Then (1160), for each pair in batch B, , select the corresponding (and if necessary enhanced) input signal x i And the input signal is supplied to the neural network 60. In this case, the parameters θ to be ignored of the neural network 60 are deactivated during the determination of the corresponding output signal, for example by temporarily setting them to the value zero. The corresponding output signal of the neural network 60 Assigned to the corresponding pair According to the output signal and The corresponding output signal y i As the desired output signal y T , respectively determine the cost function .

[0130] Then (1165), for all pairs of batch B, , together determine the complete cost function , and for each of the parameters θ to be ignored, for example by means of backpropagation, the corresponding component of the gradient g is determined. For each of the parameters θ to be ignored, the corresponding component of the gradient g is set to zero.

[0131] Now, a check (1170) is made as to whether the batch size bs was greater than the number of pairs present in the second set N during the check in step 1000. The number of.

[0132] If it is found that the batch size bs is not larger than the number of pairs present in the second set N The number of, then (1180) will batch all the pairs of B is added to the first set G and removed from the second set N. It is now checked (1185) whether the second set N is empty. If the second set N is empty, a new epoch begins (1186). For this purpose, the first set G is initialized again to an empty set and the second set N is reinitialized by assigning to it again all pairs of the training data set X. , and branch to step (1200). If the second set N is not empty, branch directly to step (1200).

[0133] If it is found that the batch size bs is larger than the number of pairs present in the second set N The first set G is reinitialized (1190) by assigning to it all pairs of batch B. , the second set N is reinitialized by assigning all pairs of the training data set X to it again And then the pairs that also exist in batch B Remove. Then, a new period begins and branches to step (1200). This ends this part of the method.

[0134] Fig.13 Another exemplary method for determining the gradient g in step 1100 is illustrated in a flowchart. First, the parameters of the method are initialized (1111). Hereinafter, W is used to represent the mathematical space of the parameters θ. That is, if the parameters θ include np individual parameters, then the space W is an np-dimensional space, e.g. The iteration counter n is initialized to the value n = 0, and then the first parameter m1 is set to (i.e. set to an np-dimensional vector), the second parameter is set to (that is, set to dimensional matrix).

[0135] Then (1121), randomly select and enhance pairs from the training dataset X if necessary. This can be achieved, for example, by making the training data set X Each input signal x i Identify possible enhancements Number of , and give each pair Assigning positional parameters

[0136] . (2)

[0137] If you then draw random numbers uniformly distributed , then we can choose to satisfy the inequality chain

[0138] (3)

[0139] The position parameter Then, the index i of , input parameter x i Enhancement From possible enhancements is randomly selected from the set and applied to the input parameter xi, that is, the selected quilt Alternative.

[0140] Input signal x i is sent to the neural network 60. According to the corresponding output signal and The output signal y i As the desired output signal y T , determine the corresponding cost function For the parameter θ, for example, the gradient d associated with it is determined by back propagation, that is, .

[0141] Then (1131), the iteration counter n, the first parameter m1 and the second parameter m2 are updated as follows:

[0142]

[0143] Then (1141), the components of the covariance matrix C are Provided as

[0144] . (7)

[0145] Based on this, the scalar product S is calculated using the (vector-valued) first parameter m1, that is,

[0146] . (8) Substitution.

[0147] It is easy to understand that in order to determine the scalar product S accurately enough using equation (8), it is not the covariance matrix C or the inverse matrix C -1All entries of must be present at the same time. It is more memory efficient to determine the entries of the covariance matrix C during the analysis of equation (8) and then .

[0148] Then check (1151) whether the scalar product S satisfies the following inequality:

[0149] , (9)

[0150] Here, λ is a predeterminable threshold value which corresponds to the confidence level.

[0151] If this inequality is satisfied, the current value of the first variable m1 is adopted as the estimated gradient g and the process branches back to step ( 1200 ).

[0152] If the inequality is not satisfied, the process can branch back to step (1121). Alternatively, it is also possible to check (1171) whether the iteration counter n has reached a predeterminable maximum iteration value n. max If this is not the case, branch back to step (1121), otherwise adopt (1181) the zero vector As the estimated gradient g, and branch back to step (1200). This part of the method ends.

[0153] This method achieves that: m1 corresponds to the extracted pair The arithmetic mean of the determined gradients d, and m2 corresponds to the extracted pairs The matrix product of the determined gradient d The arithmetic mean of .

[0154] Fig.14 An embodiment of a method for scaling the gradient g in step (1200) is shown. To represent each component of the gradient g, represents the layer with the corresponding parameter θ, and Indicates that the corresponding parameter θ is The number of layers within a layer. If the neural network is Fig.10 The structure illustrated in the figure is used to process The corresponding feature maps in the layers The multi-dimensional input data x is numbered Advantageously, by The corresponding parameter θ in is given by the position of the feature it is associated with.

[0155] Now (1220), for each component of the gradient g , determine the scaling factor For example, the scaling factor Through the The feature map of the layer The size of the receptive field rF of the corresponding feature is given. Alternatively, the scaling factor You can also use The resolution of each layer is given by the ratio of the number of features to the input layer.

[0156] Then (1220), using the scaling factor To each component of the gradient g To scale,

[0157] . (10)

[0158] If the scaling factor Given the size of the receptive field rF, overfitting of the parameters θ can be avoided particularly effectively. Given by the ratio of the resolution, this is a particularly efficient approximation of the size of the receptive field rF.

[0159] Fig.15 An embodiment of the method implemented by the scaling layer S4 is illustrated.

[0160] The scaling layer S4 is configured to realize a projection of an input signal x attached to the input of the scaling layer S4 onto a sphere with a radius ρ and a center c. This is done by the first norm and the second norm The first norm measures the distance of the center c from the output signal y attached to the output end of the scaling layer S4, and the second norm measures the distance of the input signal x attached to the input end of the scaling layer S4 from the output signal y attached to the output end of the scaling layer S4. In other words, the following equation solves the output signal y attached to the output end of the scaling layer S4:

[0161] . (11)

[0162] Fig.15 a) illustrates a particularly efficient first implementation for the case where the first norm N1 is the same as the second norm N2. To express.

[0163] First (2000), an input signal x, a center parameter c and a radius parameter ρ are provided at the input of the scaling layer S4.

[0164] Then (2100), the output signal y attached to the output terminal of the scaling layer S4 is determined as

[0165] (12)

[0166] This concludes this part of the method.

[0167] Fig.15 b) and 15c) illustrate embodiments of particularly advantageously selected combinations of the first norm N1 and the second norm N2.

[0168] Fig.15 b) Schematic diagram of the maximum norm in the regulation (12) to be satisfied To give the first norm And through the 2-norm To give the second norm This norm combination can be calculated particularly efficiently.

[0169] First (3000), similar to step (2000), an input signal x, a center parameter c and a radius parameter ρ are provided to the input end of the scaling layer S4.

[0170] Then (3100), the component y of the output signal y attached to the output of the scaling layer S4 is i Determined as

[0171] (13)

[0172] Here, i represents these components.

[0173] This method is particularly computationally efficient. This concludes this part of the method.

[0174] Fig.15 c) A diagram illustrating the 1-norm in the regulation (12) to be satisfied. To give the first norm And through the 2-norm To give the second norm This norm combination results in that in the input signal x attached to the input of the scaling layer S4 as many small components as possible are set to the value zero.

[0175] First (4000), similar to step (2000), an input signal x, a center parameter c and a radius parameter ρ are provided at the input of the scaling layer S4.

[0176] Then (4100), the symbol parameter Determined as

[0177] (14)

[0178] Furthermore, the component x of the input signal x attached to the input of the scaling layer S4 i quilt

[0179] (15)

[0180] The auxiliary parameter γ is initialized to the value zero.

[0181] Then (4200), the set N is determined as , and the distance measure is .

[0182] Then (4300), check whether the inequality is satisfied

[0183] (16).

[0184] If this is the case (4400), the auxiliary parameter γ is

[0185] , (17)

[0186] Replace and branch back to step (4200).

[0187] If (4500) inequality (16) is not satisfied, the component y of the output signal y attached to the output of the scaling layer S4 is i Determined as

[0188] (18)

[0189] In this case, the symbol Usually means

[0190] (19).

[0191] This ends this part of the method. The method corresponds to Newton's method and is computationally particularly efficient, in particular when a plurality of components of the input signal x present at the input of the scaling layer S4 are important.

[0192] Fig.16 An embodiment of a method for operating a neural network 60 is illustrated. First (5000), a neural network is trained using one of the described methods. Then (5100), the control system 40 is operated as described using the neural network 60 trained in this way. The method ends here.

[0193] It is easy to understand that the neural network is not limited to feedforward neural networks (English: "feedforward neural network"), and the present invention can be applied in the same way to any type of neural network, in particular recurrent networks, convolutional networks (English: "convolutional neural network"), autoencoders, Boltzmann machines, perceptrons or capsule networks (English: "Capsule Neural Network").

[0194] The term "computer" includes any device for executing predefinable calculation algorithms. These calculation algorithms can exist in the form of software, in the form of hardware, or in the form of a combination of software and hardware.

[0195] Furthermore, it is easy to understand that the methods can be implemented not only completely in software as described, but also in hardware or in a hybrid form of software and hardware.

Claims

1. A computer-implemented method for training a neural network (60) which is designed for classifying images, wherein an input signal (x) and an associated desired output signal (y) are supplied. T ), adapting the parameters (θ) of the neural network (60) according to the output signal (y) of the neural network (60), wherein the adaptation of these parameters (θ) is carried out according to the determined gradient (g), The components (g) of the determined gradient (g) i，l ) are scaled according to which layer (S1,...,S5) of the neural network the parameters (θ) corresponding to these components belong to, It is characterized in that The scaling is also based on the corresponding component (g i，l ) belongs to which feature of the feature map, wherein the scaling is performed according to the size of the receptive field (rF) of the feature.

2. The method according to claim 1, wherein the receptive field (rF) is realized according to the resolution of the layer (S1, ..., S5).

3. A method according to claim 2, wherein the scaling is achieved according to the quotient of the resolution of the layer (S1, ..., S5) and the resolution of the input layer (S1) of the neural network (60).

4. A training system (140) configured to carry out the method according to any one of claims 1 to 3.

5. Application of the method according to any one of claims 1 to 3 for training a neural network (60) for classifying an input signal (x) determined as a function of an output signal (S) of a sensor (30).

6. Application of the method according to any one of claims 1 to 3 for training a neural network (60) for providing a control signal (A) for controlling an actuator (10) based on a determined output signal (y) of the neural network (60).

7. The use according to claim 6, wherein an actuator (10) is actuated as a function of the actuation signal (A). 8 . Computer program product comprising a computer program which is designed to carry out the method according to claim 1 .

9. A machine-readable storage medium (46, 146) on which a computer program is stored, which is designed to carry out the method according to any one of claims 1 to 3.

10. Method for using a neural network (60), wherein in a first phase the neural network (60) is trained using a method according to any one of claims 1 to 3 and then the neural network is used according to any one of claims 5 to 7.

Images