More robust processing of measurement data by means of neural networks

Abstract

The invention relates to a method for processing measurement data in a neural network having a plurality of layers of neurons, the outputs of the neurons of at least one layer being guided as inputs into the neurons of at least one subsequent layer, comprising the following steps: determining the number of the neurons in accordance with parameters associated with the neurons; processing the input delivered to each neuron into a job result of the neuron; selecting a set of neurons; determining a target distribution for the job results of the neurons of the group; providing an inverse cumulative density function of target distribution; mapping the job results to unit values in an interval [0, 1] by means of a preset unit function, so that the unit value belonging to each job result occupies the same ranking on the list of all the unit values and the corresponding job result on the list of all the job results; calculating a normalized job result from the unit values by means of an inverse cumulative density function; and further processing the normalized operation result in the neural network instead of the original operation result into the output of the neural network.

Description

Technical Field

[0001] The invention relates to the processing of measurement data by means of neural networks, for example for monitoring the surroundings during at least partially automated driving of a vehicle. Background Art

[0002] The at least partially automated guidance of vehicles and / or robots on company premises or in public transport requires continuous monitoring of the surroundings of the vehicle and / or robot. In this case, measurement data are generated which need to be evaluated with respect to a predetermined task. For example, it can be assessed from camera images or similar measurement data: what types of objects are present in the surroundings of the vehicle and / or robot. Neural networks are often used for this task, since they have the potential to generalize well to situations that were not encountered during training.

[0003] The neural network for processing the measured data is usually divided into a plurality of layers. The outputs of the neurons of at least one layer are then respectively directed as inputs to the neurons of at least one subsequent layer. The input supplied as a whole to the neural network is thus passed continuously through the entire neural network in the forward direction until the input is mapped as a whole in the last layer of the neural network to the final output of the neural network. Conversely, the feedback obtained during training, typically in the form of the value of a cost function (loss function), is back-propagated through the neural network in the reverse direction. In this way, the gradient is determined, along which the parameters characterizing the behavior of the neural network should change in order to improve the value of the cost function speculatively.

[0004] In order to stabilize the training, a normalization technique is usually used, which brings the results of a certain group of operations in the neural network to a uniform order of magnitude. This counteracts in particular the tendency of the gradient to vanish or explode in magnitude at certain locations in the neural network during backpropagation, which can seriously hinder the training. Summary of the invention

[0005] Within the scope of the invention, a method for processing measurement data in a neural network is provided. The network has multiple layers of neurons, wherein the outputs of neurons of at least one layer are respectively fed as inputs to neurons of at least one subsequent layer.

[0006] The measurement data may be, for example, in particular camera images, radar images, lidar images, thermal images and / or ultrasound images. For example, radar data and lidar data may also be in the form of a point cloud. Furthermore, the measurement data may also include, for example, a time series of one or more measurement variables.

[0007] For example, the neural network can be designed as a classifier, which processes the input of the neural network into a classification score for one or more classes of a predetermined classification. In particular, the neural network can be designed as an image classifier, for example.

[0008] Within the scope of the method, the input supplied to each neuron is processed according to the parameters associated with the neuron into an operating result of the neuron. The operating result can already be the final output of the neuron, but can also be an intermediate result which is further processed, for example, in one or more further steps into a final output associated with the neuron.

[0009] Now a group of neurons is selected, the results of which are to be normalized subsequently. This group can be selected according to any criteria relevant for the respective application and can include, for example, all neurons of a layer or a part thereof. For example, neurons can be accommodated in this group whose contribution has been shown to be decision-relevant with respect to specific aspects of the application when analyzing the feature maps created by the layer and / or by other significance analyses of the neural network output.

[0010] A target distribution is determined for the results of the operation of the neurons of the group. This is the distribution that the results of the operation of the neurons should have after normalization. In addition, an inverse cumulative density function of the target distribution is provided. Such an inverse cumulative density function exists for each distribution function and associates the following minimum independent variable of the distribution function with each threshold between 0 and 1, in the case of the minimum independent variable, the distribution function exceeds the threshold. Therefore, the inverse cumulative density function is also called a quantile function.

[0011] Now, the operation results are mapped to the unit values ​​in the interval [0, 1] by means of a preset unit function, so that the unit value belonging to each operation result occupies the same ranking on the list of all unit values ​​as the corresponding operation result on the list of all operation results. That is to say, the operation results are mapped to the ranking of the operation results in the interval [0, 1], which includes each other. However, the numerical spacing between the operation results can be compressed and / or stretched when mapped to the interval [0, 1]. If, for example, five operation results have numerical values ​​of 10000, 1000, 5, 3 and 2, the operation results can be mapped to unit values ​​of 0.9, 0.89, 0.85, 0.4 and 0.1, for example. Thus, the ranking of the unit value is the same as the original operation result. However, the difference between 10000 and 1000 in the operation result is compressed to a difference of 0.01 in the unit value, and even the difference between 1000 and 5 in the operation result is just compressed to a difference of 0.04 in the unit value. In contrast, what can be fully understood as stretching is that the difference between 5 and 3 in the job results is mapped to a difference of 0.45 in the unit value, and the difference between 3 and 2 in the job results is mapped to a difference of 0.3 in the unit value.

[0012] The normalized operation result is calculated from the unit value by means of the inverse cumulative density function, and the normalized operation result is further processed in the neural network to replace the original operation result and become the output of the neural network.

[0013] It has been recognized that by appropriately selecting the target distribution, the processing of the measured data in the neural network can be made surprisingly robust to domain shifts relative to the domain of the training data. Each normalization evens out certain differences and changes in the input of the neural network, since these differences and changes no longer appear in the normalized results. This means that the processing in the neural network is invariant to certain changes in the input. The presetting of the target distribution of the work results now has the effect that the normalized work results still belong to the target distribution, at least within a limited range, even when the input supplied to the neural network no longer belongs to the domain and / or distribution of the training data used for training. Such a domain shift of the input relative to the training data usually causes the distribution of the work results obtained to also shift. The normalization compensates for this in such a way that the further processing of the domain-shifted measured data is carried out in the same way as the further processing of the measured data from the domain of the training data. Therefore, after normalization, the neural network no longer notices at all that it has obtained domain-shifted measured data.

[0014] Here, the main effect of mapping to unit values ​​is to place the following job results, whose values ​​can span multiple orders of magnitude, within the definition of the cumulative density function in the interval [0, 1]. It is inevitable that part of the original information contained in the job results is sacrificed. However, it has been recognized that the most important information in the job results is the ranking of the job results with respect to each other, and in contrast, it is unimportant which absolute values ​​of the job results are used to express the ranking. Therefore, the ranking is obtained exactly when mapping to the interval [0, 1].

[0015] Invariance with respect to domain shifts can lead to significant savings in training the neural network 1. For example, training examples are used in the field of at least partially automated driving, such as images or video sequences recorded during test drives. In order to train the neural network in a monitored manner, the training examples must be labeled with the target output that the neural network as a whole should output. The labeling is a manual and therefore expensive process. The training is then coupled to a domain of measurement data that can provide a measurement structure for detecting the training examples, such as an arrangement consisting of cameras and / or other sensors. If slight changes are now made to the measurement structure afterwards, for example because a camera on the vehicle is placed in a different position and the perspective observed by it changes, the measurement data recorded from that time onwards belong to a domain that is domain shifted with respect to the original training examples. The method proposed here helps: the domain shift is mostly evened out and retraining or further training with the aid of a changed measurement structure can be saved, which requires the introduction and labeling of a large number of new training examples.

[0016] In a particularly advantageous embodiment, the work results are mapped to quantiles, which are used as unit values. The quantiles indicate the numerical proportion of other work results that is smaller than the work result currently considered. The unit values ​​are therefore very suitable as independent variables for an inverse cumulative density function, which is expressed as quantiles.

[0017] The calculation of the quantile can, for example, in particular comprise: for all other job results, a score is determined, which indicates to what extent the job result is greater or smaller than the job result currently under consideration. These scores can then be averaged over the other job results. In this case, in particular, for example, a score of 1 can then be assigned to another job result that is greater than the job result currently under consideration. And a score of 0 can be assigned to another job result that is smaller than the job result currently under consideration. The assignment can optionally be softened, for example, in order to make the calculation of the quantile differentiable, wherein the assignment corresponds to a Heaviside step function applied to the pairwise differences between the job results.

[0018] In the example mentioned of the work results with the values ​​10000, 1000, 5, 3 and 2, work result 5 is the work result currently under consideration. The remaining work results 10000, 1000, 3 and 2 are included in the calculation of the quantile representing the work result 5. In this case, work results 10000 and 1000 each receive the score 1, because they are greater than 5. Work results 3 and 2 each receive the score 0, because they are less than 5. This results in a total of two score points, which can be averaged over the four work results. Therefore, the quantile of work result 5 is 0.5.

[0019] Particularly advantageously, the difference between another job result and the currently considered job result can be processed as a fraction of the other job result by using a Sigmoid function. This function is 1 for positive arguments of large absolute value and 0 for negative arguments of large absolute value. In the transition region, the function grows continuously from 0 to 1. Applying the Sigmoid function to the difference between another job result and the currently considered job result is thus a differentiable approximation that replicates the application of the Heaviside step function to the job result.

[0020] In general, it is therefore advantageous to choose a unit function that is a differentiable approximation of a discontinuous function. In particular, a differentiable unit function is required in the case of back-propagation of the gradients from the cost function to the neuron parameters. Otherwise the gradients cannot be propagated beyond the normalization. This is roughly equivalent to the fact that a barrage in a river is an insurmountable obstacle for fish migrating upstream. To alleviate this problem, fish ladders are built across the barrage.

[0021] The two types of identity functions can also be combined with each other. So for example a discontinuous but exact identity function can be used in the forward direction and a differentiable approximation in the reverse direction.

[0022] In a further particularly advantageous embodiment, the differentiable approximation is parameterized using a temperature parameter.

[0023] The temperature parameter can be used to

[0024] · The accuracy of the approximation is

[0025] · The approximate saturation speed between the other

[0026] In addition to the speed of the approximation saturation, the number of derivatives approximated also changes. The number of derivatives is also called the "differentiability class" or, in English, the "smoothness" of the function. Thus, for example, when training a neural network, the temperature parameter can be trained together as a hyperparameter.

[0027] In particular, for example, during the training of a neural network, the temperature parameters can be varied according to a preset annealing schedule. For example, in this way, convergence can be facilitated by using a softer, less accurate and therefore more differentiable approximation at the beginning of the training.

[0028] In another particularly advantageous embodiment, the operation result includes an activation obtained by summing the inputs of the neuron weighted according to the parameter. Then, further processing of the normalized operation result includes applying a predefined nonlinear activation function to the normalized activation.

[0029] Depending on the number of inputs a neuron gets, activations can develop fast dynamics, and their values ​​can span multiple orders of magnitude within the same layer. This is achieved through normalization: all neurons of the layer can make their contribution to the output of said layer, and a slightly smaller number of neurons do not monopolize the discussion about said output.

[0030] In another advantageous embodiment, the unit value is limited to a preset interval before being fed to the inverse cumulative density function. For example, it can be preset that the unit value is kept at least a certain amount (approximately 10 -3 In this way, it is possible to avoid the normalized operation results from presenting non-finite values.

[0031] The target distribution can be, for example, a normal distribution in particular. Most normalized activations are then concentrated around the mean, but there are certainly upward and downward outliers. Alternatively, the target distribution can also be, for example, another member of the exponential family or a Student's t distribution.

[0032] As already mentioned above, the goal ultimately sought with the aid of normalization is to make the evaluation of the measurement data more robust and more reliable with respect to the specific application at hand. Therefore, in a further particularly advantageous embodiment, a control signal is determined from the output of the neural network. The vehicle, the driver assistance system, the robot, the system for quality control, the system for monitoring an area and / or the system for medical imaging are controlled with the aid of the control signal. In this way, the probability is increased that the reaction executed by the respectively controlled engineering system according to the control signal is adapted to the situation reflected in the measurement data. This applies in particular when the input of the neural network comes from a domain that is displaced relative to the domain of the training data.

[0033] The method can in particular be implemented completely or partially by computer. Therefore, the present invention also relates to a computer program with machine-readable instructions, which, when executed on one or more computers and / or computing instances, cause the computers and / or computer instances to execute the method described herein. In this sense, control devices for vehicles and embedded systems for engineering equipment can also be regarded as computers, which control devices for vehicles and embedded systems for engineering equipment can also execute machine-readable instructions. The computing instance can be, for example, a virtual machine, a container or a serverless execution environment, which can be provided in particular in the cloud.

[0034] The invention also relates to a machine-readable data carrier and / or a download product with a computer program. A download product is a digital product that can be transmitted via a data network, ie downloaded by a data network user, which digital product can be sold for immediate downloading in an online store, for example.

[0035] Furthermore, one or more computers and / or computing instances can be equipped with the computer program, the machine-readable data carrier or the download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other measures for improving the present invention are described in more detail below together with the description of preferred embodiments of the present invention according to the drawings.

[0037] Figure 1 An exemplary embodiment of a method 100 for processing measurement data 9 in a neural network 1 is shown;

[0038] Figure 2 shows an illustration of the robustness to the domain shift of input 1a with respect to training example 1a*, said robustness being induced by method 100;

[0039] Figure 3 A comparison is shown of the effects of method 100 and conventional normalization methods on inputs 1a belonging to different distributions. DETAILED DESCRIPTION

[0040] Figure 1 1 is a schematic flow chart of an exemplary embodiment of a method 100 for processing measurement data 9 in a neural network 1. The neural network 1 has multiple layers 3 of neurons 2. In this case, the outputs of neurons 2 of at least one layer 3 are respectively fed as inputs to neurons 2 of at least one subsequent layer 3.

[0041] In step 110 , the input supplied to each neuron 2 is processed into a working result 2 b of the neuron 2 according to the parameters 2 a associated with the neuron 2 .

[0042] According to block 111 , the job result 2 b may include, for example, in particular an activation obtained by summing the inputs of the neuron 2 weighted according to the parameter 2 a .

[0043] In step 120 , a group 2* of neurons 2 is selected, whose operating results 2 b are then to be normalized.

[0044] In step 130, a target distribution 4 is determined for the work results 2b of the neurons 2 of the group 2*. The target distribution 4 is the distribution to which the normalized work results 2b* should belong.

[0045] According to block 131 , in particular, for example, a normal distribution or another member of the exponential family, or also a Student's t distribution may be selected as target distribution 4 .

[0046] In step 140 , an inverse cumulative density function 4 # of the target distribution 4 is provided.

[0047] In step 150, the job results 2b are mapped to unit values ​​6 in the interval [0, 1] by means of a preset unit function 5. The unit value 6 belonging to each job result 2b then occupies the same rank in the list of all unit values ​​6 as the corresponding job result 2b in the list of all job results 2b.

[0048] In this case, according to block 151, the job result 2b can be mapped, for example, to a quantile as unit value 6. The quantile indicates the numerical proportion of the other job results 2b that is smaller than the respectively currently considered job result 2b#.

[0049] According to block 151a, a score 2b' can be calculated for each of the other job results 2b. The score 2b' indicates to what extent the corresponding job result 2b is greater or less than the currently considered job result 2b#. The score 2b' can then be averaged over the other job results 2b according to block 151b.

[0050] In particular, for example, according to block 151c, a score of 1 may be assigned to another job result 2b that is greater than the currently considered job result 2b#, while according to block 151d, a score of 0 may be assigned to another job result 2b that is less than the currently considered job result 2b#.

[0051] According to block 151e, the difference between another operation result 2b and the currently observed operation result 2b# can be processed as a score 2b' of the other operation result 2b by applying a Sigmoid function, thereby softening the discontinuous step function in a differentiable manner, which only distinguishes whether the other operation result 2b is greater or less than the currently considered operation result 2b#.

[0052] According to block 152 , an identity function 5 may thus generally be selected, which is a differentiable approximation of a discontinuous function.

[0053] The differentiable approximation can be parameterized, for example, in particular according to block 152a, by means of a temperature parameter, via which the differentiable approximation can be parameterized in

[0054] · The accuracy of the approximation is

[0055] · The approximate saturation speed between the other

[0056] The compromise is adjusted.

[0057] The temperature parameter can be varied, for example, in particular according to block 152b, according to a predefined annealing schedule during the training of the neural network 1. As described above, this can facilitate the convergence of the training, for example, right at the beginning of the training.

[0058] In step 160 , a normalized operating result 2 b * is calculated from the unit value 6 with the aid of the inverse cumulative density function 4 #.

[0059] According to block 161, the unit value 6 can be limited to a preset interval (called "clamping" in English) before being fed to the inverse cumulative density function 4#. This can, for example, prevent the inverse cumulative density function 4# from outputting a non-finite value exactly at the boundary of its defined range when the input is an independent variable of 0 or 1.

[0060] In step 170, the normalized job result 2b* is further processed in the neural network 1 to replace the original job result 2b as the output 1b of the neural network 1. As mentioned above, this means that if multiple inputs 1a of the neural network become the same normalized job result 2b* through normalization, then the inputs 1a can also be processed into the same output 1b of the neural network 1 in the end. Therefore, by normalizing the job results, the neural network 1 can, for example, learn the invariance of the displacement of the domain of the input 1a relative to the domain from which the training examples 1a* used for training come.

[0061] exist Figure 1 In the example shown in FIG. 1 , in step 180, an actuation signal 8 is determined from the output 1 b of the neural network 1. In step 190, the vehicle 50, the driver assistance system 51, the robot 60, the system 70 for quality control, the system 80 for monitoring an area and / or the system 90 for medical imaging are actuated with the aid of the actuation signal 8.

[0062] Figure 2 Illustration: How normalizing job result 2b to normalized job result 2b* can make processing in neural network 1 more robust with respect to the fact that neural network 1 inputs are outside the domain and / or distribution D of training examples 1a*.

[0063] The processing of the input 1a outside the domain D is first directed to the work result 2b, which is different from the work result 2b obtained for the training example 1a*. By converting the work result 2b into a unit value 6 with the help of the unit function 5 in step 150 of the method 100, the difference has been at least leveled to the extent that the unit values ​​6 are respectively in the same interval [0, 1]. If the unit value 6 is now converted into a normalized work result 2b* with the help of the inverse cumulative density function 4# in step 160, the normalized work results 2b* all belong to the previously determined target distribution 4, which is a normal distribution in this case. This means that the normalized work results 2b* generated from the training example 1a* and the normalized work results 2b* generated from the new input 1a are sorted according to the ranking of the corresponding unit value 6 on the horizontal axis with the help of the target distribution function 4. All these normalized work results 2b* can therefore now be reasonable samples that can be extracted from the target distribution 4. Therefore, for further processing in the neural network 1, it no longer makes a significant difference whether the normalized job result 2b* originally comes from a training example 1a* in domain D or from a new input 1a shifted relative to said domain D. Therefore, the neural network 1 is invariant to said domain shift.

[0064] Figure 3 The effect of the method 100 proposed here for normalizing the job result 2b on the processing of inputs 1a belonging to different distributions is shown:

[0065] In row A, input 1a is normally distributed.

[0066] In row B, enter 1a Uniform Distribution.

[0067] In row C, input 1a is distributed according to the Student's t distribution.

[0068] In row D, enter 1a Bimodal Distribution.

[0069] In row E, enter 1a Multimodal distribution.

[0070] Draw separately in each of rows A to E:

[0071] In column a, the normalized operation result 2b* is plotted on top of the original operation result 2b using the conventional normalization method;

[0072] In column b, the job result 2b* normalized by means of the method 100 proposed here is plotted on top of the original job result 2b; and

[0073] In column c, the frequency distribution of conventionally normalized operating results 2b* (curve a) and operating results 2b* normalized by means of the method 100 proposed here is plotted, wherein the probability density f is respectively plotted over the values ​​of the operating results 2b*.

[0074] exist Figure 3 In the example shown in , the method 100 proposed herein uses a normal distribution as the target distribution 4. A conventional normalization method is group normalization using a linear transformation.

[0075] Thus, for conventional methods, the correlation between the normalized job result 2b* and the original job result 2b is always linear. For the method 100 proposed here, this is only true if the input 1a is normally distributed, which here corresponds to the target distribution 4. In all other cases, the mapping is monotonic, but not linear.

[0076] The frequency distribution of the normalized operation result 2b* according to the method 100 proposed here is almost completely independent of the distribution of the input 1a and is always substantially normally distributed. The conventionally normalized operation result 2b* is normally distributed only when the input 1a is also normally distributed. Otherwise, the suppression of multiple modes or even strong tails cannot be successful or is very imperfect.

[0077] This shows that the method 100 presented here is particularly advantageous for the operation of smaller neural networks 1 with relatively few features. In larger neural networks 1, the law of large numbers causes the distribution of the work results 2a to automatically approximate a normal distribution. In smaller neural networks 1, i.e., in particular in neural networks 1 with fewer than approximately 100 neurons 2, the probability that the work results 2a are already normally distributed becomes significantly smaller.

Claims

1. A method (100) for processing measured data (9) in a neural network (1), the neural network having multiple layers (3) of neurons (2), wherein the outputs of the neurons (2) of at least one layer (3) are respectively fed as inputs to the neurons (2) of at least one subsequent layer (3), the method comprising the following steps: Processing (110) the input delivered to each neuron (2) into an operation result (2b) of the neuron (2) according to the parameters (2a) associated with the neuron (2); · Select (120) the group (2*) of neurons (2); determining (130) a target distribution (4) for the work results (2b) of the neurons (2) of the group (2*); Providing (140) an inverse cumulative density function (4#) of the target distribution (4); Mapping (150) the operation results (2b) to unit values ​​(6) in the interval [0, 1] by means of a preset unit function (5), so that the unit value (6) belonging to each operation result (2b) occupies the same rank in the list of all unit values ​​(6) as the corresponding operation result (2b) in the list of all operation results (2b); Calculating (160) a normalized operation result (2b*) from the unit value (6) by means of the inverse cumulative density function (4#); The normalized operation result (2b*) is further processed (170) in the neural network (1) to replace the original operation result (2b) and become the output (1b) of the neural network (1).

2. The method (100) according to claim 1, wherein the job result (2b) is mapped (151) to a quantile as a unit value (6), which indicates: which numerical proportion of other job results (2b) is smaller than the job result (2b#) currently considered.

3. The method (100) according to claim 2, wherein for all other results of the operation (2b), respectively, a score (2b') is calculated (151a), which indicates to what extent the result of the operation (2b) is greater or less than the result of the operation currently under consideration (2b#), and · Averaging (151b) the score (2b') over the other work results (2b).

4. The method (100) according to claim 3, wherein Assign (151c) a score of 1 to another job result (2b) that is greater than the job result (2b#) currently being considered, and Assign (151d) a score of 0 to another job result (2b) that is smaller than the currently considered job result (2b#).

5. The method (100) according to any one of claims 3 to 4, wherein the difference between another job result (2b) and the currently considered job result (2b#) is processed (151e) into a score (2b') of the other job result (2b) by using a Sigmoid function.

6. The method (100) according to any one of claims 1 to 5, wherein an identity function (5) is selected (152), the identity function (5) being a differentiable approximation of a discontinuous function.

7. The method (100) according to claim 6, wherein the differentiable approximation is parameterized (152a) by means of a temperature parameter, via which the temperature parameter can be used to determine the temperature of the · The accuracy of the approximation is · A compromise between the near saturation speed and the other is adjusted.

8. The method (100) according to claim 7, wherein the temperature parameter is varied (152b) during the training of the neural network (1) according to a preset annealing schedule.

9. The method (100) according to any one of claims 1 to 8, wherein The operation result (2b) comprises (111) an activation obtained by summing the inputs of the neuron (2) weighted according to the parameters (2a), and The further processing of the normalized operation results includes: The preset nonlinear activation function (7) is applied to the (171) normalized activation (2b*).

10. The method (100) according to any one of claims 1 to 9, wherein the unit value (6) is limited (161) to a preset interval before being fed to the inverse cumulative density function (4#).

11. The method (100) according to any one of claims 1 to 10, wherein a normal distribution, another member of the exponential family or a Student's t distribution is selected (131) as the target distribution (4).

12. The method (100) according to any one of claims 1 to 11, wherein · determining (180) a control signal (8) from the output (1b) of the neural network (1), and A vehicle (50), a driver assistance system (51), a robot (60), a system for quality control (70), a system for monitoring an area (80) and / or a system for medical imaging (90) are controlled (190) using the control signal (8).

13. A computer program comprising machine-readable instructions which, when executed on one or more computers and / or computing instances, cause the computers and / or computing instances to perform the method (100) according to any one of claims 1 to 12.

14. A machine-readable data carrier having a computer program according to claim 13.

15. One or more computers and / or computing instances equipped with a computer program according to claim 13 and / or equipped with a machine-readable data carrier according to claim 14.

Images