Method and system for quantifying uncertainty of output data of a machine learning system and method for training a machine learning system

Abstract

The present invention relates to a method for training a machine learning system (1) for quantifying the uncertainty (U) of output data (Y'), wherein training data (X,Y) is provided, comprising training input data (X) and a training target value (Y). Using the training data (X,Y), parameters of the machine learning system (1) are adjusted such that the machine learning system (1), upon input of the training input data (X), generates output data (Y') similar to the training target value (Y) and generates reconstructed data (X';Y") representing a measure of the knowability of the training data (X,Y). The present invention also relates to a method for quantifying the uncertainty (U) of output data (Y') of the machine learning system (1), wherein the machine learning system (1) has been trained by the above method, wherein the machine learning system (1) generates output data (Y') from the input data (X), and the machine learning system (1) generates the reconstructed data (X';Y"). Using the metric (4) and the reconstructed data (X';Y") and the data (X;Y') corresponding to the reconstructed data (X';Y"), a difference value is generated, which is a measure of the known degree of the training data (X,Y) with respect to the input data (X), and quantifies the uncertainty (U) in the output data (Y') and is assigned to the output data (Y').

Description

Technical Field

[0001] The present invention relates to machine learning. In particular, the present invention relates to a method for training a machine learning system that quantifies uncertainty, a method for quantifying the uncertainty of output data of a machine learning system, a system for quantifying the uncertainty of output data of a machine learning system, and a vehicle comprising a system for quantifying the uncertainty of output data.

Background Art

[0002] In many fields, machine learning has become a valuable tool. Also, here, there is an increasing dependence on a reliable estimation of the uncertainty of the output of a machine learning system.

[0003] Thus, for example, in autonomous driving, a reliable estimation of uncertainty in classification output is an aspect related to safety. In this case, a situation that is uncertain for the system can be detected, and the control can be delegated to the driver or deceleration can be performed to avoid an accident.

[0004] In particular, in a convolutional neural network, it is a difficult problem to estimate uncertainty because in this case, classification errors are not correlated with actual uncertainty due to the non-linearity of the convolutional neural network. Also, for example, the addition of further layers with non-linear activation functions or by a nomination method, the more complex the convolutional neural network becomes, the stronger this effect becomes.

[0005] Methods for calculating known uncertainty are the Monte Carlo dropout method and the ensemble method. Both methods are based on the principle of statistical classification result dispersion modeling, which generates multiple results for the input signal. The greater the difference between these results, the greater the uncertainty. For this reason, in the case of the Monte Carlo dropout method, the weights of the network are randomly switched, while in the case of the ensemble method, the output is generated for each member of the ensemble. In this case, a large number of results are required for representative statistics. Due to the high computational complexity required for this, these methods are generally not suitable for implementation in real-time systems. Furthermore, the ensemble method requires a large memory cost.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] Therefore, an object of the present invention is to provide a method for quantifying the uncertainty of output data of a machine learning system with a moderate computational cost. This object is solved by the subject matter of the independent claims. Developments of the present invention will become apparent from the dependent claims and the following description.

Means for Solving the Problems

[0008] One aspect of the present invention relates to a method for training a machine learning system that quantifies the uncertainty of output data. In this case, training data including training input data and training target values is supplied. Here, the training target values are assigned to each of the training input data. In this case, the training target values are, for example, manually created for individual training input data and, for example, represent the classification of the training input data.

[0009] Using the training data, the machine learning system is trained. In other words, the parameters of the machine learning system are adjusted so that the machine learning system generates output data similar to the training target values when the training input data is input. The parameters include, for example, the weights between individual input values and neurons in the case of a neural network. Here, the training of the machine learning system is performed using a supervised learning method, which is well-known to many learning methods. In this way, for example, the error backpropagation method can be used to train a neural network. In this case, during training, the parameters of the machine learning system are adjusted so that the error between the output data and the training target values becomes as small as possible. Here, the error between the output data and the training target values is determined, for example, via the distance between the output data and the training target values and the corresponding metric for the output data. Note that overfitting is avoided here, which is achieved, for example, by taking into account the error between the output data generated from the test input data and the corresponding test target values. At that time, the test target values are assigned to the test input data, and the test input data and the test target values are not used for adjusting the parameters of the machine learning system.

[0010] Also, the parameters of the machine learning system are adjusted so that the machine learning system generates reconstructed data using the training data. Here, the reconstructed data represents a measure of the familiarity of the training data. When the training data is known for the input data to be processed using the trained learning system, the uncertainty of the output data of the machine learning system is small. On the other hand, when the training data is almost unknown or completely unknown for the input data to be processed using the trained learning system, the uncertainty of the output data of the machine learning system is large. That is, the uncertainty of the output data of the machine learning system can be quantified via the reconstructed data. At that time, the computational cost is appropriate, and the additional computational cost for calculating the reconstructed data is at most, for example, only equal to the computational cost for calculating the output data. Also, since the calculation of the reconstructed data is performed based on the training data that is already available, no further input is required here.

[0011] Here, the reconstructed data may be generated as a further output of the machine learning system that generates the output data. Alternatively, the machine learning system may include two subsystems, a first subsystem that generates the output data and a second subsystem that generates the reconstructed data. When the output data and the reconstructed data are generated by the two subsystems of the machine learning system, it is important that the training of both subsystems is based on the same training data, because only by this can the reconstructed data represent a measure of the familiarity of the same training data used for the generation of the output data.

[0012] In some embodiments, the reconstructed data is similar to the training input data. That is, the parameters of the machine learning system are adjusted so that the error between the reconstructed data and the training input data is minimized as much as possible. In other words, the input data is reconstructed via the reconstructed data.

[0013] In some embodiments, the reconstructed data is similar to the training target value, and the difference between the reconstructed data and the training target value is smaller than the difference between the output data and the training target value. That is, the parameters of the machine learning system are adjusted such that the error between the reconstructed data and the training target value is smaller than the error between the output data and the training target value. Since the parameters of the machine learning system have already been adjusted such that the error between the output data and the training target value is as small as possible while avoiding overfitting, this means that overfitting of the machine learning system occurs to generate the reconstructed data.

[0014] In some embodiments, the training input data is used as input to generate the reconstructed data. This is possible whether the reconstructed data is generated as a further output of the machine learning system that generates the output data or when the reconstructed data is generated by a second subsystem of the machine learning system. Alternatively or additionally, the output data may be used as input to generate the reconstructed data. For this, first, the output data needs to be available, so this is only possible when the reconstructed data is generated by a second subsystem of the machine learning system. Thus, in this case, the output of the first subsystem of the machine learning system is used as input for the second subsystem of the machine learning system.

[0015] In some embodiments, the machine learning system is a neural network. Here, the neural network is particularly suitable for the above method because the neural network can be easily adjusted. Here, the neural network is, in particular, a convolutional neural network. In this case, the non-linearity of the convolutional neural network does not pose a problem with respect to the usability of the above method.

[0016] On the other hand, the above method may be applied using other machine learning systems, such as decision tree learning, support vector machines, regression analysis, or Bayesian networks. Further, the above method can be applied in a multi-task classification system. In this case, the multi-task classification system includes, for example, an encoder and a number of decoders. Here, for each decoder, reconstructed data is generated, and accordingly, the parameters of the machine learning system are adjusted. Also, the machine learning system can be divided into a plurality of subsystems. In this case, while each subsystem has the functionality of the machine learning system of the present disclosure, the subsystems are different from each other, for example, in terms of the type of machine learning system or the selection of training data. In that case, the output data generated using the subsystems can be combined, thereby obtaining improved output.

[0017] In some embodiments, the input data includes sensor data, particularly, image data, radar data, and / or lidar data. As an example in the case of image data, the reconstructed data represents the reconstruction of the original image when the reconstructed data is similar to the training input data. In this case, the more accurately the original image can be reconstructed, the more the training data for this image becomes known, and the uncertainty of the output data becomes smaller.

[0018] Here, the input data may be sensor data of a vehicle. In this case, the output data is, for example, the classification of an object recorded in the sensor data. In this way, the above method can be applied to quantify the uncertainty of the output data for an autonomous driving system.

[0019] A further aspect of the present invention relates to a method for quantifying the uncertainty of the output data of a machine learning system, wherein the machine learning system is trained by the method for training the above machine learning system.

[0020] Therefore, the machine learning system may be, for example, a decision tree learning system, a support vector machine, a learning system based on regression analysis, a Bayesian network, a neural network or a convolutional neural network. Also, the machine learning system may be a multi-task classification system. Also, the machine learning system may include two subsystems.

[0021] The machine learning system generates output data from the input data. Here, the input data may be, for example, sensor data, particularly, image data, radar data and / or lidar data, and may be acquired, for example, by a vehicle. The output data may be a classification of the input data, and thus, for example, a classification of an object recorded in the sensor data.

[0022] Also, the machine learning system generates reconstruction data. Here, a difference value is generated from the reconstruction data and the data corresponding to the reconstruction data using a metric. In this case, this difference value is a measure of the degree of knowledge of the training data regarding the input data. Therefore, the difference value quantifies the uncertainty of the output data and is assigned to the output data.

[0023] Therefore, the quantification of uncertainty is achieved by using a trained machine learning system, and for this reason, the computational cost is at most only equal to, for example, the computational cost for calculating the output data.

[0024] The metric used here depends on what data the reconstruction data and the data corresponding to the reconstruction data are. Examples of such metrics are distance metrics or similarity measures, such as mean squared error, mean absolute error, structural similarity index or binary cross entropy.

[0025] Also, the quality of the uncertainty estimation can be determined. Known methods for this are recognized, for example, in Non-Patent Document 1 or Non-Patent Document 2.

[0026] In some embodiments, the data corresponding to the reconstructed data is the input data. In this case, the machine learning system is trained such that the reconstructed data obtained during training is similar to the training input data. Therefore, a difference value is generated from the reconstructed data and the input data using a metric. If there was training input data that was similar to the input data, the reconstructed data would also be similar to the input data, and thus the difference value would be small. In this case, since there was input data that was similar to the training input data, it is predicted that the reconstructed data would also have a small error, resulting in a small difference value with good output data, and thus small uncertainty is obtained in the output data. On the other hand, if there was no training input data that was similar to the input data, there would be a large difference between the reconstructed data and the input data, and thus the difference value would be large. This corresponds to a large uncertainty and is due to the input data not corresponding to any of the training input data.

[0027] In some embodiments, the data corresponding to the reconstructed data is output data generated from the input data. In this case, the machine learning system is trained such that the reconstructed data obtained during training is similar to the training target value, and the difference between the reconstructed data and the training target value is smaller than the difference between the output data generated from the training input data and the training target value. When the difference between the reconstructed data and the output data calculated using a metric is small, this indicates that there was training input data that was similar to the input data, and thus the uncertainty of the output data is also small. On the other hand, when the difference between the reconstructed data and the output data calculated using a metric is large, this indicates that there was no training input data that was similar to the input data, and thus, correspondingly, the uncertainty of the output data is large.

[0028] When the difference between the reconstructed data and the output data calculated using the metric is small, the reconstructed data can be used to improve the output of the machine learning system. For this purpose, for example, instead of the output data, reconstructed data that can represent a result more accurate than the output data based on the training of the machine learning system may be output. Alternatively, for example, instead of the output data, an average value of the output data and the reconstructed data that can represent a result more accurate than the output data may be output.

[0029] In some embodiments, input data is used to generate the reconstructed data. Alternatively or additionally, output data generated from the input data is used to generate the reconstructed data. Here, which data is used to generate the reconstructed data is based on which data was used to generate the reconstructed data during the training of the machine learning system.

[0030] In some embodiments, the metric is applied only to the reconstructed data and a part of the data corresponding to the reconstructed data. In this way, for example, the metric may be limited using, for example, a bounding box and applied only to a part of the image, particularly including the detected object. In this case, the uncertainty is analyzed separately in the classification of this object. As a further example, the metric may be applied to the entire low-resolution image if small details are considered to be of relatively little importance, for example.

[0031] In some embodiments, an uncertainty warning is issued for output data where the uncertainty exceeds a predetermined value. This uncertainty warning may be correspondingly considered by the system using the output data. In this way, for example, when the uncertainty of the output data of the machine learning system increases during autonomous driving where an object detected based on sensor data using the machine learning system is critically important, corresponding safety countermeasure measures may be initiated, for example, deceleration or transfer of vehicle control to the driver may be introduced.

[0032] In some embodiments, input data for further training the machine learning system is stored for output data where the uncertainty exceeds a predetermined value. Next, for the input data thus stored, for example, a corresponding target value is generated by classification by a user. In this case, since these new pairs consisting of the input data and the target value are used in training the machine learning system, the machine learning system can generate reliable output data from input data similar to the new input data.

[0033] A further aspect of the present invention relates to a system for quantifying the uncertainty of output data of a machine learning system. This system includes an input unit, a calculation unit, and an output unit. Here, the input unit is configured to receive input data and is configured, for example, as an interface with a sensor. The calculation unit is configured to execute a method for quantifying the uncertainty of the output data of the above machine learning system. Here, the machine learning system has already been trained using the method for training the above machine learning system. Therefore, the calculation unit generates output data from the input data and generates reconstructed data. Further, from the reconstructed data and the data corresponding to the reconstructed data, the calculation unit generates a difference value for quantifying the uncertainty of the output data. The output unit is configured to output the output data generated by the calculation unit, the uncertainty of the output data, and / or an uncertainty warning based on this uncertainty. The output unit is, for example, an interface with a system that further processes the output data.

[0034] A further aspect of the present invention relates to a vehicle equipped with the system for quantifying the uncertainty of the above output data. When an increase in uncertainty is detected in the output data generated by the machine learning system, corresponding measures may be taken. In the case of autonomous driving, for example, deceleration or transfer of vehicle control to the driver may be performed, and safety will be significantly improved.

[0035] The present invention will be described in further detail based on the embodiments described in the drawings. These embodiments are merely examples and should not be construed as limitations.

Brief Description of the Drawings

[0036]

Figure 1a

Figure 1b

Figure 2a

Figure 2b

Figure 3a

Figure 3b

Figure 4a

Figure 4b

Figure 5a

Figure 5b

Figure 6

Embodiments for Carrying Out the Invention

[0037] FIG. 1a shows a flowchart of an embodiment of a method for training a machine learning system 1, and FIG. 1b shows a flowchart of a method suitable for the embodiment of FIG. 1a for quantifying the uncertainty U of the output data Y'.

[0038] For training a machine learning system 1, such as a decision tree learning system, a support vector machine, a learning system based on regression analysis, a Bayesian network, a neural network or a convolutional neural network, training input data X and training target values Y are supplied. Using the machine learning system 1, output data Y' and reconstructed data X' similar to the training input data X are generated from the training input data X. The purpose of training is for the output data Y' to be as similar as possible to the training target values Y without causing overfitting, and for the reconstructed data X' to be as similar as possible to the training input data. For this purpose, from the generated output data Y', reconstructed data X', training target values Y and training input data X, using an error function 2, the differences still existing between the output data Y' and the training target values Y and between the reconstructed data X' and the training input data X are calculated. This difference is used to adjust the parameters of the machine learning system 1 using the error backpropagation method 3. This is repeated until a predetermined consistency is achieved or signs of overfitting appear.

[0039] Next, using the machine learning system 1 trained in this way, as shown in FIG. 1b, output data Y' and reconstructed data X' are generated from the input data X. Then, the difference between the reconstructed data X' and the input data X is determined using a metric 4. The difference value thus determined quantifies the uncertainty U of the output data Y'.

[0040] As an example, the input data X may be image data of an image, and the output data Y' may be classification data corresponding to the object represented in the image. When the machine learning system 1 can be used to generate a reconstructed image X' similar to the original image X, this indicates that similar input data X was already available during the training of the machine learning system 1, so the uncertainty U of the output data Y' is small. On the other hand, when the reconstructed image X' is significantly different from the original image X, this indicates that similar input data X was not used to train the machine learning system 1, so the uncertainty U of the output data Y' is large.

[0041] The methods of training the machine learning system 1 or quantifying the uncertainty U of the output data Y' shown in FIGS. 2a and 2b differ from the methods shown in FIGS. 1a and 1b in that the generation of the output data Y' and the reconstructed data X' is performed not by the common machine learning system 1 but by the first subsystem 1.1 or the second subsystem 1.2 of the machine learning system 1. As a result, the training of the second subsystem 1.2 may be performed, for example, after the training of the first subsystem 1.1, and it should be noted that the same training input data X needs to be used. In other respects, these methods are consistent with the methods of FIGS. 1a and 1b. Naturally, the error function 2 and the error backpropagation method 3 are adjusted for each subsystem 1.1 or 1.2 of the machine learning system 1.

[0042] On the other hand, in the method of training the machine learning system 1 or quantifying the uncertainty U of the output data Y' shown in FIGS. 3a and 3b, the machine learning system 1 includes two subsystems 1.1 and 1.2. The subsystem 1.1 generates the output data Y' from the input data X and is trained upstream of the subsystem 1.2. For the sake of clarity, the illustration of the training of the subsystem 1.1 is omitted here. The subsystem 1.2 acquires the output data Y' generated by the subsystem 1.1 as an input, and based on this, generates the reconstructed data Y'' that is similar to the training target value Y. Here, in order to train the subsystem 1.2, the generated reconstructed data Y'' is compared with the training target value Y, and the parameters of the subsystem 1.2 are adjusted using the error backpropagation method 3 so that the reconstructed data Y'' is as similar as possible to the training target value Y.

[0043] Next, in order to quantify the uncertainty U of the output data Y', the difference between the output data Y' and the reconstructed data Y'' is determined using a metric 4. Here, since a high similarity between the output data Y' and the reconstructed data Y'' indicates that there was training input data X similar to the input data X, the uncertainty U of the output data Y' is considered to be small. On the other hand, a large difference between the output data Y' and the reconstructed data Y'' indicates that there was no training input data X similar to the input data X, so the uncertainty U of the output data Y' is quantified as large.

[0044] The method of training the machine learning system 1 or quantifying the uncertainty U of the output data Y' shown in FIGS. 4a and 4b is different from the method shown in FIGS. 3a and 3b in that the second subsystem 1.2 uses the training input data X or the input data X as an input in addition to the output data Y' of the first subsystem 1.1. This further improves the determination of the uncertainty U of the output data Y'.

[0045] Finally, the method of training the machine learning system 1 or quantifying the uncertainty U of the output data Y', as shown in FIGS. 5a and 5b, is different from the method shown in FIGS. 3a and 3b or FIGS. 4a and 4b in that the second subsystem 1.2 generates reconstructed data X' corresponding to the input data X, rather than generating reconstructed data Y" corresponding to the output data Y'. Here, with respect to the second subsystem 1.2, it is optional to use the training input data X or the input data X as the input data, which is indicated by a dotted line.

[0046] FIG. 6 shows an exemplary embodiment of a vehicle 5 comprising a system 6 for quantifying the uncertainty U of the output data Y'. Here, the system 6 comprises an input unit 7, a computing unit 8, and an output unit 9. In this case, the input unit 7 receives input data X from sensors 10 of the vehicle 5, for example, an image sensor, a radar sensor, or a lidar sensor. Next, the received input data X is transferred to a computing device 8 that executes a method for quantifying the uncertainty U of the output data Y'. Then, the output data Y' and the uncertainty U thus generated are transferred via the output unit 9 to a further system 11 of the vehicle 5, for example, a system 11 for autonomous driving. Along with the output data Y', an uncertainty warning may also be transmitted, and the uncertainty warning is issued when the uncertainty U exceeds a predetermined value.

[0047] If the uncertainty U is too large or an uncertainty warning is issued, for example, the system 11 for autonomous driving may decelerate the vehicle 5 or transfer vehicle control to the driver. Although this application relates to the invention described in the claims, it also includes the following from other perspectives. 1. A method for training a machine learning system (1) that quantifies the uncertainty (U) of output data (Y'), wherein training data (X, Y) including training input data (X) and training target values (Y) is supplied, and using the training data (X, Y), the machine learning system (1) generates output data (Y') similar to the training target value (Y) when the training input data (X) is input, A method in which the parameters of the machine learning system (1) are adjusted so as to generate reconstruction data (X'; Y") representing a measure of the degree of knowledge of the training data (X, Y). 2. The reconstruction data (X') is similar to the training input data (X), the method according to 1 above. 3. The reconstruction data (Y") is similar to the training target value (Y), and the difference between the reconstruction data (Y") and the training target value (Y) is smaller than the difference between the output data (Y') and the training target value (Y), the method according to 1 above. 4. The method according to any one of 1 to 3 above, wherein the training input data (X) and / or the output data (Y') are used as inputs to generate the reconstruction data (X'; Y"). 5. The method according to any one of 1 to 4 above, wherein the machine learning system (1) is a neural network, particularly a convolutional neural network. 6. The method according to any one of 1 to 5 above, wherein the input data (X) particularly includes sensor data of a vehicle, particularly image data, radar data, and / or lidar data. 7. A method for quantifying the uncertainty (U) of the output data (Y') of a machine learning system (1), the method being a method in which the machine learning system (1) is trained by the method for training the machine learning system (1) according to any one of 1 to 6 above. The machine learning system (1) generates output data (Y') from the input data (X), The machine learning system (1) generates reconstruction data (X'; Y"), A method in which a difference value is generated using a metric (4), the reconstructed data (X′; Y″), and data (X; Y′) corresponding to the reconstructed data (X′; Y″), the difference value being a measure of the degree of knowledge of training data (X, Y) regarding the input data (X), quantifying the uncertainty (U) of the output data (Y′), and being assigned to the output data (Y′). 8. The method according to 7 above, wherein the data corresponding to the reconstructed data (X′) is the input data (X). 9. The method according to 7 above, wherein the data corresponding to the reconstructed data (X′) is the output data (Y′) generated from the input data (X). 10. The method according to any one of 7 to 9 above, wherein the input data (X) and / or the output data (Y′) generated from the input data (X) is used to generate the reconstructed data (X′; Y″). 11. The method according to any one of 7 to 10 above, wherein the metric (4) is applied only to a part of the reconstructed data (X′; Y″) and the data (X; Y′) corresponding to the reconstructed data (X′; Y″). 12. The method according to any one of 7 to 11 above, wherein an uncertainty warning is issued regarding the output data (Y′) whose uncertainty (U) exceeds a predetermined value. 13. The method according to any one of 7 to 12 above, wherein the input data (X) for further training the machine learning system (1) is stored regarding the output data (Y′) whose uncertainty (U) exceeds a predetermined value. 14. A system for quantifying the uncertainty (U) of the output data (Y′) of a machine learning system (1), comprising: An input unit (7) for receiving input data (X); A calculation unit (8) configured to execute the method according to any one of 7 to 13 above; and An output unit (9) for outputting the output data (Y′) generated by the calculation unit (8), the uncertainty (U) of the output data (Y′), and / or an uncertainty warning. 15. A vehicle comprising the system (6) for quantifying the uncertainty (U) of the output data (Y′) according to 14 above.

Claims

A method for training a machine learning system (1) that quantifies the uncertainty (U) of output data (Y') using a calculation unit (8), comprising: training data (X, Y) including training input data (X) and training target values (Y) is supplied, and using the training data (X, Y), the machine learning system (1) generates output data (Y') similar to the training target value (Y) when the training input data (X) is input, generates reconstruction data (X'; Y") representing a measure of the degree of knowledge of the training data (X, Y), wherein the reconstruction data (Y") is similar to the training target value (Y), and the parameters of the machine learning system (1) are adjusted such that the difference between the reconstruction data (Y") and the training target value (Y) is smaller than the difference between the output data (Y') and the training target value (Y). **Claim 2** The method according to claim 1, wherein the reconstruction data (X') is similar to the training input data (X). **Claim 3** The method according to claim 1, wherein the training input data (X) and / or the output data (Y') are used as inputs to generate the reconstruction data (X'; Y"). **Claim 4** The method according to claim 1, wherein the machine learning system (1) is a neural network, in particular a convolutional neural network. **Claim 5** The input data (X) used in the trained machine learning system (1) particularly includes sensor data of a vehicle, in particular image data, radar data and / or lidar data. The method according to claim 1. **Claim 6** A method for quantifying the uncertainty (U) of the output data (Y') of a machine learning system (1), wherein the machine learning system (1) is trained by the method for training the machine learning system (1) according to claim 1. In the method, the machine learning system (1) generates output data (Y') from input data (X), the machine learning system (1) generates reconstruction data (X'; Y"), a difference value is generated using a metric (4), the reconstruction data (X'; Y"), and data (X; Y') corresponding to the reconstruction data (X'; Y"), the difference value being a measure of the degree of knowledge of the training data (X, Y) regarding the input data (X), quantifying the uncertainty (U) of the output data (Y'), and assigned to the output data (Y'). **Claim 7** The method according to claim 6, wherein the data corresponding to the reconstructed data (X') is the input data (X).

8. The method according to claim 6, wherein the data corresponding to the reconstructed data (Y") is the output data (Y') generated from the input data (X).

9. The method according to claim 6, wherein the input data (X) and / or the output data (Y') generated from the input data (X) are used to generate the reconstructed data (X'; Y").

10. The method according to claim 6, wherein the metric (4) is applied only to a part of the reconstructed data (X'; Y") and the data (X; Y') V corresponding to the reconstructed data (X'; Y").

11. The method according to claim 6, wherein an uncertainty warning is issued for the output data (Y') for which the uncertainty (U) exceeds a predetermined value.

12. The method according to claim 6, wherein the input data (X) for further training the machine learning system (1) is stored for the output data (Y') for which the uncertainty (U) exceeds a predetermined value.

13. A system for quantifying the uncertainty (U) of the output data (Y') of a machine learning system (1), comprising: an input unit (7) for receiving input data (X); a calculation unit (8) configured to execute the method according to any one of claims 6 to 12; and an output unit (9) for outputting the output data (Y') generated by the calculation unit (8), the uncertainty (U) of the output data (Y') and / or an uncertainty warning.

14. A vehicle comprising the system (6) for quantifying the uncertainty (U) of the output data (Y') according to claim 13.

Images