Method and system for quantifying uncertainty in output data of a machine learning system and method for training a machine learning system - Patents.com

A method for training machine learning systems to generate reconstructed data for uncertainty quantification addresses computational complexity issues, enabling efficient and safe uncertainty estimation in applications like autonomous driving.

JP2024542050AActive Publication Date: 2024-11-13コンチネンタル·オートナマス·モビリティ·ジャーマニー·ゲゼルシャフト·ミト·ベシュレンクテル·ハフツング
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024525881
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-29
Filing Date
2022-11-24
Publication Date
2024-11-13
Estimated Expiration
2042-11-24

AI Technical Summary

Technical Problem

Existing methods for quantifying uncertainty in machine learning systems, such as convolutional neural networks, are computationally complex and memory-intensive, making them unsuitable for real-time applications, particularly in safety-critical systems like automated driving.

Method used

A method for training a machine learning system to quantify uncertainty by generating reconstructed data that measures familiarity with training data, allowing for uncertainty estimation with moderate computational costs, using techniques like error backpropagation and supervised learning to adjust parameters, and employing metrics like mean squared error or binary cross entropy to determine uncertainty.

Benefits of technology

The method enables accurate uncertainty quantification with computational costs comparable to standard output generation, improving safety in systems like autonomous driving by issuing warnings or taking corrective actions when uncertainty exceeds predetermined thresholds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542050000001_ABST
    Figure 2024542050000001_ABST
Patent Text Reader

Abstract

The present invention relates to a method for training a machine learning system (1) for quantifying the uncertainty (U) of output data (Y'), wherein training data (X,Y) is provided, comprising training input data (X) and a training target value (Y). Using the training data (X,Y), parameters of the machine learning system (1) are adjusted such that the machine learning system (1), upon input of the training input data (X), generates output data (Y') similar to the training target value (Y) and generates reconstructed data (X';Y") representing a measure of the knowability of the training data (X,Y). The present invention also relates to a method for quantifying the uncertainty (U) of output data (Y') of the machine learning system (1), wherein the machine learning system (1) has been trained by the above method, wherein the machine learning system (1) generates output data (Y') from the input data (X), and the machine learning system (1) generates the reconstructed data (X';Y"). Using the metric (4) and the reconstructed data (X';Y") and the data (X;Y') corresponding to the reconstructed data (X';Y"), a difference value is generated, which is a measure of the known degree of the training data (X,Y) with respect to the input data (X), and quantifies the uncertainty (U) in the output data (Y') and is assigned to the output data (Y').
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to machine learning, in particular to a method for training a machine learning system that quantifies uncertainty, a method for quantifying uncertainty in output data of a machine learning system, a system for quantifying uncertainty in output data of a machine learning system, and a vehicle equipped with a system for quantifying uncertainty in output data. [Background technology]

[0002] Machine learning has become a highly valuable tool in many fields, where we increasingly rely on reliable estimates of the uncertainty in the output of machine learning systems.

[0003] Thus, for example, in automated driving, a reliable estimation of the uncertainty in the classification output is a safety-relevant aspect, where uncertain situations for the system can be detected and accidents can be avoided by handing over control to the driver or by slowing down.

[0004] In particular, in convolutional neural networks, uncertainty estimation is a challenging task since in this case classification errors are uncorrelated with the actual uncertainty due to the nonlinearity of convolutional neural networks, and this effect becomes stronger as the convolutional neural network becomes more complex, for example by adding more layers with nonlinear activation functions or by nomination techniques.

[0005] Known methods for calculating uncertainty are the Monte Carlo dropout method and the ensemble method. Both methods are based on the principle of statistical classification result scatter modeling, which generates multiple results for an input signal. The more these results differ from each other, the greater the uncertainty. For this purpose, in the case of the Monte Carlo dropout method, the weights of the network are switched randomly, while in the case of the ensemble method, an output is generated for each member of the ensemble. In this case, a large number of results are required for the representative statistics. Due to the high computational complexity required for this, these methods are generally not suitable for implementation in real-time systems. Furthermore, the ensemble method requires a high memory cost. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] B. Lakshminarayanan, A. Pritzel, C. Blundell, “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles” (arXiv:1612.01474, 2017) [Non-Patent Document 2] C. Guo, G. Pleiss, Y. Sun, K. Weinberger, “On Calibration of Modern Neural Networks” (arXiv:1706.04599, 2017) Summary of the Invention [Problem to be solved by the invention]

[0007] It is therefore an object of the present invention to provide a method for quantifying the uncertainty of output data of a machine learning system with moderate computational cost. This object is achieved by the subject matter of the independent claims. Developments of the invention become apparent from the dependent claims and the following description. [Means for solving the problem]

[0008] One aspect of the invention relates to a method for training a machine learning system that quantifies uncertainty in output data, where training data is provided that includes training input data and a training target value, where a training target value is assigned to each training input data, where the training target value has been, for example, manually created for each training input data, and represents, for example, a classification of the training input data.

[0009] Using the training data, the machine learning system is trained, in other words the parameters of the machine learning system are adjusted so that the machine learning system generates output data similar to the training target value when the training input data is input. The parameters include, for example, weights between the individual input values ​​and the neurons in the case of a neural network. Here, the training of the machine learning system is performed using a supervised learning method, of which a large number of learning methods are known. In this way, for example, backpropagation can be used to train a neural network. In this case, during training, the parameters of the machine learning system are adjusted so that the error between the output data and the training target value is as small as possible. Here, the error between the output data and the training target value is determined, for example, via the distance between the output data and the training target value and the corresponding metric for the output data. It should be noted that overfitting is avoided here, which is achieved, for example, by taking into account the error between the output data generated from the test input data and the corresponding test target value. In this case, the test target value is assigned to the test input data, and the test input data and the test target value are not used for adjusting the parameters of the machine learning system.

[0010] The parameters of the machine learning system are adjusted using the training data, such that the machine learning system generates reconstruction data, where the reconstruction data represents a measure of the knownness of the training data. If the training data is known for the input data to be processed with the trained learning system, the uncertainty of the output data of the machine learning system is small. On the other hand, if the training data is almost unknown or completely unknown for the input data to be processed with the trained learning system, the uncertainty of the output data of the machine learning system is large. That is, the uncertainty of the output data of the machine learning system can be quantified via the reconstruction data. In this case, the computational cost is moderate, and the additional computational cost for calculating the reconstruction data corresponds at most to, for example, the computational cost for calculating the output data. Furthermore, since the calculation of the reconstruction data is performed based on the already available training data, no further input is required here.

[0011] Here, the reconstructed data may be generated as a further output of a machine learning system that generates the output data. Alternatively, the machine learning system may comprise two subsystems, a first subsystem generating the output data and a second subsystem generating the reconstructed data. When the output data and the reconstructed data are generated by two subsystems of a machine learning system, it is important that the training of both subsystems is based on the same training data, since only by this means does the reconstructed data represent a measure of familiarity of the same training data used to generate the output data.

[0012] In some embodiments, the reconstructed data is similar to the training input data, i.e., the parameters of the machine learning system are adjusted so that the error between the reconstructed data and the training input data is as small as possible, in other words, the input data is reconstructed via the reconstructed data.

[0013] In some embodiments, the reconstructed data is similar to the training target values, and the difference between the reconstructed data and the training target values ​​is smaller than the difference between the output data and the training target values. That is, the parameters of the machine learning system are adjusted so that the error between the reconstructed data and the training target values ​​is smaller than the error between the output data and the training target values. Since the parameters of the machine learning system have already been adjusted so that the error between the output data and the training target values ​​is as small as possible while avoiding overfitting, this means that the machine learning system is overfitted to generate the reconstructed data.

[0014] In some embodiments, the training input data is used as input to generate the reconstruction data. This is possible both when the reconstruction data is generated as a further output of the machine learning system generating the output data, and when the reconstruction data is generated by a second subsystem of the machine learning system. Alternatively or additionally, the output data may be used as input to generate the reconstruction data. This is only possible when the reconstruction data is generated by the second subsystem of the machine learning system, since this requires that the output data is available first. Thus, in this case, the output of the first subsystem of the machine learning system is used as input for the second subsystem of the machine learning system.

[0015] In some embodiments, the machine learning system is a neural network, which is particularly suitable for the method because it can be easily tuned, and in particular the neural network is a convolutional neural network, in which case the nonlinearity of the convolutional neural network is not an issue with respect to the usability of the method.

[0016] However, the method may be applied using other machine learning systems, such as decision tree learning, support vector machines, regression analysis or Bayesian networks. The method may also be applied in a multi-task classification system, which may, for example, comprise an encoder and a number of decoders. Here, for each decoder, reconstruction data is generated and the parameters of the machine learning system are adjusted accordingly. The machine learning system may also be divided into multiple subsystems, each of which may have the functionality of the machine learning system of the present disclosure, while the subsystems may differ from each other, for example in terms of the type of machine learning system or the selection of training data. In that case, the output data generated using the subsystems may be combined, which may result in an improved output.

[0017] In some embodiments, the input data includes sensor data, in particular image data, radar data and / or lidar data. As an example in the case of image data, the reconstructed data represents a reconstruction of the original image if the reconstructed data is similar to the training input data. In this case, the more accurately the original image can be reconstructed, the more the training data for this image is known, and the less uncertainty there is in the output data.

[0018] Here, the input data may be sensor data of a vehicle, and the output data is, for example, a classification of objects recorded in the sensor data. In this way, the method can be applied to quantify the uncertainty of output data for an autonomous driving system.

[0019] A further aspect of the invention relates to a method of quantifying uncertainty in output data of a machine learning system, the machine learning system having been trained by a method for training a machine learning system as described above.

[0020] Thus, the machine learning system may be, for example, a decision tree learning system, a support vector machine, a regression-based learning system, a Bayesian network, a neural network, or a convolutional neural network. The machine learning system may also be a multi-task classification system. The machine learning system may include two subsystems.

[0021] A machine learning system generates output data from input data, where the input data may for example be sensor data, in particular image data, radar data and / or lidar data, e.g. acquired by a vehicle, and the output data may be a classification of the input data and thus for example a classification of objects recorded in the sensor data.

[0022] The machine learning system also generates reconstructed data, where a difference value is generated from the reconstructed data and the data corresponding to the reconstructed data using a metric, where the difference value is a measure of how well known the training data is with respect to the input data, and thus quantifies the uncertainty of the output data and is assigned to the output data.

[0023] Therefore, the quantification of uncertainty is achieved by using a trained machine learning system, so that the computational cost is at most equal to the computational cost for calculating, for example, the output data.

[0024] The metric used here depends on what kind of data the reconstructed data and the data that corresponds to the reconstructed data are. Examples of such metrics are distance metrics or similarity measures, such as mean squared error, mean absolute error, structural similarity index or binary cross entropy.

[0025] It is also possible to determine the quality of the uncertainty estimates. Known techniques for this can be found, for example, in [1] or [2].

[0026] In some embodiments, the data corresponding to the reconstructed data is the input data. In this case, the machine learning system is trained so that the reconstructed data obtained during training is similar to the training input data. Thus, a difference value is generated from the reconstructed data and the input data using a metric. If there is training input data that is similar to the input data, the reconstructed data will also be similar to the input data, and therefore the difference value will be small. In this case, since there is input data that is similar to the training input data, it is expected that the reconstructed data will also have a small error, resulting in a small difference value due to good output data, and therefore a small uncertainty is obtained in the output data. On the other hand, if there is no training input data that is similar to the input data, there will be a large difference between the reconstructed data and the input data, and therefore the difference value will be large. This corresponds to a large uncertainty, and is due to the input data not corresponding to any of the training input data.

[0027] In some embodiments, the data corresponding to the reconstructed data is output data generated from the input data. In this case, the machine learning system has been trained such that the reconstructed data obtained during training is similar to a training target value, and the difference between the reconstructed data and the training target value is smaller than the difference between the output data generated from the training input data and the training target value. If the difference between the reconstructed data and the output data calculated using the metric is small, this indicates that there was training input data that was similar to the input data, and therefore the uncertainty of the output data is also small. On the other hand, if the difference between the reconstructed data and the output data calculated using the metric is large, this indicates that there was no training input data that was similar to the input data, and therefore the uncertainty of the output data is correspondingly large.

[0028] If the difference between the reconstructed data and the output data calculated using the metric is small, the reconstructed data can be used to improve the output of the machine learning system. For this purpose, for example, instead of the output data, the reconstructed data may be output, which may represent a more accurate result than the output data based on the training of the machine learning system. Alternatively, for example, instead of the output data, an average value of the output data and the reconstructed data may be output, which may also represent a more accurate result than the output data.

[0029] In some embodiments, the input data is used to generate the reconstruction data. Alternatively or additionally, output data generated from the input data is used to generate the reconstruction data, where which data is used to generate the reconstruction data is based on which data was used to generate the reconstruction data during training of the machine learning system.

[0030] In some embodiments, the metric is applied only to a part of the reconstruction data and the data corresponding to the reconstruction data. Thus, for example, the metric may be applied only to a part of the image, limited, for example, by a bounding box, and specifically including the detected object. In this case, the uncertainty in the classification of this object is analyzed separately. As a further example, the metric may be applied to the entire image having a low resolution, for example, in case small details should be considered as rather unimportant.

[0031] In some embodiments, an uncertainty warning is issued for output data whose uncertainty exceeds a predetermined value, which may be taken into account by the system using the output data accordingly. In this way, for example in automated driving, where objects detected on the basis of sensor data using a machine learning system are crucial, if the uncertainty of the output data of the machine learning system increases, corresponding safety measures may be initiated, for example deceleration or transfer of vehicle control to the driver.

[0032] In some embodiments, input data for further training the machine learning system is stored for output data with an uncertainty exceeding a predetermined value. Then, for the input data thus stored, corresponding target values ​​are generated, for example by classification by a user. In this case, these new pairs of input data and target values ​​are used during training of the machine learning system, so that the machine learning system can generate reliable output data even from input data similar to the new input data.

[0033] A further aspect of the invention relates to a system for quantifying the uncertainty of output data of a machine learning system. The system comprises an input unit, a calculation unit and an output unit. Here, the input unit is configured to receive input data, for example configured as an interface with a sensor. The calculation unit is configured to execute the method for quantifying the uncertainty of output data of the machine learning system described above. Here, the machine learning system has already been trained using the method for training a machine learning system described above. Thus, the calculation unit generates output data from the input data and generates reconstruction data. Furthermore, from the reconstruction data and data corresponding to the reconstruction data, the calculation unit generates a difference value quantifying the uncertainty of the output data. The output unit is configured to output the output data generated by the calculation unit, the uncertainty of the output data and / or an uncertainty warning based on this uncertainty. The output unit is, for example, an interface with a system for further processing the output data.

[0034] A further aspect of the invention relates to a vehicle equipped with a system for quantifying the uncertainty of output data as described above. If an increased uncertainty is detected in the output data generated by the machine learning system, corresponding measures may be taken, e.g. in case of autonomous driving, decelerating or transferring control of the vehicle to the driver, thus significantly improving safety.

[0035] The invention will now be explained in more detail on the basis of the embodiments illustrated in the drawings, which are merely illustrative and should not be understood as limiting. [Brief description of the drawings]

[0036] [Figure 1a] FIG. 1a is a flow diagram of an example embodiment of a method for training a machine learning system. [Figure 1b] FIG. 1b is a flow diagram of a method suitable for the example embodiment of FIG. 1a for quantifying uncertainty in output data. [Figure 2a] FIG. 2a is a flow diagram of a further example embodiment of a method for training a machine learning system. [Figure 2b] FIG. 2b is a flow diagram of a method suitable for the example embodiment of FIG. 2a for quantifying uncertainty in the output data. [Figure 3a] FIG. 3a is a flow diagram of a further example embodiment of a method for training a machine learning system. [Figure 3b] FIG. 3b is a flow diagram of a method suitable for the example embodiment of FIG. 3a for quantifying uncertainty in the output data. [Figure 4a] FIG. 4a is a flow diagram of a further example embodiment of a method for training a machine learning system. [Figure 4b] FIG. 4b is a flow diagram of a method suitable for the example embodiment of FIG. 4a for quantifying uncertainty in the output data. [Figure 5a] FIG. 5a is a flow diagram of a further example embodiment of a method for training a machine learning system. [Figure 5b] FIG. 5b is a flow diagram of a method suitable for the example embodiment of FIG. 5a for quantifying uncertainty in the output data. [Figure 6] FIG. 6 is a schematic diagram of an example embodiment of a vehicle equipped with a system for quantifying uncertainty in output data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] FIG. 1a shows a flow diagram of an example embodiment of a method for training a machine learning system 1, and FIG. 1b shows a flow diagram of a method suitable for the example embodiment of FIG. 1a for quantifying the uncertainty U in the output data Y'.

[0038] To train a machine learning system 1, for example a decision tree learning system, a support vector machine, a learning system based on regression analysis, a Bayesian network, a neural network or a convolutional neural network, training input data X and training target values ​​Y are provided. Using the machine learning system 1, output data Y' and reconstructed data X' similar to the training input data X are generated from the training input data X. The objective of training is that the output data Y' is as similar as possible to the training target value Y and that the reconstructed data X' is as similar as possible to the training input data without causing overfitting. For this purpose, from the generated output data Y', reconstructed data X', training target value Y and training input data X, an error function 2 is used to calculate the differences that still exist between the output data Y' and the training target value Y and between the reconstructed data X' and the training input data X. This difference is used to adjust the parameters of the machine learning system 1 using an error backpropagation method 3. This is repeated until a predetermined consistency is achieved or until signs of overfitting appear.

[0039] The machine learning system 1 thus trained is then used to generate output data Y' and reconstructed data X' from the input data X, as shown in Figure 1b. The difference between the reconstructed data X' and the input data X is then determined using a metric 4. The difference value thus determined quantifies the uncertainty U of the output data Y'.

[0040] As an example, the input data X may be image data of an image, and the output data Y' may be classification data corresponding to an object depicted in the image. If the machine learning system 1 can be used to generate a reconstructed image X' that is similar to the original image X, this indicates that similar input data X was already available during training of the machine learning system 1, and therefore the uncertainty U of the output data Y' is small. On the other hand, if the reconstructed image X' is significantly different from the original image X, this indicates that similar input data X was not used to train the machine learning system 1, and therefore the uncertainty U of the output data Y' is large.

[0041] The methods of training a machine learning system 1 or quantifying the uncertainty U of the output data Y' shown in Figures 2a and 2b differ from the methods shown in Figures 1a and 1b in that the generation of the output data Y' and the reconstruction data X' is not performed by a common machine learning system 1 but by a first subsystem 1.1 or a second subsystem 1.2 of the machine learning system 1. As a result, it should be noted that the training of the second subsystem 1.2 may be performed, for example, after the training of the first subsystem 1.1, with the same training input data X having to be used. In other respects, these methods correspond to the methods of Figures 1a and 1b. Naturally, the error function 2 and the backpropagation method 3 are adjusted for each subsystem 1.1 or 1.2 of the machine learning system 1.

[0042] On the other hand, in the method for training a machine learning system 1 or quantifying the uncertainty U of output data Y', shown in Figs. 3a and 3b, the machine learning system 1 comprises two subsystems 1.1 and 1.2. The subsystem 1.1 generates output data Y' from input data X and is trained upstream of the subsystem 1.2. For clarity, the illustration of the training of the subsystem 1.1 is omitted here. The subsystem 1.2 takes as input the output data Y' generated by the subsystem 1.1 and based on this generates reconstructed data Y" similar to the training target value Y. Here, in order to train the subsystem 1.2, the generated reconstructed data Y" is compared with the training target value Y and the parameters of the subsystem 1.2 are adjusted using the backpropagation method 3 so that the reconstructed data Y" is as similar as possible to the training target value Y.

[0043] Next, to quantify the uncertainty U of the output data Y', the difference between the output data Y' and the reconstructed data Y" is determined using metric 4. Here, a high similarity between the output data Y' and the reconstructed data Y" indicates that there was training input data X similar to the input data X, so the uncertainty U of the output data Y' is considered to be small. On the other hand, a large difference between the output data Y' and the reconstructed data Y" indicates that there was no training input data X similar to the input data X, so the uncertainty U of the output data Y' is quantified as large.

[0044] The method of training the machine learning system 1 or quantifying the uncertainty U of the output data Y' shown in Figures 4a and 4b differs from the method shown in Figures 3a and 3b in that the second subsystem 1.2 also uses as input the training input data X or the input data X in addition to the output data Y' of the first subsystem 1.1, which further improves the determination of the uncertainty U of the output data Y'.

[0045] Finally, the method of training a machine learning system 1 or quantifying the uncertainty U of the output data Y′ shown in Figures 5a and 5b differs from the method shown in Figures 3a and 3b or Figures 4a and 4b in that the second subsystem 1.2 generates reconstructed data X′ corresponding to the input data X, but not reconstructed data Y″ corresponding to the output data Y′. Here, the use of training input data X or input data X as input data is optional for the second subsystem 1.2, and is therefore shown with dotted lines.

[0046] Fig. 6 shows an example embodiment of a vehicle 5 equipped with a system 6 for quantifying the uncertainty U of output data Y'. Here, the system 6 comprises an input unit 7, a calculation unit 8 and an output unit 9. In this case, the input unit 7 receives input data X from a sensor 10 of the vehicle 5, for example an image sensor, a radar sensor or a lidar sensor. The received input data X is then forwarded to a calculation device 8 which executes a method for quantifying the uncertainty U of the output data Y'. The output data Y' thus generated and the uncertainty U are then forwarded via the output unit 9 to a further system 11 of the vehicle 5, for example a system 11 for autonomous driving. Together with the output data Y', an uncertainty warning may also be transmitted, which is issued if the uncertainty U exceeds a predefined value.

[0047] If the uncertainty U is too large or an uncertainty warning is issued, for example, the system for automated driving 11 may slow down the vehicle 5 or hand over vehicle control to the driver.

Claims

1. A method for training a machine learning system (1) that quantifies uncertainty (U) in output data (Y'), comprising: providing training data (X, Y) including training input data (X) and a training target value (Y), the machine learning system (1) using the training data (X, Y) performing: generating output data (Y') similar to the training target value (Y) upon input of the training input data (X); The parameters of the machine learning system (1) are adjusted to generate reconstruction data (X';Y") that represents a measure of the familiarity of training data (X,Y).

2. The method of claim 1 , wherein the reconstructed data (X′) is similar to the training input data (X).

3. 2. The method of claim 1, wherein the reconstructed data (Y") is similar to the training target value (Y), and a difference between the reconstructed data (Y") and the training target value (Y) is smaller than a difference between the output data (Y') and the training target value (Y).

4. The method according to any one of claims 1 to 3, wherein the training input data (X) and / or the output data (Y') are used as inputs to generate the reconstruction data (X';Y'').

5. The method according to any one of claims 1 to 4, wherein the machine learning system (1) is a neural network, in particular a convolutional neural network.

6. The method according to any one of the preceding claims, wherein the input data (X) comprises sensor data, in particular image data, radar data and / or lidar data, in particular of a vehicle.

7. A method for quantifying uncertainty (U) of output data (Y') of a machine learning system (1), the machine learning system (1) being trained by the method for training a machine learning system (1) according to any one of claims 1 to 6, comprising: The machine learning system (1) generates output data (Y') from input data (X); The machine learning system (1) generates reconstructed data (X'; Y"); using a metric (4), the reconstructed data (X';Y") and data (X;Y') corresponding to the reconstructed data (X';Y"), a difference value is generated, the difference value being a measure of the known degree of training data (X,Y) with respect to the input data (X) and quantifying an uncertainty (U) in the output data (Y') and assigned to the output data (Y').

8. The method of claim 7 , wherein the data that corresponds to the reconstructed data (X′) is the input data (X).

9. 8. The method of claim 7, wherein the data corresponding to the reconstructed data (X') is the output data (Y') generated from the input data (X).

10. 10. The method according to claim 7, wherein the input data (X) and / or the output data (Y') generated from the input data (X) are used to generate the reconstructed data (X';Y'').

11. 11. The method according to claim 7, wherein the metric (4) is applied only to the reconstructed data (X';Y") and to a portion of the data (X;Y') V that corresponds to the reconstructed data (X';Y").

12. The method according to any one of claims 7 to 11, wherein an uncertainty warning is issued for output data (Y') whose uncertainty (U) exceeds a predefined value.

13. The method according to any one of claims 7 to 12, wherein for output data (Y') whose uncertainty (U) exceeds a predetermined value, the input data (X) for further training of the machine learning system (1) is stored.

14. A system for quantifying uncertainty (U) of output data (Y') of a machine learning system (1), comprising: an input unit (7) for receiving input data (X); A computing unit (8) configured to carry out the method according to any one of claims 7 to 13, and an output unit (9) for outputting the output data (Y') generated by the calculation unit (8), an uncertainty (U) of the output data (Y') and / or an uncertainty warning.

15. A vehicle comprising a system (6) for quantifying the uncertainty (U) of output data (Y') according to claim 14.

Citation Information

Patent Citations

  • Evaluation device, motion control device, evaluation method, and evaluation program

    JP2019139277A

  • Reliability determination device and reliability determination method

    JP6972444B1

  • Method and system for image analysis

    US20200250812A1

  • Data evaluation method and device, and breakdown diagnosis method and device

    WO2016132468A1