Concept for quantifying the uncertainty of a prediction in a control unit of a system

Abstract

The invention relates to a computer-implemented method for calculating a plurality of neural networks for quantifying the uncertainty of a prediction in a control unit of a system, and to a corresponding computer program. The invention also relates to a device for processing sensor data from a system, in particular a vehicle, and to a corresponding computer program.The procedure for computing a plurality of neural networks for quantifying a prediction uncertainty in a control unit of a system comprises obtaining (1) a first neural network, computing (2, 3, 4) a Laplace approximation for at least a subset of weight edges of the first neural network, wherein the Laplace approximation corresponds to a multidimensional probability distribution over at least the subset of weight edges of the first neural network, and computing (6) the plurality of neural networks based on the multidimensional probability distribution over at least the subset of weight edges.

Description

[0001] The invention relates to a computer-implemented method for calculating a plurality of neural networks for quantifying the uncertainty of a prediction in a control unit of a system, and to a corresponding computer program.

[0002] The invention also relates to a device for processing sensor data of a system, in particular of a vehicle, and to a corresponding computer program.

[0003] Neural networks still suffer from various shortcomings, such as poor calibration, over-reliance, and the "forgetting" of aspects and content. Furthermore, training neural networks presents challenges in selecting suitable neural network architectures and hyperparameters. Bayesian approaches can address such problems by, for example, enabling the quantification of prediction uncertainty and aiding in the selection of appropriate models and hyperparameters. However, deep neural networks using conventional Bayesian inference (Markov-chain Monte Carlo or variational inference) are difficult to implement and expensive to train.

[0004] Control units in systems such as vehicles generally have limited computing power available. Simultaneously, a large number of different sensor data points are evaluated using neural networks, which consequently have a large number of inputs and outputs. In this scenario, the application of conventional Bayesian inference is not feasible due to both the training and computational overhead of the inference process.

[0005] There is a need to make the advantages of a Bayesian approach, such as modeling the uncertainty of the neural network's prediction, usable in control units.

[0006] This need is met by the subject matter of the independent claims.

[0007] The present invention is based on the understanding that the advantages of a Bayesian approach can also be achieved using approximation functions. In particular, E. Daxberger et al. described an approach in "Laplace Redux - Effortless Bayesian Deep Learning" that uses a Laplace approximation to model the posterior of a neural network. In Bayesian inference, the prior and the posterior are important concepts: The prior is the probability distribution that reflects existing knowledge or assumptions about the parameters of a statistical model before new data are considered. The prior serves as a starting point and influences the computation of the posterior. The posterior is the updated probability distribution of the parameters of a statistical model after new data have been considered. It is computed using the prior and the data by Bayes' theorem.Bayesian inference combines these two components to calculate the probability distribution of a model's parameters. According to Daxberger, the calculation of the posterior can now be simplified using a Laplace approximation. To make the Laplace probability distribution usable for modeling uncertainty in an electronic control unit (ECU), it is used here to derive an ensemble of neural networks from a first neural network, for example, using Monte Carlo simulations based on the Laplace probability distribution. The Laplace approximation is used to obtain an estimate of the uncertainty of the trained neural networks. From this, uncertainty limits for the model prediction can then be derived. The uncertainty estimation can then be used to improve performance, since the uncertainty limits for specific operating points during the ECU's runtime are known.In contrast, conventional neural networks do not account for the uncertainty arising from the small amount of data. Therefore, the prediction of a neural network does not also represent the uncertainty of the prediction; that is, the prediction is more of a point estimate than a distribution. The approach according to the invention, however, can estimate the epistemic uncertainty and provide uncertainty limits for the model prediction on the control unit.

[0008] One aspect of the present invention relates to a computer-implemented method for calculating a plurality of neural networks to quantify the uncertainty of a prediction in a control unit of a system. The method comprises obtaining a (previously trained) first neural network. The method comprises calculating a Laplace approximation for at least a subset of weight edges of the first neural network. The Laplace approximation corresponds to a multidimensional probability distribution over at least the subset of weight edges of the first neural network. The method comprises calculating the plurality of neural networks based on the multidimensional probability distribution over at least the subset of weight edges.By using a Laplace approximation, the advantages of Bayesian networks (estimation of uncertainty) can be leveraged without incurring their disadvantages (the prohibitively high computational effort required for complex deep neural networks). The resulting neural networks can then be used for inference on a control unit within an ensemble, where the variance of the neural network predictions can be used to estimate the uncertainty.

[0009] In particular, the first neural network can be trained to make a prediction based on sensor data from a system, especially a vehicle. Especially in the (real-time) evaluation of sensor data, estimating the uncertainty is helpful in order to (not) initiate appropriate reactions to the sensor data, depending on the degree of uncertainty.

[0010] Preferably, the majority of neural networks can be computed by drawing Monte Carlo samples, and especially quasi-Monte Carlo samples (e.g., using Latin hypercube sampling or Sobol sampling), based on the multidimensional probability distribution of at least the subset of weight edges. This creates neural networks in which the weight edges, starting from the weight of the first neural network, which is considered the maximum a posterior (MAP), differ slightly from those of the first neural network based on the Gaussian distribution over the MAP. This leads to a small deviation between the predictions of the individual neural networks, which can be used to model the uncertainty on the control unit.

[0011] For example, at least 4 (or at least 6, or at least 8) (quasi-) Monte Carlo samples and / or at most 1000 (or at most 100, or at most 30) (quasi-) Monte Carlo samples can be drawn to compute at least 4 (or at least 6, or at least 8) and / or at most 1000 (or at most 100, or at most 30) neural networks for the majority of neural networks. The number of (quasi-) Monte Carlo draws is a trade-off between better modeling of the deviations and the amount of resources required by the control unit for inference across the ensemble of neural networks.

[0012] In particular, the multidimensional probability distribution can be based on maximum a posterior values ​​of the respective weighting edges and at least on an approximation of a Hessian matrix. Specifically, in the multivariate case, the posterior distribution can be given by p(θ|D)≃N(θ;θMAP,Σ)with Σ:=(∇θ2L(D;θ)|θ=θMAP)−1 are calculated, where p(θ|D) is of the posterior distribution, N the Laplace approximation, θ the weighting edges, θ MAP the maximum a posteriori values, and ∇θ2L(D;θ)|θ=θMAP corresponds to the Hessian matrix, where L(D;θ) This refers to empirical risk. In this context, the Hessian matrix corresponds to a covariance matrix of the probability distribution.

[0013] The Hessian matrix is ​​an N × N matrix, meaning it grows quadratically with the parameters N of the neural network. Therefore, it is advantageous to calculate only an approximation or factorization of the Hessian matrix. Accordingly, determining the Laplace approximation can involve calculating an approximation or factorization of the Hessian matrix. Alternatively or additionally, the subset of weighted edges over which the probability distribution is calculated can be chosen to be small (for example, only edges connected to specific levels of the neural network). For instance, diagonal factorization, Kronecker factorization, or low-rank factorization can be chosen, or the Fisher information matrix can be used as an approximation of the Hessian matrix. Kronecker factorization, in particular, has proven to be both accurate and efficient.The possible factorizations of the Hesse matrix can be found, for example, in the publication by Daxberger or the Python module "Laplace approximations for deep learning" by Alexander Immer.

[0014] As mentioned earlier, most neural networks form an ensemble of neural networks. The uncertainty in an ensemble prediction can arise from the variance in the individual predictions of the ensemble's neural networks. The variance of the ensemble's predictions thus allows the uncertainty of the prediction to be quantified – the greater the variance, the more uncertain the actual prediction, since even minor changes can lead to jumps in the prediction.

[0015] The method is a computer-implemented method, i.e., it is carried out by a computer. Accordingly, the invention also relates to a computer program comprising instructions which, when the program is executed by a computer or control unit, cause the computer or control unit to execute the method.

[0016] Preferably, the ensemble of neural networks is used in a control unit. A further aspect of the invention therefore relates to a device for processing sensor data from a system, in particular a vehicle, and to a system, in particular a vehicle, with such a control unit. The device comprises an interface for receiving sensor data from at least one sensor of the system. The device includes a memory in which an ensemble of neural networks is stored. The ensemble of neural networks comprises a plurality of neural networks. The plurality of neural networks are computed based on a Laplace approximation for at least a subset of weight edges of a first neural network, for example, using the method described above. The device comprises at least one processor configured for processing the sensor data using the ensemble.The processor is designed to calculate a predicted value based on the output of the ensemble's neural networks, for example, by calculating the average, median, or voting of the individual predicted values. The processor is also designed to estimate a value representing the uncertainty of the predicted value based on the output of the ensemble's neural networks. When calculating multiple predicted values ​​for the same prediction using an ensemble of neural networks, particularly in regression applications, the resulting predicted values ​​may differ slightly. The greater the difference between the predicted values, the greater the variance of the individual predicted values ​​and, consequently, the greater the uncertainty of the prediction.

[0017] For example, at least one processor can be trained to estimate the uncertainty of the prediction value based on the variance of the prediction value in the outputs of the ensemble's neural networks. The variance is a suitable value for estimating the uncertainty. Alternatively, another value, such as the difference between the largest and smallest prediction values, perhaps in normalized form, can be used to estimate the uncertainty.

[0018] In some cases, the ensemble can be used to pass not only the ensemble's prediction value but also its uncertainty to a subsequent processing step. Alternatively, the passed-through prediction value can be directly adjusted, for example, to a conservative prediction value that lies within the range of predictions from the individual networks. For instance, at least one processor can be configured to adjust the prediction value if the value representing the uncertainty of the prediction value meets a certain condition (such as if the uncertainty is too large). This prevents subsequent processing steps from being disrupted by a highly likely incorrect prediction value. Alternatively or additionally, at least one processor can be configured to issue a warning signal if the value representing the uncertainty of the prediction value meets a certain condition.The warning signal allows the subsequent processing step to take measures to prevent far-reaching decisions or reactions from being triggered based on the uncertain forecast step. For example, a decision or reaction can be delayed to wait and see if the uncertainty improves.

[0019] The invention further relates to a corresponding method for processing sensor data from a system, in particular a vehicle. In particular, the method can be executed by the system / vehicle, for example, by a control unit or computer of the system / vehicle. The method comprises receiving sensor data from at least one sensor of the system. The method comprises processing the sensor data using an ensemble of neural networks, wherein the ensemble of neural networks comprises a plurality of neural networks and wherein the plurality of neural networks are computed based on a Laplace approximation for at least a subset of weight edges of a first neural network. The method comprises calculating a prediction value based on an output from the neural networks of the ensemble.The method involves estimating a value representing the uncertainty of the prediction based on the output of the ensemble's neural networks. It is evident that the method replicates the functionality of the previously presented device. Accordingly, features introduced in relation to the device can also be incorporated into the method.

[0020] Another aspect of the present invention relates to a computer program comprising instructions which, when the program is executed by a computer or a control unit, cause the computer or the control unit to execute the last presented method.

[0021] There are now various ways to advantageously elaborate and further develop the teaching of the present invention. For this purpose, reference should be made, on the one hand, to the claims subordinate to claims 1 and 7, and on the other hand, to the following explanation of preferred embodiments of the invention with reference to the figures. The figures show: Fig. Figure 1 shows a flowchart of a procedure for calculating a plurality of neural networks to quantify the uncertainty of a prediction in a control unit of a system; Fig. Figure 2a shows a flowchart of a procedure for processing sensor data from a system, in particular a vehicle; and Fig. Figure 2b shows a schematic diagram of a control unit for processing sensor data of a system, in particular a vehicle.

[0022] The present invention relates to the estimation of model uncertainty on an electronic control unit (ECU), such as a vehicle ECU. To efficiently calculate the model uncertainty on an ECU, a neural network (NN) can be used as a basis, which, for example, has been trained using mean square error and weight decay. In particular, an already trained neural network can be used, or a neural network can be trained specifically for this purpose. Next, a Laplace approximation is performed to obtain the uncertainty of the neural network. The result is the distribution of the parameters of the neural network. For this purpose, techniques such as those shown by Daxberger in the aforementioned publication can be used. The Laplace approximation (David JC MacKay - Bayesian Interpolation (1992), E. Daxberger, et al.)Laplace Redux - Effortless Bayesian Deep Learning (2022) appears to be a pragmatic and promising approach to Bayesian deep learning. The Laplace approach is one of the oldest and fastest available approximations. To give an example: Let's consider... p(x)=1zf(x), where Z = ∫ f(x)dx is the normalization constant. Then p(x) is the posterior and f(x) is the probability times the prior. The Laplace approximation aims to find a Gaussian approximation q(x) for p(x) whose center is the modal maximum (MAP) x0 of p(x).

[0023] Considering the Taylor extension of In f(x), centered on the mode x0, such that p'(x0) = 0 lnf(x)≃lnf(x0)+(x−x0)ddxlnf(x)|x=x0,with A=−d2dx2lnf(x)where (x−x0)ddxlnf(x)=0(sin x=x0).

[0024] If you take the exponential value, you get f(x)≃f(x0)exp(−A2(x−x0)2)⇒p(x)≃(A2π)1 / 2exp(−A2(x−x0)2) It follows p(x)≃N(x;x0,A−1), with A > 0, i.e. the posterior corresponds to the Laplace approximation.

[0025] This principle can also be applied to a trained neural network. Within the framework of empirical risk minimization, the training of a network, i.e., the minimization of empirical risk, breaks down into several components. L(D;θ), into a sum over empirical loss terms and a regularizer (such as weight decay). In a Bayesian context, these terms can be identified with iid log probabilities and a log-priority. Training a regularized network thus yields a MAP θ MAP for a specific choice of prior depending on the chosen regularizer.

[0026] If the Laplace approximation is derived in a similar way as before, the posterior distribution in the multivariate case is given by (see Daxberger 2022) p(θ|D)≃N(θ;θMAP,Σ)with Σ:=(∇θ2L(D;θ)|θ=θMAP)−1

[0027] The advantage of this approach is that the Laplace approximation can be applied to a pre-trained network (i.e., θ as parameters / weighting edges and θ). MAP as MAP values). However, the N × N Hessian matrix must be constructed. ∇θ2L(D;θ)|θ=θMAP The computational effort depends on the number of parameters N. This effort can be reduced by choosing a suitable factorization or approximation.

[0028] Various approximations and factorizations (diagonal, low-rank, etc.) of the Hessian matrix are available, such as the Fisher information matrix. Furthermore, when calculating the Laplace approximation, and thus also the Hessian matrix, only a subset of the network parameters can be considered, e.g., parameters with high magnitudes, specific layers, etc.

[0029] The prediction distribution can be calculated as follows. The prediction for a sample x* ∈ ℝ m Based on the posterior p(θ|D), the evaluation requires p(y|f(x*),D)=∫p(y|fθ(x*))p(θ|D)dθ

[0030] This integral is generally unsolvable, but it can be approximated (again). For example, a Monte Carlo estimation can be used. Furthermore, the neural network can be linearized so that the prediction is again Gaussian.

[0031] The Laplace approximation can now be used to model the uncertainty in the control unit. For this purpose, (a few) Monte Carlo samples can be drawn from the obtained (posterior) distribution of the neural network parameters. In particular, quasi-Monte Carlo samples, for example using Latin hypercube sampling or Sobol sampling, can be drawn, as these yield better results than classical Monte Carlo sampling. The use of quasi-Monte Carlo sampling has the advantage that fewer parameter sets are required, since the drawn parameter sets are better distributed (compared to classical Monte Carlo sampling). (Classical) Monte Carlo sampling can lead to the drawn parameters being close to each other and forming clusters, and therefore, especially with a large number of samples, to good estimates. The drawn parameter sets, i.e.,The neural networks, based on the extracted parameter sets, can now be stored on the control unit or generated there from the extracted parameter sets. The resulting ensemble is then used for prediction. The uncertainty can then be estimated based on the ensemble prediction. This approach enables improved performance due to knowledge of the model uncertainty. This allows for a reduction of the previously used large margins of error.

[0032] This approach is in the Fig. Figures 1 to 2b are shown. Fig. Figure 1 shows a flowchart of a procedure for computing a plurality of neural networks to quantify the uncertainty of a prediction in a control unit of a system. The procedure includes obtaining a first neural network (1). In particular, an existing trained neural network can be loaded, or the neural network can be trained. The procedure includes computing a Laplace approximation for at least a subset of weight edges of the first neural network (2, 3, 4). The Laplace approximation corresponds to a multidimensional probability distribution over at least the subset of weight edges of the first neural network. Computing the Laplace approximation can, for example, include obtaining (2) (e.g., extracting) the MAP value from the first neural network, and computing an approximation or factorization of the Hessian matrix (3).From the multidimensional probability distribution, (quasi-)Monte Carlo samples of parameters (i.e., in particular weighting edges) can now be drawn. These are then used to compute the majority of neural networks based on the multidimensional probability distribution over at least the subset of weighting edges.

[0033] The parameter sets or neural networks are then stored in a memory 15 (see Fig. 2b) of control unit 13 stored and used for prediction. Fig. Figure 2a shows a flowchart of a procedure for processing sensor data from a system, in particular a vehicle. The procedure includes obtaining sensor data from at least one sensor of the system. The procedure includes processing the sensor data using the ensemble of neural networks. The ensemble of neural networks comprises a plurality of neural networks. The plurality of neural networks are computed based on a Laplace approximation for at least a subset of weight edges of a first neural network. The procedure includes calculating a predicted value based on an output from the neural networks of the ensemble. The procedure includes estimating a value representing the uncertainty of the predicted value based on the output from the neural networks of the ensemble.If the value representing the uncertainty of the prediction indicates that the uncertainty is particularly high, the prediction can also be modified, for example by choosing a "more certain" prediction. The prediction can then be provided, possibly together with the value representing the uncertainty of the prediction and / or with a warning about the uncertainty of the prediction, for example to a subsequent processing step. For instance, the prediction can be used in the context of at least partially autonomous driving of a vehicle, whereby the value representing the uncertainty of the prediction can be used to better assess the vehicle's condition in traffic.

[0034] The procedure of Fig. 2a can, for example, be from the control unit of Fig. 2b will be carried out. Fig. Figure 2b shows a schematic diagram of a control unit 13 for processing sensor data from a system, in particular a vehicle. The control unit 13 includes an interface 14 for receiving sensor data from at least one sensor of the system. The control unit 13 includes a memory 15 on which the ensemble of neural networks is stored. The control unit further includes at least one processor 16, which is coupled to the interface 14 and the memory and is configured to carry out the method of Fig. 2a.

[0035] Regarding further advantageous embodiments of the device according to the invention, reference is made to the general part of the description and to the attached claims in order to avoid repetition.

[0036] Finally, it should be expressly pointed out that the exemplary embodiments of the device according to the invention described above serve only to discuss the claimed teaching, but do not limit it to these exemplary embodiments. Reference sign 1. Obtaining / Training a Neural Network 2. Obtaining a maximum a posterior value for at least a subset of weighting edges 3. Calculating an approximation or factorization of a Hessian matrix 4. Calculating a Laplace approximation 5. Drawing Monte Carlo Samples 6. Calculating a plurality of neural networks 7. Obtaining sensor data 8. Processing the sensor data using an ensemble of neural networks 9. Calculating a predictive value 10. Calculating a value representing the uncertainty of the forecast value 11. Adjusting the prediction value 12. Providing the forecast value and, if applicable, the value representing the uncertainty of the forecast value. 13 Control unit 14 Interface 15 storage 16 processor QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] David JC MacKay - Bayesian Interpolation (1992

[0022] E. Daxberger, et al. - Laplace Redux - Effortless Bayesian Deep Learning, (2022

[0022]

Claims

[1] Computer-implemented method for calculating a plurality of neural networks to quantify the uncertainty of a prediction in a control unit of a system, the method comprising: Obtaining (1) a first neural network; Compute (2, 3, 4) a Laplace approximation for at least a subset of weight edges of the first neural network, where the Laplace approximation corresponds to a multidimensional probability distribution over at least the subset of weight edges of the first neural network; Compute (6) the plurality of neural networks based on the multidimensional probability distribution over at least the subset of weighting edges. [2] Method according to claim 1, wherein the first neural network is trained to make a prediction based on sensor data of a system, in particular a vehicle. [3] Method according to one of claims 1 or 2, wherein the plurality of neural networks are computed by drawing (4) Monte Carlo samples, in particular quasi-Monte Carlo samples, based on the multidimensional probability distribution of at least the subset of weighting edges. [4] Method according to any one of claims 1 to 3, wherein at least 4 Monte Carlo samples are used to compute at least 4 neural networks for the plurality of neural networks. [5] Method according to any one of claims 1 to 4, wherein the multidimensional probability distribution is based on maximum a posterior values ​​of the respective weighting edges and at least on an approximation of a Hessian matrix, wherein the Hessian matrix corresponds to a covariance matrix of the probability distribution, wherein optionally determining the Laplace approximation includes calculating (3) an approximation or factorization of the Hessian matrix. [6] Method according to any one of claims 1 to 5, wherein the plurality of neural networks form an ensemble of neural networks, wherein the uncertainty results from a variance of the respective predictions of the neural networks of the ensemble. [7] Device (13) for processing sensor data of a system, in particular a vehicle, comprising: an interface (14) for receiving sensor data from at least one sensor of the system; a memory (15) on which an ensemble of neural networks is stored, wherein the ensemble of neural networks comprises a plurality of neural networks and wherein the plurality of neural networks are computed based on a Laplace approximation for at least a subset of weight edges of a first neural network; and at least one processor (16), trained to: Processing the sensor data using the ensemble, Calculating a prediction value based on an output from the ensemble's neural networks, and Estimating a value representing the uncertainty of the prediction value based on the output of the ensemble's neural networks. [8] Device according to claim 7, wherein the at least one processor is configured to estimate the value representing the uncertainty of the prediction value based on a variance of the prediction value in the outputs of the ensemble neural networks, and / or wherein the at least one processor is configured to adjust the prediction value if the value representing the uncertainty of the prediction value satisfies a condition, or wherein the at least one processor is configured to output a warning signal if the value representing the uncertainty of the prediction value satisfies a condition. [9] Methods for processing sensor data of a system, in particular of a vehicle, comprising: (7) Obtaining sensor data from at least one sensor of the system; Processing (8) the sensor data using the ensemble of neural networks, wherein the ensemble of neural networks comprises a plurality of neural networks and wherein the plurality of neural networks are computed based on a Laplace approximation for at least a subset of weight edges of a first neural network; Calculating (9) a predictive value based on an output of the ensemble's neural networks, and Estimating (10) a value representing the uncertainty of the prediction value based on the output of the ensemble's neural networks. [10] Computer program comprising instructions which, when the program is executed by a computer or control unit, cause the computer or control unit to execute the method according to any one of claims 1 to 6 or the method according to claim 9.

Images