Device and method for determining an output signal and a determined output signal

Abstract

A computer-implemented method of determining a first element and a second element, the first element characterizing a sensor signal classification or regression result and the second element characterizing a possible classification or regression result confidence interval, the first element and the second element being determined by a premature neural network, the method comprises the steps of: determining, by a premature neural network, a sensor signal feature representation provided to the head of the premature neural network; providing a predicted posteriori distribution or a predicted posteriori distribution maximum independent variable as a first element, wherein the predicted posteriori distribution is determined based on head weight set posteriori distribution and given head weight set feature representation likelihood; distributing the sampling weight set from the head weight set posteriori; determining a possible classification or regression result likelihood ratio by dividing the predicted posteriori distribution value at the feature representation by the feature representation likelihood of the given sampling weight set; the confidence interval is determined as a possible class or a regression result with a likelihood ratio equal to or lower than a predefined threshold and the confidence interval or a value characterizing the width of the confidence interval is provided as a second element.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for determining a classification and / or regression result, a method for training a neural network, a method for using the determined classification and / or regression result for determining a control signal for a technical system, a computer program and a machine-readable storage device. Prior art

[0002] Meronen et al., “Fixing Overconfidence in Dynamic Neural Networks”, 2023, https: / / arxiv.org / abs / 2302.06359 discloses an early-falling neural network that determines its output based on Bayesian inference.

[0003] Waudby-Smith and Ramdas, “Confidence sequences for sampling without replacement”, 2020, https: / / arxiv.org / abs / 2006.04347 discloses a general method for constructing frequentist confidence sequences using Bayesian tools based on the fact that the ratio of the prior to the posterior in the ground truth case is a martingale. Background Art

[0005] Today, many applications rely on using neural networks to predict specific quantities of the real world. Common applications include classifying sensor signals or performing regression on sensor signals in order to determine specific physical aspects of the environment in which the sensor signals were recorded. For example, in robotics, neural networks are heavily used to predict a model of a robot's virtual environment based on sensors such as cameras, lidar, or radar.

[0006] Uncertainty quantification is critical for these applications, especially for safety and reliability in autonomous driving. However, computing uncertainty estimates is often expensive, and this cost is particularly limited in low-resource or latency-critical settings, such as those common for robotics applications. Early exit neural networks (EENNs) provide a partial solution to these problems. As the name implies, these architectures have multiple exits that allow predictions to be generated at arbitrary stopping points along the EENN's architecture. Standard uncertainty quantification techniques, such as Bayesian or conformal reasoning, can then be applied to each exit.

[0007] However, the inventors have found that the known methods result in insufficient uncertainty quantification because in the known methods, the uncertainty for each outlet (or head) of the EENN is modeled as independent, while in fact the uncertainties of adjacent outlets are correlated. For example, the uncertainty interval of the prediction at a given outlet should be consistent with the intervals at the previous and subsequent outlets. If a candidate prediction (classification of the regression result) is in an interval at outlet t-1 and exits the interval at outlet t, then the prediction should not re-enter the interval at outlet t+1 because the prediction has been excluded by another outlet. An even worse case would be that the intervals at time t and t+1 are disjoint sets. This can happen even when using conformal inference, because its coverage guarantee only holds minimally (i.e., on average).

[0008] The present invention proposes a method for treating the uncertainties determined from different heads of an EENN as correlated. Advantageously, this is achieved by treating the prediction uncertainties across the outlets of the EENN as a confidence sequence (AVCS) that is valid at any time. The AVCS was developed for statistical inference in streaming (data) settings and is therefore guaranteed to have non-increasing interval widths. In turn, this allows for consistent uncertainty estimates across the heads of the EENN, which leads to better uncertainty quantification of the EENN concerning its own predictions. Subsequently, this leads to more reliability in the EENN's predictions and therefore to reduced risk when using the EENN for safety-critical applications (such as autonomous driving). Summary of the invention

[0009] In a first aspect, the invention relates to a computer-implemented method for determining a first element and a second element, wherein the first element represents a classification or regression result of a sensor signal and the second element represents a confidence interval of a likely classification or regression result, wherein the first element and the second element are determined by an early falloff neural network, the method comprising the steps of:

[0010] Determining a feature representation of the sensor signal by the early-falling neural network, the feature representation being provided to a head of the early-falling neural network;

[0011] providing as a first element an argument that predicts a posterior distribution or a maximum of a posterior distribution, wherein the posterior distribution is determined based on the posterior distribution of the head's weight set and the likelihood of the feature representation given the head's weight set;

[0012] Sampling weights from the posterior distribution of the head’s weights.

[0013] Determine the likelihood ratio of a possible classification or regression outcome by dividing the value of the predicted posterior distribution at the feature representation by the likelihood of the feature representation given a set of sampling weights;

[0014] • Determine confidence intervals for possible classes or regression outcomes whose likelihood ratios are equal to or below a predefined threshold, and provide the confidence interval or a value characterizing the width of the confidence interval as a second element.

[0015] An early exit neural network (EENN) can generally be understood as determining predictions (e.g., classification or regression results) at various depths (i.e., after a specific layer of the EENN) by having several prediction heads branch out from a shared backbone network. Thus, an EENN can be understood as a sequence of prediction models defined: f(x|W t , U 1：t ): X→Y, t=1,...,T, where W t represents the parameters (also called weights) of the prediction head at the exit t, and U t Identify the parameters of the tth block in the backbone architecture. In other words, all layers of the EENN that do not form at least part of the head can be considered to form at least part of the backbone. The EENN is usually trained by fitting all outlets at once. At test time (also known as inference time), the intermediate predictions of the EENN can be exploited in a variety of ways. For example, if the EENN is considered confident enough at head t, the computation can be stopped without propagating through later blocks, thereby speeding up prediction time.

[0016] The head can be understood as performing a Bayesian linear regression, where the weights are the parameters of the Bayesian linear regression. Therefore, the distribution of weights characterizes the likelihood of possible sets of weights, that is, for each possible set of weights in the Bayesian linear regression, the likelihood can be determined by the distribution. Therefore, sampling weights from the distribution can be understood as sampling a set of weights from the distribution, for example, all the weights necessary to perform the Bayesian linear regression. The regression coefficients (also referred to as (one or more) biases in the field of neural networks) are considered to be part of the weights of the Bayesian linear regression.

[0017] The first element characterizing a classification may be understood as a first element comprising or consisting of one or more values ​​indicating the classification, such as a class index, a class label, and / or a plurality of probabilities or logics for each class in a possible set of classes.

[0018] The first element characterizing the regression result may be understood as a first element comprising or consisting of one or more real values ​​indicative of a characteristic of the sensor signal, in particular a physical characteristic of the input signal. For example, the output signal may be understood as the result of a virtual sensor with an early-degenerate neural network acting as a virtual sensor of a physical property that can be derived from the input signal. An example would be to derive the temperature present in an electric machine (e.g. an electric motor) by means of a sensor signal comprising the current in the electric machine and / or the external temperature of the electric machine.

[0019] The sensor signal can be understood as the result of a measurement performed with the sensor. The measurement can occur just before applying the method (e.g. as part of an online method for classification / performing regression), or the measurement can occur earlier and the sensor signal can be obtained from, for example, a database.

[0020] Determining the feature representation of the sensor signal by the early regression neural network can be understood as providing the sensor signal as input to the EENN and propagating it through one or more layers of the EENN, wherein the layers are configured to each determine a feature representation. One of the feature representations is used as input to the head, i.e., as a feature representation of the sensor signal with respect to the head. Instead of using the sensor signal as input, the sensor signal can also be transformed by one or more preprocessing operations (e.g., smoothing, normalization, extracting a portion from the sensor signal), and then the result of the preprocessing of the sensor signal is provided as input to the EENN.

[0021] Sensor signals can generally be understood as the result of using sensors to measure the real world. EENN can be configured to process specific sensor signals, for example, by training the EENN on other sensor signals similar to the sensor signal. Alternatively, if the EENN is to perform reasoning on pre-processed sensor signals, it can be trained on pre-processed sensor signals.

[0022] The sensor signal may in particular be an image recorded by an optical sensor, such as a camera, a thermal camera, a lidar, a radar or an ultrasonic sensor. Thus, an image may be represented by a collection of low-level features extracted by measurements, such as pixels, depth measurements, voxels, etc.

[0023] The sensor signal may also include multiple measurements from different or similar sensors, for example, multiple images from different cameras are images from at least one camera and at least one lidar. The sensor signal may also include a sequence of measurements, i.e. a time series of measurements.

[0024] Providing an independent variable that predicts the posterior distribution or the maximum value of the posterior distribution can be understood as performing Bayesian inference to determine the predicted posterior distribution, i.e., marginalizing the predicted distribution (i.e., possible classification or possible regression results) on the posterior distribution of the weights of the head, and then providing an independent variable of that distribution or the maximum value of that distribution. Thus, the first element can be a distribution of possible classification or regression values, or a maximum independent variable of such a distribution.

[0025] Preferably, a conjugate prior is chosen for the distribution of the weights, which allows for exact reasoning when determining the predicted posterior distribution (conjugate Bayesian reasoning). This is particularly appropriate when predicting regression results. For predicting classification, approximate methods can be used for Bayesian reasoning, such as the Laplace approximation.

[0026] Determining the likelihood ratio of a possible classification or regression outcome by dividing the value of the predicted posterior distribution at the feature representation by the likelihood of the feature representation for a given set of sampling weights can be implemented as follows:

[0027]

[0028] where y is a possible class or regression result, l is the index of a head among the multiple heads, pl is the likelihood of the predicted posterior distribution of the th head given a training dataset D and an input x to the EENN (i.e., a sensor signal or the result of preprocessing the sensor signal), and p is the probability of the predicted posterior distribution of the th head when using the sampled weight set W l When in the head, the likelihood of predicting y at .

[0029] In the formula, the predicted posterior distribution is defined with respect to the input x. That is, while the distribution is conditional on x, the evaluation of the predicted posterior distribution can be performed, among other things, by propagating the input through the EENN, determining the feature representation to be provided to the ,th head, and then determining the posterior predictive distribution by performing a Bayesian linear regression. The notation above is merely a selection of possible definitions of the predicted posterior distribution. In the formula above, the training dataset Including sensor signal x i and a tuple of the desired classification or regression result yi about the sensor signal. It is also possible to define x as the feature representation of the ,th head and have the dataset consist of the feature representation x i tuple (i.e., determined by forwarding the sensor signals from the original training dataset through the EENN, thereby determining the feature representation of each head and sensor signal) and the corresponding expected classification or regression result y i .

[0030] The predefined threshold is preferably a non-negative number, and preferably not zero.

[0031] Preferably, the early-falloff neural network comprises a plurality of heads, wherein the set of weights for each head is characterized by a posterior distribution of the set of weights, and an order of the plurality of heads is given by their positioning within the early-falloff neural network, and wherein a head precedes at least one other head in the order of the heads, and wherein the likelihood ratio of a possible class or regression outcome is determined by multiplying the likelihood ratio of a class or regression outcome determined for another head by the likelihood ratio determined for the head.

[0032] Advantageously, multiple heads are used in the EENN and the likelihood ratios are multiplied, which the inventors have found to cause the uncertainty represented by a head to incorporate the uncertainty of the prediction about another head or other heads. The authors have advantageously found that in statistical terms, the likelihood ratio can be understood as a martingale, which possesses various advantageous properties. One of these properties is that the sequence of confidence intervals determined by means of the likelihood ratios characterizes a confidence sequence that is valid at any time.

[0033] In other words, if for one of the heads, a possible class or regressor value falls outside the confidence interval of the corresponding head, then due to the multiplication, the possible class or regressor value cannot fall into the confidence interval of the previous head. Therefore, this results in the heads "agreeing" on the confidence interval, i.e. the union of the confidence intervals.

[0034] The inventors have found that taking the first element into account advantageously leads to more accurate uncertainty predictions, i.e., a classification or regression result can be assigned an accurate value of how uncertain the EENN is about its prediction. The uncertainty value can then be used to decide whether the prediction (i.e., the first element) can be trusted.

[0035] Preferably, the likelihood ratio of any head among the plurality of heads is determined according to the following formula:

[0036]

[0037] Where y is the possible class or regression result, l is the index of the head in multiple heads, pl is the likelihood of the predicted posterior distribution of the th head, and p is the probability of using the sampling weight set W. l When in the head, the likelihood of predicting y at, and R t (y) is the likelihood ratio (martingale) of head t.

[0038] The authors found that when predicting classification, R t can be determined for each class in the set of classes that can be predicted by the EENN, where the confidence interval is then determined by the confidence interval for its R t All classes that are less than or equal to a predefined threshold are characterized. The predefined threshold can be considered a hyperparameter of the method. The threshold can advantageously be defined in terms of a reciprocal, for example, characterized by the formula 1 / α. Such a formula allows a straightforward interpretation of the threshold, i.e. the probability that the confidence interval is correct (i.e., the confidence interval includes the correct classification or regression result) is equal to or lower than α. Preferably, α can be defined in terms of probability.

[0039] Preferably, if the first element characterizes the regression result, and the confidence interval is determined by determining the limits of the confidence interval, wherein the limits of the confidence interval are the roots of the function characterized by the following formula:

[0040] logR t(y)-log(1 / α)=0,

[0041] Where 1 / α is a predefined threshold.

[0042] Advantageously, the inventors have found that if, for example, the predicted posterior distribution is Gaussian, then logR t (y) is a quadratic function. The limits of the confidence interval can therefore be found analytically since they are the roots of the formula above.

[0043] Preferably, the prior distribution of the weights of the corresponding head is a conjugate prior. Using a conjugate prior allows finding a closed form solution to the root and is therefore particularly valuable from a computational point of view because it eliminates the additional overhead of iterating over all or a large number of possible regression outcomes.

[0044] Preferably, the head is not the last head of the multiple heads, and wherein the predicted posterior distribution or the maximum value of the predicted posterior distribution is provided as a first element, and if the confidence interval is less than or equal to a predefined threshold but not empty, then the confidence interval or a value characterizing the width of the confidence interval is provided as a second element, and wherein otherwise the predicted posterior distribution or the maximum value of the predicted posterior distribution corresponding to the head following the head is provided as a first element, and a second confidence interval or a value characterizing the width of the second confidence interval corresponding to the head following the head is provided as the confidence interval.

[0045] This can also be understood as the user of the EENN being able to stop calculations at the head, "peek" at the current result, and continue calculating a sufficiently high confidence value about the current prediction. If the confidence value is not high enough, the decision about the prediction can be "postponed" to a later head of the EENN.

[0046] Advantageously, this approach allows for reduced computational complexity since, for early heads in the EENN, most prediction confidences are already high.

[0047] The confidence in the prediction about the head can be derived from the confidence interval. For example, the inverse or negative value of the width of the confidence interval can be understood as the confidence in the prediction. In other words, the narrower the confidence interval, the higher the confidence in the prediction. The case where the confidence interval is empty can be considered a special case because in this case the EENN cannot confidently provide any prediction. In this case, the prediction can be considered rejected from the EENN, i.e., the EENN cannot make a reliable prediction about the class or regression result.

[0048] Alternatively, it is also possible that the head is in the last position within the sequence of heads.

[0049] According to these embodiments, all confidence intervals of the header are combined in order to determine one single confidence interval, wherein the combination may be understood as the union of all confidence intervals.

[0050] Advantageously, the inventors have found that these embodiments include all confidence levels and are therefore the most accurate with respect to the aggregation of decisions made by different people.

[0051] In general, different applications of "peeking" at confidence levels and stopping if the confidence level is high enough or determining all confidence intervals may be weighed against each other depending on available computation and timing constraints and constraints on the size of the confidence intervals.

[0052] For example, if the prediction may not be highly safety critical, but should be delivered in a short amount of time, the threshold may be chosen high so that the prediction is provided from an early head of the EENN. Alternatively, if the prediction is more safety critical, a smaller threshold may be chosen, potentially running the method through all heads of the EENN.

[0053] All or some of the preferred or alternative embodiments presented above may be combined to form further embodiments of the method.

[0054] In another aspect, the present invention relates to a computer-implemented method for determining a third element characterizing a classification or regression result, wherein the method comprises determining a first element and a second element according to any one of claims 1 to 6, wherein if the class characterized by the first element or the regression result characterized by the first element is within a confidence interval characterized by the second element, the first element is provided as the third element, and wherein otherwise a value characterizing a rejection is provided as the third element.

[0055] In another aspect, the invention relates to a method for training an EENN. In particular, the different methods presented above may further comprise determining the posterior distribution of the weights by means of Bayesian inference, in particular conjugate Bayesian inference.

[0056] Conjugate Bayesian inference can be understood as Bayesian inference using a conjugate prior. Advantageously, conjugate Bayesian inference leads to particularly advantageous properties of the determined likelihood ratio. In particular, using conjugate Bayesian inference leads to an exact Bayesian inference of the posterior distribution.

[0057] Preferably, the early-falloff neural network is pre-trained in a pre-training step, and wherein the step of Bayesian inference is performed after the pre-training.

[0058] These embodiments of the method can be understood as first training the EENN, for example using regular gradient descent, and thereby determining the maximum likelihood estimates of the weights of the backbone and head of the EENN, and then discarding the weights of the head and determining the weight distribution of the head with the aid of Bayesian inference.

[0059] Advantageously, this allows lightweight Bayesian inference, i.e., only the heads are equipped with a distribution of their respective weights, rather than having a backbone, and thereby the entire EENN forms a Bayesian neural network. The inventors have found that having the heads perform Bayesian inference is sufficient to allow accurate confidence interval predictions. Therefore, this reduces the computational complexity of the method compared to having the entire EENN as a Bayesian neural network.

[0060] Embodiments of the present invention will be discussed in more detail with reference to the following figures. The figures show:

[0061] Figure 1 is an early exit neural network:

[0062] Figure 2 is a control system comprising an early-decompression neural network for controlling actuators in its environment;

[0063] Figure 3 is a control system for controlling a robot that is at least partially autonomous;

[0064] Figure 4 It is the control system that controls the manufacturing machines;

[0065] Figure 5 It is a control system that controls the imaging system;

[0066] Figure 6 It is a training system used to train classifiers. DETAILED DESCRIPTION

[0067] Figure 1 An embodiment of an early exit neural network (EENN, 60) is shown. A sensor signal (x) is provided as an input to the EENN. In further embodiments (not shown), the input to the EENN (60) may also be the result of preprocessing the sensor signal (x). The description will be made with respect to using the sensor signal (x) as an input to the EENN (60). Figure 1 However, all further details and all implementation choices of the EENN (60) and the method are also valid and possible when using the result of preprocessing the sensor signal (x) as input to the EENN (60).

[0068] The EENN (60) includes multiple blocks (B 1 , B 2 , B T), such as multiple residual blocks. At various points along the EENN (60), a head (h 1 ,h 2 ,h T ), where each head (h 1 ,h 2 ,h T ) is configured to receive feature representations (f 1 , f 2 , f T ), and determine the first element (p 1 , p 2 , p T ) and the second element (c 1 , c 2 , c T ), where the first element (p 1 , p 2 , p T ) represents the classification or regression result of the sensor signal (x), and the second element (c 1 , c 2 , c T ) represents the confidence interval. The first element (p 1 , p 2 , p T ) can be one or more values ​​indicating a class index, a probability of a list of possible classes, or a true value as a regression result. 1 , p 2 , p T ) represents the classification, then the second element (c 1 , c 2 , c T ) can be a set of possible clusters. If the first element represents the regression result, then the second element (c 1 , c 2 , c T ) may include two values ​​indicating the lower and upper limits of the confidence interval. Alternatively or additionally, the second element (c 1 , c 2 , c T ) may also include a single value indicating the width of the confidence interval. 1 , B 2 , B T ) can be considered as the backbone of EENN (60).

[0069] Each head (h 1 ,h 2 ,h T ) receives different feature representations (f 1 , f 2 , f T) as input. Each head receives a different distribution of possible weights (p 1 (W|D), p 2 (W|D), p T (W|D)), given a training dataset D, the different distributions of possible weights (p 1 (W|D), p 2 (W|D), p T (W|D)) can each be considered as a corresponding header (h 1 ,h 2 ,h T ) is the posterior distribution of the weights of the head (h 1 ,h 2 ,h T ) can thus be understood as performing Bayesian inference on the predicted posterior distribution. In other words, each head (h 1 ,h 2 ,h T ) can be thought of as performing Bayesian regression.

[0070] Each head (h 1 ,h 2 ,h T ) is further configured to determine the likelihood ratio (R 1 , R 2 , R T-1 To this end, the head is configured to draw a posterior distribution of possible weights (p 1 (W|D), p 2 (W|D), p T The likelihood ratio is determined by the following formula:

[0071]

[0072] Where y is the possible class or regression result, l is the index of the head in the multiple heads, and p l is the likelihood of the predicted posterior distribution of the lth head, and p is the probability of using the sampling weight set W l The likelihood of predicting y at head l when R t (y) is the likelihood ratio (martingale) of head t.

[0073] In other words, the sensor signal (x) may be passed through the EENN (60), and the head (h 1 ,h 2 ,h T ) can preferably be evaluated according to their order. Each head can preferably calculate its likelihood ratio (R 1 , R 2 , R T-1) is passed to the next head, i.e., the head that follows the head in the head order. In this way, the likelihood ratio (R 1 , R 2 , R T-1 ), because each head (h 1 ,h 2 ,h T ) requires only calculating the fraction in the above formula and multiplying the result by the likelihood ratio received from the previous header to determine the likelihood ratio (R 1 , R 2 , R T-1 ).

[0074] If the head (h 1 ,h 2 ,h T ) is configured for classification, then the confidence interval (c 1 , c 2 , c T ) can preferably be determined by determining the likelihood ratio of each class from the set of possible classes and comparing it to a threshold characterized by the formula 1 / α. Alternatively, if the head (h 1 ,h 2 ,h T ) is configured to determine the regression result, then the confidence interval can be determined in terms of the upper and lower limits of the confidence interval. The upper and lower limits of the confidence interval can be determined as the solution of the following expression:

[0075]

[0076] Preferably, the predicted posterior distribution is Gaussian (e.g., by defining the prior as a conjugate prior), which causes the likelihood ratio formula in y as described above to be quadratic, which in turn allows a closed form solution of the roots of the above formula.

[0077] The inventors have found that determining the likelihood ratio as described above results in 1 ,h 2 ,h T ) to form a (1-α) confidence sequence. In other words, the true class (i.e., the class that the EENN (60) should predict given the sensor signal (x)) or the true regression result (i.e., the regression result that the EENN (60) should predict given the sensor signal (x)) has a probability of (1-α) in all determined confidence intervals. Therefore, this value should be defined in terms of probability values ​​that do not include 0 and 1, that is, all numbers between 0 and 1.

[0078] As shown in the figure, the last head (h T) can be used to provide the output of the EENN (60). In particular, the first element (p T ) and the second element (c T ) can be used as the output of the EENN (60). Alternatively, if the first element (p T ) is less than a predefined threshold (which may in particular be a threshold different from the threshold applied to the likelihood ratio) but is not empty, the EENN (60) may also be configured to output only the first element (p T ). For classification, the width of the confidence interval can be expressed in terms of the number of possible classes in the confidence interval. In general, smaller confidence intervals result in smaller first elements (p 1 , p 2 , p T ), where it is particularly noteworthy that the confidence interval is empty, which indicates that EENN (60) cannot predict the first element (p T ) makes a prediction with sufficient confidence. In this case, the first element can be set to a special value indicating rejection.

[0079] The sensor signal (x) (or the result of preprocessing the sensor signal (x)) may be propagated through the entire EENN (60) to determine the sensor signals from all heads (h 1 ,h 2 ,h T ). In these cases, it is preferred that only the last header (h T ) outputs the first and second elements. In other words, except for the last header (h T ) can preferably be used only to determine the likelihood ratio (R 1 , R 2 , R T-1 ), and then determine the last head (h T )'s first element (p T ) and the second element (c T ).

[0080] Alternatively, it is also possible to iteratively run the calculation in the EENN (60) until the next head (h 1 ,h 2 ,h T ) and "peek" at the confidence interval. If the confidence interval is as wide as or narrower than a predefined threshold, the first element of the current head can be output and further calculation of the sensor signal (x) can be terminated, i.e., the sensor signal (x) is further propagated through the EENN (60).

[0081] The described thresholds can all be understood as hyperparameters of the method.

[0082] Figure 2 An embodiment of a control system (40) using an EENN (60) for determining a control signal (A) for an actuator (10) or a display is shown. The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably evenly spaced points in time, a sensor (30) senses a condition of the actuator system. The sensor (30) may include several sensors. Preferably, the sensor (30) is an optical sensor that captures an image of the environment (20). An output signal (S) of the sensor (30) (or an output signal (S) of each of the sensors, if the sensor (30) includes multiple sensors) encoding the sensed condition is transmitted to the control system (40).

[0083] Thus, the control system (40) receives the sensor signal stream. It then calculates a series of control signals (A) depending on the sensor signal stream (S), which are then transmitted to the actuator (10).

[0084] The control system (40) receives a sensor signal stream of a sensor (30) in an optional receiving unit (50). The receiving unit (50) preprocesses the sensor signal (S) into an input signal (x). Alternatively, in the absence of a receiving unit (50), each sensor signal can be taken directly as an input signal (x). The input signal (x) can be given, for example, as an extract from a sensor signal. Alternatively, the sensor signal can be processed to generate the input signal (x). In other words, the input signal (x) is provided based on the sensor signal.

[0085] The input signal (x) is then passed to the EENN (60).

[0086] The EENN (60) is parameterized by parameters (Φ, e.g., the parameters of the backbone and the posterior distribution of the weights of the head) which are stored in a parameter storage device (St 1 ) and provided by it.

[0087] The EENN (60) determines a first element (p) and a second element (c) from an input signal (x). The first element (p) and the second element (c) are transmitted to a conversion unit (80) which converts the first element (p) and the second element (c) into a control signal (A). For example, the first element (p) and the second element (c) may be outputs of the last head of the EENN (60). In this case, if the width of the confidence interval represented by the second element (c) is equal to or narrower than a predefined threshold, the first element (p) may be used to automatically determine the control signal (A). Otherwise, the control signal (A) may be selected, for example, to hand over the operation of the actuator to a human operator. In a further embodiment (indicated by a dashed arrow), the first element (p) may be outputs of a head other than the last head of the EENN (60). If the width of the confidence interval represented by the second element (c) is wider than the predefined threshold, the EENN (60) may be instructed to continue determining the first element and the second element of the next head. The method can be run iteratively until the confidence interval is narrow enough, or until the first element and the second element have been determined by the last head of the EENN (60). If the confidence interval is still not narrow enough, the operation can be handed over to a human operator again.

[0088] In all the embodiments described above, it is assumed that an empty confidence interval is also considered as a confidence interval that is not narrow enough, because in such a case the prediction about the input signal (x) is rejected by the EENN (60). The confidence of the prediction of the first element (p) can be understood as corresponding to the width of the confidence interval characterized by the second element (c), for example, the confidence can be the inverse of the width of the confidence interval, again being a special case of an empty confidence interval. Insufficient confidence can be for the case where its corresponding confidence interval is not as wide as a predefined threshold or is narrower than a predefined threshold.

[0089] The control signal (A) is then transmitted to the actuator (10) to control the actuator (10) accordingly.

[0090] The actuator (10) receives the control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that converts the control signal (A) into a further control signal, which is then used to control the actuator (10).

[0091] In a further embodiment, the control system (40) may include a sensor (30). In still further embodiments, the control system (40) may alternatively or additionally include an actuator (10).

[0092] In still further embodiments, it is contemplated that the control system (40) controls the display (10a) instead of or in addition to the actuator (10). For example, the confidence of the current prediction of the EENN (60) may be output on the display (10a).

[0093] Still further, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) having stored thereon instructions which, if executed, cause the control system (40) to perform a method according to aspects of the present invention.

[0094] Figure 3 An embodiment is shown in which a control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (100).

[0095] The sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors. Some or all of these sensors are preferably but not necessarily integrated in the vehicle (100). Therefore, the input signal (x) can be understood as an input image, and the EENN (60) can be understood as an image classifier.

[0096] The EENN (60) may be configured to detect objects in the vicinity of the at least partially autonomous robot based on an input image (x), i.e., to perform object detection. The first element (p) may include information characterizing where the object is located in the vicinity of the at least partially autonomous robot. A control signal (A) may then be determined based on this information, for example to avoid a collision with the detected object.

[0097] The actuator (10), preferably integrated in the vehicle (100), may be given by the brakes, propulsion system, engine, drive train or steering of the vehicle (100). The control signal (A) may be determined so that the actuator (10) is controlled so that the vehicle (100) avoids a collision with a detected object. The detected objects may also be classified according to what the EENN (60) thinks they are most likely to be, such as a pedestrian or a tree, and the control signal (A) may be determined depending on the classification.

[0098] If the second element (c) determined by the EENN (60) characterizes insufficient confidence in the first element (p), operation of the vehicle (100) may be handed over to a human driver or external operator. Preferably, the handover is performed for situations where the confidence has been insufficient for a predetermined amount of time or prediction of the EENN (60). Alternatively, the vehicle (100) may also perform an emergency maneuver (e.g., emergency braking, moving to an emergency lane, deceleration, etc.) if the confidence is insufficient or has been insufficient for a predefined amount of time or prediction of the EENN (60).

[0099] Alternatively or additionally, the control signal (A) may also be used to control a display (10a), for example to display objects detected by the EENN (60). It is also conceivable that the control signal (A) may control the display (10a) so that the display (10a) generates a warning signal if the vehicle (100) is close to colliding with at least one of the detected objects. The warning signal may be a warning sound and / or a tactile signal, such as vibration of the steering wheel of the vehicle.

[0100] In a further embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. The mobile robot may in particular be an at least partially autonomous lawnmower, or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined such that a propulsion unit and / or a steering and / or a braking of the mobile robot is controlled such that the mobile robot may avoid a collision with the identified object.

[0101] In a further embodiment, the at least partially autonomous robot may be provided by a gardening robot (not shown) which uses sensors (30), preferably optical sensors, to determine the state of plants in the environment (20). The actuator (10) may control a nozzle for spraying a liquid and / or a cutting device, such as a blade. Depending on the identified type of plant and / or the identified state, a control signal (A) may be determined to cause the actuator (10) to spray the plant with an appropriate amount of an appropriate liquid and / or to cut the plant.

[0102] In a further embodiment, the at least partially autonomous robot may be provided by a household appliance (not shown), such as a washing machine, a stove, an oven, a microwave or a dishwasher. A sensor (30), such as an optical sensor, may detect the state of an object to be subjected to a process by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor (30) may detect the state of the laundry inside the washing machine. The control signal (A) may then be determined depending on the material of the detected laundry.

[0103] Figure 4An embodiment is shown in which a control system (40) is used to control a manufacturing machine (11), such as a punch, cutter, gun drill or fixture, of a manufacturing system (200) as part of a production line. The manufacturing machine may include a transport device, such as a conveyor belt or an assembly line, to move the manufactured products (12). The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).

[0104] The sensor (30) may be given by an optical sensor that captures, for example, properties of the manufactured product (12). Hence, the EENN (60) may be understood as an image classifier.

[0105] The EENN (60) may determine the positioning of the manufactured product (12) with respect to the transport device. The actuator (10) may then be controlled depending on the determined positioning of the manufactured product (12) for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) may be controlled to cut the manufactured product at a specific location on the manufactured product itself. Alternatively, it is conceivable that the EENN (60) classifies whether the manufactured product is broken and / or exhibits defects. The actuator (10) may then be controlled to remove the manufactured product from the transport device. The manufactured product may also be removed from the transport device if the confidence of the EENN (60) is insufficient or a predefined amount of time or prediction for the EENN (60) has been insufficient.

[0106] Figure 5 An embodiment of a medical imaging system (500) controlled by a control system (40) is shown. The imaging system may be, for example, an MRI device, an x-ray imaging device or an ultrasound imaging device. The sensor (30) may be, for example, an imaging sensor that captures at least one image of a patient, for example showing different types of body tissue of the patient.

[0107] The EENN (60) may then determine a classification of at least a portion of the sensed image. Thus, at least a portion of the image is used as an input image (x) to the EENN (60). Thus, the EENN (60) may be understood as an image classifier.

[0108] A control signal (A) may then be selected based on the classification, thereby controlling the display (10a). For example, the EENN (60) may be configured to detect different types of tissue in the sensed image, for example by classifying the tissue shown in the image as malignant or benign tissue. This may be done with the aid of a semantic segmentation of the input image (x) performed by the EENN (60). The control signal (A) may then be determined so that the display (10a) displays different tissues, for example by displaying the input image (x) and coloring different areas of the same tissue type with the same color. If the confidence in the classification is insufficient, the display (10a) may be controlled so as to highlight areas where the confidence is insufficient.

[0109] In further embodiments (not shown), the imaging system (500) may be used for non-medical purposes, for example, determining material properties of a workpiece. In these embodiments, the EENN (60) may be configured to receive an input image (x) of at least a portion of the workpiece and perform semantic segmentation of the input image (x) to thereby classify the material properties of the workpiece. A control signal (A) may then be determined to cause the display (10a) to display the input image (x) and information about the detected material properties. If the confidence in the classification is insufficient, the display (10a) may be controlled to highlight areas of insufficient confidence.

[0110] Figure 6 An embodiment of a training system (140) for pre-training or training an EENN (60) of a control system (40) with the aid of a training data set (T) is shown. The training data set (T) includes a plurality of input signals (x i ), where for each input signal (x i ), the training data set (T) further includes the corresponding input signal (x i ) and characterizes the input signal (x i ) is the expected classification or expected regression result of the expected output signal (t i ).

[0111] For pre-training, the weights of the head of the EENN (60) are selected as point estimates (ie, not distributions). The training data unit (150) accesses a computer-implemented database (St 2 ), the database (St 2 ) provides a training data set (T). The training data unit (150) preferably randomly determines at least one input signal (x) from the training data set (T). i ) and corresponds to the input signal (x i ) of the expected output signal (t i ), and the input signal (x i) is transmitted to the EENN (60). The EENN (60) is based on the input signal (x i ) determines the output signal (y i ).

[0112] The expected output signal (t i ) and the determined output signal (y i ) is transmitted to the modification unit (180).

[0113] Based on the expected output signal (t i ) and the determined output signal (y i ), the modification unit (180) then determines new parameters (Φ′) for the EENN (60). For this purpose, the modification unit (180) uses a loss function to compare the desired output signal (t i ) and the determined output signal (y i The loss function determines a first loss value, which characterizes the determined output signal (y i ) from the desired output signal (t i ) deviates far. In a given embodiment, a negative log-likelihood function is used as a loss function. In alternative embodiments, other loss functions are also conceivable.

[0114] Furthermore, it is conceivable that the determined output signal (y i ) and the desired output signal (t i ) each includes multiple sub-signals, such as sub-signals in the form of tensors, where the desired output signal (t i ) corresponds to the determined output signal (y i For example, it is conceivable that the EENN (60) is configured for object detection, and the first sub-signal characterizes the object with respect to the input signal (x i ), and the second sub-signal characterizes the exact location of the object. If the determined output signal (y i ) and the desired output signal (t i ) includes multiple corresponding sub-signals, then a second loss value is preferably determined for each corresponding sub-signal with the aid of a suitable loss function, and the determined second loss values ​​are suitably combined (for example, with the aid of a weighted sum) to form the first loss value.

[0115] The modification unit (180) determines new parameters (Φ') based on the first loss value. In a given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam or AdamW. In further embodiments, the training can also be based on an evolutionary algorithm or a second order method for training neural networks.

[0116] In other preferred embodiments, the described training is iteratively repeated for a predefined number of iteration steps, or iteratively repeated until the first loss value drops below a predefined threshold. Alternatively or additionally, it is also conceivable that the training is terminated when the average first loss value with respect to the test or validation data set drops below a predefined threshold. In at least one of the iterations, the new parameters (Φ′) determined in the previous iteration are used as parameters (Φ) of the EENN (60).

[0117] For training, the training dataset (T) can then be used to perform Bayesian inference on the weights of the corresponding heads of the EENN (60). To this end, the weights of the backbone of the EENN (60) can preferably be fixed to the values ​​determined during pre-training, and the weights of the heads of the EENN (60) can be discarded. For each training input signal (x i ), the head is then provided with a corresponding feature representation from its backbone. Based on the feature representation, a posterior distribution of weights is then determined for each head using Bayesian inference, preferably conjugate Bayesian inference.

[0118] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions, which, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

[0119] The term "computer" may be understood to cover any device for the processing of predefined computing rules. These computing rules may be in the form of software, hardware or a mixture of software and hardware.

[0120] In general, a plurality may be understood to be indexed, i.e. each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality comprises N elements, where N is the number of elements in the plurality, the elements are assigned integers from 1 to N. It may also be understood that the elements in the plurality may be accessed by their index.

Claims

1. Used to determine the first element (p1, p2, p T ) and the second element (c1, c2, c T ), wherein the first element (p1, p2, p T ) represents the classification or regression result of the sensor signal (x), and the second element (c1, c2, c T ) represents a confidence interval for a likely classification or regression result, where the first element (p1, p2, p T ) and the second element (c1, c2, c T ) is determined by the early-falling neural network (60), the method comprising the following steps: Determine the characteristic representation (f1, f2, f T ), the characteristic representation of the sensor signal (x) (f1, f2, f T ) is provided to the head (h1, h2, h T ); An independent variable providing as the first element (p1) a predicted posterior distribution or a maximum value of a predicted posterior distribution, wherein the predicted posterior distribution is based on the head (h1, h2, h T )’s posterior distribution of the weight set (p1(W|D), p2(WID), p T (W|D)) and given headers (h1, h2, h T ) is determined by the likelihood of the feature representation of the weight set; · From the head (h1, h2, h T )’s posterior distribution of the weight set (p1(W|D), p2(W|D), p T (W|D)) in the sampling weight set; By representing the features (f1, f2, f T ) is divided by the likelihood of the feature representation given the sampled weight set to determine the likelihood ratio (R1, R2, R T-1 ); Determine a confidence interval for possible classes or regression outcomes whose likelihood ratios are equal to or below a predefined threshold, and provide the confidence interval or a value characterizing the width of the confidence interval as the second element (c1, c2, c T ).

2. The method according to claim 1, wherein the early-retirement neural network (60) comprises a plurality of heads (h1, h2, h T ), where each header (h1, h2, h T ) is composed of the posterior distribution of the weight set (p1(W|D), p2(W|D), p T (W|D)) characterizes, and the multiple heads (h1, h2, h T ) is given by their positioning within the early exit neural network (60), and wherein the heads (h1, h2, h T ) in the header (h1, h2, h T ) before at least one other head in the order of , and the likelihood ratios of possible classes or regression results (R1, R2, R T-1 ) by multiplying the likelihood ratios of the classes or regression results determined for the other heads for the head (h1, h2, h T ) is determined by determining the likelihood ratio.

3. The method according to claim 2, wherein any head (h1, h2, h T ) likelihood ratio (R1, R2, R T-1 ): where y is the possible class or regression result, l is the index of the head in the multiple heads, pl is the likelihood of the predicted posterior distribution of the th head given the training dataset D and the input x to the EENN, and p is the probability of the predicted posterior distribution of the th head when using the sampled weight set W l The likelihood of predicting y at head l when .

4. The method according to claim 3, wherein the first element (p1, p2, p T ) characterizes the regression results, and determines the confidence interval by determining the limits of the confidence interval, wherein the limits of the confidence interval are the roots of the function characterized by the following formula: log R t (y)-log(1 / α)=0, where 1 / α is a predefined threshold.

5. The method according to any one of claims 2 to 4, wherein the header (h1, h2, h T ) is not the last head of the plurality of heads, and wherein the predicted posterior distribution or the maximum value of the predicted posterior distribution is provided as the first element (p1, p2, p T ), and if the confidence interval is less than or equal to the predefined threshold but not empty, the confidence interval or a value characterizing the width of the confidence interval is provided as the second element (c1, c2, c T ), and where otherwise corresponds to the following head (h1, h2, h T ) is provided as the first element, and the second confidence interval or characterization corresponds to the following head (h1, h2, h T )’s head is provided as the confidence interval.

6. The method according to any one of claims 2 to 5, wherein the header (h1, h2, h T ) at the last position within the sequence of the header.

7. A computer-implemented method for determining a third element characterizing a classification or regression result, wherein the method comprises determining a first element (p1, p2, p3) according to any one of claims 1 to 6 T ) and the second element (c1, c2, c T ), where if the first element (p1, p2, p T ) or by the first element (p1, p2, p T ) represents the regression result represented by the second element (c1, c2, c T ) is within the confidence interval characterized by the rejection, the first element (p1, p2, pT) is provided as the third element, and wherein otherwise the value representing the rejection is provided as the third element.

8. The method according to any one of claims 1 to 7, wherein the method further comprises determining the posterior distribution of the weights p1(WID), p2(W|D), pT(W|D) by means of Bayesian inference, in particular conjugate Bayesian inference.

9. The method according to claim 8, wherein the early-falloff neural network (60) is pre-trained in a pre-training step, and wherein the steps of claim 8 are performed after the pre-training.

10. A training system (140) configured to implement the training method according to claim 8 or 9.

11. A control system (40) configured to implement the method according to any one of claims 1 to 7, wherein the control system (40) is based on a first element (p1, p2, p T ) and the second element (c1, c2, c T ) or determining a control signal (A) based on a third element, wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).

12. Computer program configured to cause a computer to carry out the method according to any one of claims 1 to 9 and all its steps, if the computer program is executed by a processor (45, 145).

13. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 12.

Images