Train the trainable module using learning data whose labels are noisy

Abstract

A method (100) for training a trainable module (1) comprises the following steps: • pre-training (110) of a plurality of modification processes (1a‑lc) of the trainable module (1) using at least a subset of a learning data set (2), the modification processes (1a‑lc) being so different from one another that they are not identically transformed into one another during learning; • learning input variable values ​​(11a) of at least one learning data set (2) are fed (120) as input variables (11) to all modification processes (1a‑lc); • determining, from the deviations between the output variable values ​​(13), the output variable values ​​for the input variables (13) A method (200) is provided in which a trainable module is operated (220) and the trainable module controls the system (50, 60, 70) using the control signal (5).

Description

Technical Field

[0001] The present invention relates to the training of trainable modules, such as are used, for example, for classification tasks and / or object recognition in at least partially automated driving. Background Art

[0002] Typically, human drivers are trained to drive a vehicle in road traffic by repeatedly subjecting the driver's trainee to specific canons for various situations within their training. The driver must respond to each of these situations and, through comments or even intervention from the driving instructor, receive feedback on whether their response was correct or incorrect. Training with a limited number of situations is crucial to enable the driver's trainee to master unfamiliar situations even when driving independently.

[0003] In order to allow vehicles to participate in road traffic fully or partially automatically, the goal is to control the vehicles using modules that are trainable in a completely similar manner. For example, these modules receive sensor data from the vehicle's surroundings as input variables and, as output variables, supply control signals for intervening in vehicle operation and / or supply primary products from which such control signals are generated. For example, the classification of objects in the vehicle's surroundings can be such a primary product.

[0004] For training, a sufficient number of learning data sets are required, each of which includes learning input variable values ​​and corresponding learning output variable values. For example, the learning input variable values ​​may include images, and information about which objects are contained in the images may be used as labels for the learning output variable values. Summary of the Invention

[0005] Within the scope of the present invention, a method has been developed for training a trainable module. A trainable module transforms one or more input variables into one or more output variables.

[0006] A trainable module is particularly defined as a module that embodies a function parameterized by adaptable parameters with high generalization power. During training of a trainable module, the parameters can be adapted in particular so that, when learning input variable values ​​are input into the module, the associated learning output variable values ​​are reproduced as accurately as possible. A trainable module can particularly comprise an artificial neural network (KNN) and / or can be a KNN.

[0007] The training is performed using a learning data set comprising learning input variable values ​​and associated learning output variable values. At least the learning input variable values ​​include measurement data, which have been obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a technical system observable using such a measurement process.

[0008] The term "learning dataset" does not denote the entirety of all available learning data, but rather the combination of one or more learning input variable values ​​and learning output variable values ​​that are assigned to these learning input variable values ​​as labels. In a trainable module used for classification and / or regression, the learning dataset may, for example, include images as a matrix of learning input variable values ​​in combination with softmax scores (scores) as a vector of learning output variable values, from which the trainable module ideally generates the softmax scores.

[0009] Within the scope of the present method, multiple modification processes of a trainable module are pretrained using at least a subset of the learning dataset. The modification processes differ so much that they are not identically transformed into one another during the learning process. For example, the modification processes can be structurally different. For example, multiple modification processes of a KNN can be generated by deactivating different neurons within the context of "dropouts." However, modification processes can also be generated, for example, by pretraining using sufficiently different subsets of the total available learning dataset and / or by pretraining from sufficiently different initializations.

[0010] For example, these modification processes can be pre-trained independently of one another. However, it is also possible, for example, to bundle the pre-training by training only a trainable module or modification process and only after the training is complete, to generate further modification processes from this module or this modification process.

[0011] After pre-training, learning input variable values ​​of at least one learning dataset are fed as input variables to all modification processes. These same learning input variable values ​​are converted into different output variable values ​​by different modification processes. Uncertainty measures for these output variable values ​​are determined from the deviations between these output variable values, and these uncertainty measures are assigned to the learning dataset as uncertainty measures for the learning dataset.

[0012] In this case, the output variable value can be, for example, a Softmax score which indicates with what probability the learning data set was classified into which of the possible classes.

[0013] Any statistical function can be used to determine uncertainty from a large number of output variable values. Examples of such statistical functions are variance, standard deviation, mean, median, appropriately chosen quantiles, entropy, and variation ratio.

[0014] If the modification processes of the trainable module have been generated in different ways, for example, on the one hand by "dropout" and on the other hand by another structural change or by another initialization of the pretraining, it is particularly possible, for example, to compare the deviations between the output variable values ​​supplied by the modification processes generated in different ways separately. In other words, for example, the deviations between the output variable values ​​supplied by the modification process generated by "dropout" and the deviations between the output variable values ​​supplied by the modification process that was structurally modified in another way can be observed separately from each other.

[0015] In this context, the terms "bias" and "uncertainty" are not limited to the one-dimensional, univariate case, but encompass variables of any dimension. That is, for example, multiple uncertainty features can be combined to obtain multivariate uncertainty. This improves the accuracy of distinguishing between learning datasets with a correct assignment of learning output variable values ​​to learning input variable values ​​(i.e., "correctly labeled" learning datasets), on the one hand, and learning datasets with an incorrect assignment (i.e., "incorrectly labeled" learning datasets), on the other.

[0016] Now, an evaluation of the learning data set is determined in terms of uncertainty, which is a measure of how correctly the values ​​of the learning output variables are assigned to the values ​​of the learning input variables in the learning data set.

[0017] It has been found that, when the values ​​of the learned output variables are correctly assigned to the values ​​of the learned input variables, different modification processes of the trainable module tend to have consistent "opinions" about the output of the output variables. The information inherent in the correct assignment appears to be captured during pre-training and has the effect that differences between the modification processes are rarely or even absent in the different output variables. The more incorrect the assignment, the greater the lack of this precise effect, and the greater the deviations between the output variable values ​​that were supplied to the modification processes for the same learning input variable values.

[0018] If all learning data sets are analyzed in this way, it will typically be found that the assignments for some learning data sets are more correct than for others. This primarily reflects the fact that in most applications of trainable modules, assignment (i.e., labeling) is performed by humans and is accordingly prone to error. For example, to achieve a high throughput for each learning data set, a human may only be given very little time, so that in doubtful cases, they cannot conduct a more precise investigation and must instead make a decision. For example, different staff members may interpret the criteria used to label images differently. For example, if an object casts a shadow in an image, a staff member may include the shadow as an object because it is indeed caused by the object's presence. However, another staff member may not include the shadow as an object for the following reason: the shadow is not something that a person or vehicle could collide with.

[0019] A final useful application of the determined evaluation is the ability to selectively implement measures to improve the final training of the trainable module. For example, the trained module can then perform classification and / or regression with greater accuracy on the measurement data supplied as input variables. This allows, in corresponding technical applications, such as, for example, in at least partially automated driving, decisions appropriate to the situation to be made with a higher probability based on the given measurement data.

[0020] In a particularly advantageous embodiment, the adaptable parameters that characterize the properties of the trainable module are optimized with the goal of improving the value of the cost function. In KNN, these parameters include, for example, weights, with which the inputs supplied to a neuron are calculated in order to activate the neuron. The cost function measures the extent to which the trainable module maps the values ​​of the learning input variables contained in the learning data set to the associated learning output variable values. During conventional training of the trainable module, all learning data sets are equal in this respect, i.e., the cost function measures how well the learning output variable values ​​are reproduced on average. The determined evaluation is incorporated into the process as follows: the weighting of at least one learning data set in the cost function is dependent on the evaluation of this learning data set.

[0021] For example, the worse a learning dataset is evaluated, the less weight it can be assigned. This can continue until, in response to the evaluation of a learning dataset meeting a predetermined criterion, the learning dataset is completely excluded from the cost function, meaning it is no longer used for further training of the trainable module. The underlying cognitive rationale is that inconsistencies arising from inaccurate or erroneous learning output values ​​during the training process can be fully or partially compensated (or even overcompensated) by considering the additional benefits provided by other learning datasets. In other words, no information is better than erroneous information.

[0022] In another particularly advantageous embodiment, an update of at least one learning output variable value contained in the learning dataset can be requested in response to the evaluation of the learning dataset meeting a predetermined criterion. For example, the criterion may be that the evaluation of the learning dataset falls below a predetermined minimum criterion and / or is particularly poor compared to other learning datasets. The requested update can be performed, for example, by a human expert or retrieved via a network. This is based on the following cognitive intervention: many errors that occur during labeling are single errors, such as volatile errors. However, the need for an update may arise, for example, in situations where there are simply not enough examples in the learning dataset to train reliable recognition of a specific object. For example, a specific traffic sign (such as sign 129, "Dyke") occurs relatively rarely and may be underrepresented in the images recorded during the test drive. The requested update then appears to provide guidance to the trainable module at this point.

[0023] In a particularly advantageous embodiment, an uncertainty distribution is determined based on a large number of learning data sets. Based on this distribution, an evaluation of a specific learning data set is determined. In this distribution, information from the large number of learning data sets is aggregated, allowing a decision on the evaluation of a specific learning data set to be made with greater accuracy.

[0024] In a particularly advantageous embodiment, the distribution is modeled as a superposition of multiple parameterized contributions, each originating from a learning dataset with the same or similar evaluations. The parameters of these contributions are optimized such that the deviation of the observed uncertainty distribution from the superposition is minimized. In this way, the contributions are determined.

[0025] There is freedom here as to the type of superposition. For example, the superposition can be additive. However, the superposition can also consist in selecting the respective highest value of the different contributions for each value of the uncertainty.

[0026] For example, the distribution can be modeled as a superposition of contributions from correctly labeled learning datasets ("clean labels") and contributions from incorrectly labeled learning datasets ("noisy labels"). However, it is also possible to introduce additional contributions for learning datasets whose labels are moderately reliable, for example.

[0027] By means of modelling, in particular the following additional information can be taken into account: which function rule respectively characterizes the distribution of the individual contributions. After the parameters of the contributions are determined and the contributions are therefore fixed as a whole, the contributions can be used, for example, to evaluate a specific learning data set. In a particularly advantageous embodiment, the evaluation of at least one learning data set is thus determined on the basis of local probability densities and / or on the basis of ratios of such local probability densities, if the uncertainty of the learning data set is fed as input to at least one contribution to the superposition, which then outputs said local probability density. For example, the distribution can be modelled by a superposition consisting of a first contribution, which represents a correctly labelled ("clean") learning data set, and a second contribution, which represents an incorrectly labelled ("noisy") learning data set. When the uncertainty u is input, the first contribution then supplies the probability p for the correctly labelled learning data set. c (u). Given the input uncertainty u, the second contribution supplies the probability p for the incorrectly labeled learning dataset n (u).

[0028] Furthermore, the chance (odds ratio) r of the learning dataset being incorrectly labeled compared to being correctly labeled can be determined. For example, one can follow the rule

[0029]

[0030] To determine the odds ratio r. From the odds ratio r or also from p n (u) and p c If the ratio of ΔH to ΔH (u) exceeds a certain value, it can now be determined that the learning data set is an incorrectly labeled (“noisy”) learning data set.

[0031] Alternatively, or in conjunction therewith, the evaluation of at least one learning dataset can also include the following: to which contribution the learning dataset is assigned when optimizing the parameters of the contribution. A specific algorithm for optimizing the parameters (e.g., an expectation maximization algorithm) immediately returns which learning datasets were used to fit these contributions to the distribution. In the example described above, for example, the portion of the learning dataset that was used to fit a second contribution representing the incorrectly labeled learning dataset to the distribution can be evaluated as an estimate of the incorrectly labeled learning dataset for that portion.

[0032] Similarly, for example, during pre-training, for example, at every nth epoch (epoch), it can be observed whether the learning dataset has been used to fit the first contribution representing correctly labeled learning datasets, or the second contribution representing incorrectly labeled learning datasets. This distribution can vary from epoch to epoch. At the end of pre-training, if the learning dataset has been classified as incorrectly labeled in the majority of the investigated epochs, then the learning dataset can be classified as incorrectly labeled, for example.

[0033] However, it is also possible to read out other information at the contribution, which characterizes the entirety of the learning data set analyzed in the distribution. Thus, in a particularly advantageous embodiment, it is at least determined, based on the deviation of the distribution from the superposition, whether essentially only learning data sets with identical or similar evaluations contribute to the distribution. For example, in this way it is possible to test whether essentially only correctly labeled learning data sets are present, or whether incorrectly labeled learning data sets are also present, with respect to which one or more of the described selective measures can also be taken. This means that the test can, for example, be used as a termination criterion for such selective measures.

[0034] If, for example, a scheme with two parameterized contributions is used for superposition, then the superposition is more or less forced to include both contributions, depending on the specific algorithm used to optimize the parameters. However, if both contributions are not actually present in the distribution, for example because essentially all learning data sets are correctly labeled, then the deviation between the superposition and the distribution will be relatively large even after the optimization is terminated. The actual distribution of uncertainty is then centered around a relatively low value, while the superposition seeks a second such center. It is then no longer sensible to "relabel" the other learning data sets by updating the values ​​of the learning output variables or to underweight them in the cost function used to train the trainable module.

[0035] For example, statistical tests can be used to determine whether essentially only learning data sets with identical or similar evaluations contribute to the distribution. Such tests check whether the sampled basic data follow a predetermined distribution or whether the determined superposition is consistent with the learning data set. Examples of this are the Shapiro-Wilk test (for a normal distribution) and the Kolmogorov-Smirnov test. Alternatively or in combination with this, for example, a visual plot of the deviation between the distribution and the superposition (such as a QQ plot) can also be converted into a metric. In the case of a QQ plot, for example, the mean deviation from the diagonal can be used for this purpose.

[0036] In another particularly advantageous embodiment, the different contributions to the superposition are modeled using the same parameterized function, but with independent parameters. No single contribution then stands out relative to the others, so that which learning data set is assigned to which contribution depends solely on the resulting statistics across all learning data sets.

[0037] Examples of parameterized functions that can be used to model the contributions include statistical distributions, in particular distributions from the exponential family, such as the normal, exponential, gamma, chi-quadrat, beta, exponential Weibull, and Dirichlet distributions. It is particularly advantageous if the function has a vector (non-zero set) in the interval [0, 1] or (0, 1), since several possibilities used to calculate uncertainty (e.g., the mean value of the softmax scores) provide values ​​in the interval (0, 1). The beta distribution is an example of a function with such a vector.

[0038] The parameters of the contribution can be optimized, for example, using likelihood methods and / or Bayesian methods, in particular using an expectation-maximization algorithm, an expectation / conditional-maximization algorithm, an expectation-conjugate-gradient algorithm, a Riemann batch algorithm, a Newton-based method (e.g., Newton-Ralphson), a Markov Chain Monte Carlo-based method (e.g., Gibbs sampling or Metropolis-Hastings algorithm), and / or a stochastic gradient algorithm. In this case, the expectation-maximization algorithm is particularly suitable. As explained above, this algorithm directly provides information about which learning data sets have been used to fit the contribution to the distribution. The Riemann batch algorithm is described in more detail in arXiv: 1706.03267.

[0039] In another particularly advantageous embodiment, the Kullback-Leibler divergence, the Hellinger distance, the Lévy distance, the Lévy-Prochorow metric, the Wasserstein metric, the Jensen-Shannon divergence, and / or another scalar measure of how different the contributions are from one another is determined from the modeled contributions. This makes it possible to assess how clearly the different contributions are separated from one another.

[0040] Furthermore, the scalar metric can be used to optimize the duration of the pre-training of the modification process. Thus, in a further particularly advantageous embodiment, a correlation of the scalar metric with a number of epochs of the pre-training of the modification process and / or with a number of training steps of the pre-training of the modification process is determined.

[0041] For example, there may be a tendency for the uncertainty distribution to be initially divided into multiple contributions during pretraining, but this division may be partially eliminated as pretraining progresses. As explained above, incorrectly labeled learning datasets lead to inconsistencies in pretraining. Pretraining may attempt to resolve these inconsistencies using a "compromise." The distinction between correctly labeled and incorrectly labeled learning datasets is most pronounced before this process is implemented.

[0042] Thus, in further particularly advantageous embodiments, a number of epochs and / or a number of training steps is taken into account for further determining the uncertainty of the learning data set, in which number of epochs and / or a number of training steps the scalar measure indicates the maximum difference in the contribution to the superposition.

[0043] The present invention also relates to a further method that continues the training chain by running a trained trainable module. In this method, a trainable module is first trained using the method described above, which converts one or more input variables into one or more output variables. Subsequently, the trainable module is run by supplying the input variable values ​​to the trainable module.

[0044] In this case, the input variable values ​​include measurement data which have been obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a technical system observable using such a measurement process.

[0045] The trainable module converts input variable values ​​into output variable values. Based on these output variable values, a control signal is used to control a vehicle, a classification system, a system for quality control of serially manufactured products, and / or a system for performing medical imaging.

[0046] For example, the trainable module can provide semantic segmentation of images from the vehicle's surroundings. This semantic segmentation classifies image pixels by the type of object they belong to. Based on this semantic segmentation, the vehicle can be controlled to ensure it moves only within freely traversable areas and avoid collisions with other objects, such as lane boundaries, buildings, or other road users.

[0047] For example, within the scope of quality control, the trainable module can classify samples of a specific product into two or more quality classes based on physical measurement data. Depending on the quality class, the specific sample can then be marked, for example, or a screening device can be operated to separate it from other samples with different quality classes.

[0048] For example, within the context of medical imaging, a trainable module can classify whether a recorded image indicates a specific pathological condition and, if necessary, the degree to which manifestations of the disease are present. Based on the results of this classification, the physical process of image recording can be adapted, for example, so that, based on further recorded images, it is possible to even more clearly distinguish whether the corresponding pathological condition is present. For example, the focus or lighting of a camera-based imaging system can be adapted.

[0049] In the field of medical imaging, labeling—that is, assigning the correct learning output variable value to a given learning input variable value—is particularly error-prone, as it is often based on the empirical knowledge of human experts who have evaluated images. However, it is difficult (if not impossible) to express this empirical knowledge in quantitative terms used to evaluate images.

[0050] The present invention also relates to a parameter set having the following parameters: these parameters characterize the characteristics of a trainable module and are obtained using the method described above. For example, these parameters can be weights used to calculate the inputs of neurons or other computational units in a KNN to activate these neurons or computational units. This parameter set now accounts for the cost of training and is therefore an independent product.

[0051] The method can in particular be implemented completely or partially by software. The invention therefore also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform one of the described methods.

[0052] Likewise, the present invention relates to a machine-readable data carrier having a computer program and / or to a download product having a computer program. A download product is a digital product that can be transmitted via a data network, that is, can be downloaded by a user of the data network and can be offered for immediate downloading, for example, in an online store.

[0053] Furthermore, the computer can be equipped with the computer program, with a machine-readable data carrier or with a download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Below, together with the description of preferred exemplary embodiments of the present invention, further measures for improving the present invention are shown in more detail with reference to the accompanying drawings.

[0055] in:

[0056] Figure 1 An embodiment of a method 100 for training is shown;

[0057] Figure 2An exemplary embodiment of a method 200 is shown, which has a complete chain of actions up to the control of the technical system;

[0058] Figure 3 An exemplary modeling of the distribution 3 of the uncertainty 2a by a superposition 4 consisting of two contributions 41 , 42 is shown;

[0059] Figure 4 The failure of modeling is shown in the case where essentially all of the learning dataset 2 is correctly labeled. DETAILED DESCRIPTION

[0060] Figure 1 An embodiment of a method 100 for training a trainable module 1 is shown. In step 110, a plurality of modification processes 1a-1c of the trainable module 1 are pre-trained using at least a subset of an existing learning dataset 2. Each learning dataset 2 comprises learning input variable values ​​11a and associated learning output variable values ​​13a.

[0061] In step 120, learning input variable values ​​11a from the learning dataset 2 are fed as input variables 11 to all modification processes 1a-1c. Each modification process 1a-1c then generates its own output variable values ​​13. In step 130, a measure of uncertainty 13b for these output variable values ​​is determined from their deviations from one another. This measure of uncertainty 13b is assigned to the learning dataset 2 from which the learning input variable values ​​11a were extracted as a measure of uncertainty for said learning dataset 2.

[0062] In step 140, an evaluation 2b of the learning dataset 2 is determined from the uncertainty 2a. This evaluation 2b is a measure of the degree to which the assignment of the learned output variable values ​​13a to the learned input variable values ​​11a in the learning dataset 2 (i.e., the labeling of the learning dataset 2) is correct. Within block 140, exemplary methods for determining the evaluation 2b are categorized.

[0063] For example, according to block 141 , a distribution 3 of uncertainty 2a may be determined based on a large set of learning data 2 and then the distribution 3 may be further evaluated.

[0064] For example, according to block 142, the distribution 3 can be modeled as a superposition of multiple parameterized contributions 41, 42. In this case, for example, according to block 142a, the different contributions 41, 42 can be modeled using the same parameterized function, but with independent parameters 41a, 42a. According to block 142b, for example, a statistical distribution can be used, in particular a distribution from the exponential family, such as, for example, a normal distribution, an exponential distribution, a gamma distribution, a chi-squared distribution, a beta distribution, an exponential Weibull distribution, and / or a Dirichlet distribution.

[0065] The contributing parameters 41a, 42a can be optimized according to block 143, for example, in such a way that the deviation of the observed distribution 3 from the determined superposition 4 is minimized. For this optimization, according to block 143a, for example, likelihood methods and / or Bayesian methods can be used, such as, for example, the expectation-maximization algorithm, the expectation / conditional maximization algorithm, the expectation-conjugate gradient algorithm, the Riemann batch algorithm, Newton-based methods (for example, Newton-Raphson), Markov chain Monte Carlo-based methods (for example, Gibbs sampling or Metropolis-Hastings algorithm), and / or stochastic gradient algorithms.

[0066] According to block 144, the deviation of distribution 3 from superposition 4 can already provide important information: whether essentially only learning data sets 2 with the same or similar evaluations 2b contribute to distribution 3. For example, if contributions 41 and 42 to superposition 4 are used to distinguish correctly labeled learning data sets 2 from incorrectly labeled learning data sets 2, then measures taken after identifying incorrectly labeled data sets 2 may always result in essentially only correctly labeled learning data sets 2 remaining. This can be detected according to block 144. For example, termination conditions for the aforementioned measures can be derived from this.

[0067] In general, according to block 145, the desired evaluation 2b can be determined from the distribution 3. According to block 145a, the contributions 41, 42 to the superposition 4 can be used for this purpose, with which the distribution 3 is modeled. For example, such contributions 41, 42 can assign local probability densities to the uncertainty 2a of the learning data set 2, which learning data set 2 is correctly or incorrectly labeled with said local probability density. A corresponding odds ratio can be formed from a plurality of local probability densities. Alternatively or also in combination therewith, it can be observed according to block 145b to which contribution 41, 42 the learning data set 2 is assigned when optimizing 143 the parameters 41a, 42a of the contributions 41, 42. As explained above, some algorithms for optimization directly supply the following information: on which learning data set 2 they each rely.

[0068] According to block 146 , a scalar measure 43 can be determined from the contributions 41 , 42 specified by the parameters 41 a, 42 a, regarding the extent to which these contributions 41 , 42 differ from one another. This scalar measure 43 can be, for example, the Kullbeck-Leibler divergence. In particular, according to block 146 a , the dependence of this scalar measure 43 on a number of epochs of pre-training 110 of modification processes 1 a-1 c and / or the dependence of this scalar measure 43 on a number of training steps of pre-training 110 of modification processes 1 a-1 c can be determined. According to block 146 b , a possible useful application is to selectively select the number of epochs and / or training steps used in pre-training 110 so that the scalar measure 43 is maximized and the contributions 41 , 42 can thus be distinguished from one another in the best possible manner.

[0069] exist Figure 1 In addition, an exemplary useful application of the evaluation 2b of the learning data set 2 determined in step 140 is given.

[0070] In step 150, the ultimately required trainable module 1 can be trained by optimizing adaptable parameters 12 that characterize the trainable module 1 with the goal of improving the value of a cost function 14. According to block 151, cost function 14 measures the degree to which trainable module 1 maps learning input variable values ​​11a contained in the learning dataset to associated learning output variable values ​​13a. According to block 152, at least one learning dataset 2 in cost function 2 is weighted in dependence on an evaluation 2b of the learning dataset 2.

[0071] Alternatively or in conjunction therewith, in step 160, a check can be performed to determine whether evaluation 2b of learning data set 2 meets a predetermined criterion. This criterion can be, for example, that evaluation 2b exceeds or falls short of a predetermined threshold value and / or that evaluation 2b classifies learning data set 2 as incorrectly labeled. If this is the case (true value 1), an update 13a* of the learning output variable values ​​13a contained in learning data set 2 can be requested in step 170.

[0072] Figure 2An exemplary embodiment of a method 200 is shown. In step 210 of method 200, a trainable module 1 is trained using the previously described method 100. The thus trained module is operated in step 220 by being fed with input variable values ​​11 comprising physically recorded and / or simulated measurement data relating to a technical system. In step 230, control signals 5 are generated from the output variable values ​​13 supplied by the trainable module 1. These control signals 5 are used to control a vehicle 50 and / or a sorting system 60 and / or a system 70 for quality control of series-produced products and / or a system 80 for medical imaging.

[0073] Figure 3 By way of example, it is shown how the distribution 3 of the uncertainties 2a, u can be modeled by a superposition 4 consisting of two contributions 41, 42. For each value of the uncertainties 2a, u, a value of the local probability density ρ is plotted, which results from the respective contribution 41, 42 as a function of the respective uncertainty 2a. In this example, the superposition 4 is formed as a weighted addition of all uncertainties 2a and can be presented decomposed into the contributions 41, 42. However, it is also possible, for example, to select the higher function value of the two contributions 41, 42 for each value of the uncertainties 2a, u. Figure 3 In the example shown in , a first contribution 41 , which is large when the uncertainties 2a, u are low, comes from correctly labeled learning data set 2. A second contribution 42 , which is large when the uncertainties 2a, u are high, comes from incorrectly labeled learning data set 2.

[0074] Figure 4 It is shown by way of example that when the learning dataset 2 is correctly labeled, Figure 3 How the modeling explained in [1] may fail. The distribution 3 of the uncertainties 2a, u is then centered around low values. However, the three modelings using superposition 4 shown as examples still each assume the existence of two contributions 41, 42 and attempt to somehow make this scenario coincide with the distribution 3 according to an error metric (e.g., the minimum sum of squared errors). Figure 4 As shown, the deviation is large. From this, a clear signal can be drawn that all learning data sets 2 are correctly labeled.

Claims

1. A computer-implemented method (100) for training a trainable module (1) with the aid of a learning data set (2), the trainable module (1) converting one or more input variables (11) into one or more output variables (13), the learning data set (2) comprising learning input variable values ​​(11a) and associated learning output variable values ​​(13a), wherein the input variables (11) each comprise an image pixel and the output variables (13) each comprise a class, wherein the learning input variable values ​​(11a) each comprise an image pixel and the associated learning output variable values ​​(13a) each comprise a class assigned to the image pixel, and wherein at least the learning input variable values ​​(11a) have been obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a technical system observable using such a measurement process, the method (100) comprising the following steps: pre-training (110) a plurality of modification processes (1a-1c) of the trainable module (1) using at least a subset of the learning data set (2), the modification processes (1a-1c) being so different from each other that the modification processes (1a-1c) are not congruently transformable into each other during continuous learning; The learned input variable values ​​(11a) of at least one learning data set (2) are fed (120) as input variables (11) to all modification processes (la-lc); a measure of uncertainty (13b) for the output variable (13) is determined from the deviations of the output variables (13) from one another, and the measure of uncertainty (13b) for the output variable (13) is assigned (130) to the learning data set (2) as a measure of uncertainty (2a) for the learning data set (2), the modification process (1a-1c) converting the learning input variable values ​​(11a) into the output variable (13); determining (140) an evaluation (2b) of the learning data set (2) based on the uncertainty (2a), the evaluation (2b) being a measure of how correctly the assignment of the learned output variable values ​​(13a) to the learned input variable values ​​(11a) in the learning data set (2) is performed; in, Determining (141) a distribution (3) of the uncertainty (2a) based on a plurality of learning data sets (2), and wherein determining (145) the evaluation (2b) based on the distribution (3); In this case, the distribution (3) is modeled (142) as a superposition (4) of multiple parameterized contributions (41, 42), each of which originates from a learning data set (2) having the same or similar evaluation (2b), and in which the parameters (41a, 42a) of the contributions (41, 42) are optimized (143) as follows: the deviation of the distribution (3) of the uncertainty (2a) from the superposition (4) is minimized in order to determine the contributions (41, 42) in this way.

2. The method (100) according to claim 1, wherein: Adaptable parameters (12) characterizing properties of the trainable module (1) are optimized (150) with the aim of improving the value of a cost function (14), wherein the cost function (14) measures (151) the extent to which the trainable module (1) maps the learning input variable values ​​(11a) contained in the learning data set (2) to the associated learning output variable values ​​(13a), wherein the weighting of at least one learning data set (2) in the cost function (14) is dependent (152) on the evaluation (2b) of the at least one learning data set (2).

3. The method (100) according to claim 2, wherein: In response to the evaluation (2b) of the learning data set (2) meeting a predetermined criterion, the learning data set (2) is no longer considered in the cost function (14).

4. The method (100) according to any one of claims 1 to 3, wherein: In response to the evaluation (2b) of the learning data set (2) satisfying (160) a predetermined criterion, a request (170) is made to update (13a*) at least one learning output variable value (13a) contained in the learning data set (2).

5. The method (100) according to any one of claims 1 to 3, wherein: Based on the deviation of the distribution (3) from the superposition (4), it is at least determined (144) whether essentially only learning data sets (2) with identical or similar evaluations (2b) contribute to the distribution (3).

6. The method (100) according to any one of claims 1 to 3, wherein: Different contributions (41, 42) to the superposition (4) are modeled (142a) using the same parameterized function, but with mutually independent parameters (41a, 42a).

7. The method (100) according to any one of claims 1 to 3, wherein: At least one parameterized contribution (41, 42) is modeled (142b) as a statistical distribution, including: modeled (142b) as a normal distribution, modeled (142b) as an exponential distribution, modeled (142b) as a gamma distribution, modeled (142b) as a chi-square distribution, modeled (142b) as a beta distribution, modeled (142b) as an exponential Weibull distribution, and / or modeled (142b) as a Dirichlet distribution.

8. The method (100) according to any one of claims 1 to 3, wherein: The parameters (41a, 42a) of the contribution (41, 42) are optimized (143a) according to a likelihood method and / or according to a Bayesian method, including: using an expectation maximization algorithm, using an expectation / conditional maximization algorithm, using an expectation conjugate gradient algorithm, using a Riemann batch algorithm, using a Newton-based method, using a Markov chain Monte Carlo-based method and / or using a stochastic gradient algorithm (143a).

9. The method (100) according to any one of claims 1 to 3, wherein: The evaluation (2b) of at least one learning data set (2) is determined (145a) based on a local probability density and / or based on a ratio of such local probability densities, if the uncertainty (2a) of the learning data set (2) is fed as input to at least one contribution (41, 42) to the superposition (4), then the at least one contribution (41, 42) to the superposition (4) outputs the local probability density.

10. The method (100) according to any one of claims 1 to 3, wherein: The following is entered into the evaluation (2b) of at least one learning data set (2) (145b): when optimizing (143) the parameters (41a, 42a) of the contribution (41, 42), to which contribution (41, 42) the learning data set (2) is assigned.

11. The method (100) according to any one of claims 1 to 3, wherein: From the contributions (41, 42), the Kullbeck-Leibler divergence, the Hellinger distance, the Lévy distance, the Lévy-Prokhorov metric, the Wasserstein metric, the Jensen-Shannon divergence and / or another scalar metric (43) for how far the contributions (41, 42) differ from one another is determined (146).

12. The method (100) according to claim 11, wherein: determining (146a) a correlation of the scalar measure (43) with a number of epochs of the pre-training (110) of the modification process (1a-1c) and / or with a number of training steps of the pre-training (110) of the modification process (1a-1c), Therein, a number of epochs and / or a number of training steps are taken into account (146b) for further determining the uncertainty (2a) of the learning data set (2b), in which number of epochs and / or number of training steps the scalar measure (43) indicates a maximum difference in the contributions (41, 42) to the superposition (4).

13. A computer-implemented method (200) comprising the steps of: a method (100) according to any one of claims 1 to 12, wherein a trainable module (1) is trained (210) with the aid of a learning data set (2), the trainable module (1) converting one or more input variables (11) into one or more output variables (13), the learning data set (2) comprising learning input variable values ​​(11a) and associated learning output variable values ​​(13a); The trainable module (1) is operated (220) in the following manner: input variable values ​​(11) are supplied to the trainable module (1), wherein the input variables (11) each comprise an image pixel, the output variables (13) each comprise a class, wherein the learning input variable values ​​(11a) each comprise an image pixel, and the associated learning output variable values ​​(13a) each comprise the class assigned to the image pixel, and wherein the input variable values ​​(11) have been obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a technical system observable by means of such a measurement process; controlling (230) the vehicle (50), and / or the classification system (60), and / or the system for quality control of series-produced products (70), and / or the system for medical imaging (80) using the control signal (5) as a function of the output variable (13) supplied by the trainable module (1), in, Determining (141) a distribution (3) of uncertainty (2a) based on a plurality of learning data sets (2), and wherein determining (145) an evaluation (2b) based on said distribution (3); In this case, the distribution (3) is modeled (142) as a superposition (4) of multiple parameterized contributions (41, 42), each of which originates from a learning data set (2) having the same or similar evaluation (2b), and in which the parameters (41a, 42a) of the contributions (41, 42) are optimized (143) as follows: the deviation of the distribution (3) of the uncertainty (2a) from the superposition (4) is minimized in order to determine the contributions (41, 42) in this way.

14. A computer program product comprising a computer program, the computer program containing machine-readable instructions which, when executed on one or more computers, cause the one or more computers to perform the method (100, 200) according to any one of claims 1 to 13.

15. A machine-readable data carrier having a computer program, the computer program comprising machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the method (100, 200) according to any one of claims 1 to 13.

16. Computer device equipped with a computer program product according to claim 14 and / or equipped with a machine-readable data carrier according to claim 15.

Images