editing learning data records with noisy labels for a classifier

Abstract

A method (100) for editing learning data records (2) for a classifier (1) includes the following steps: • The learning input parameter value (11a) of at least one learning data record (2) is obtained from the trained epoch E 2 One or more classifiers (1, 1a-1c) process the learning input parameter values ​​multiple times in an inconsistent manner, such that these learning input parameter values ​​are mapped to different output parameter values ​​(13); • A measure of the uncertainty (13b) of these output parameter values ​​is determined based on the deviations between these output parameter values; • As a response to the uncertainty meeting a pre-given criterion, at least one learning output parameter value (13a*) is determined for the learning data record (2) based on one or more other output parameter values ​​(14), wherein the one or more classifiers (1, 1a-1c) reset the learning input parameter value (11a) to have epoch E 1 <E 2 The earlier training state is then mapped to one or more other output parameter values.

Description

Edit learning data records with noisy labels for the classifier. Technical Field

[0001] This invention relates to the editing of training data that is needed to train a classifier, for example, to identify objects in the environment surrounding a vehicle. Background Technology

[0002] Typically, training human drivers to operate vehicles in road traffic involves repeatedly subjecting driving school students to specific rules within their training scope. Students must react to these situations and receive feedback from the instructor—whether their reactions were right or wrong—through comments or even intervention. This training using a limited number of scenarios should enable students to handle unfamiliar situations even when driving independently.

[0003] To enable vehicles to participate in road traffic fully or partially automatically, efforts are being made to control these vehicles using modules that can be trained in a very similar manner. These modules, for example, acquire sensor data from the vehicle's surrounding environment as input parameters and determine the classification of objects in the vehicle's surrounding environment, so as to obtain a machine-processable representation of the traffic situation.

[0004] For this training, a sufficient number of learning data records are required, each including learning input parameter values ​​and associated learning output parameter values. For example, learning input parameter values ​​may include images, and these images may be "labeled" as learning output parameters with information about which objects are contained within them. This "labeling" is typically manual and therefore expensive. DE 10 2017 219 269 A1 discloses a method for identifying learning data records whose "labeling" guarantees maximum classification accuracy. Summary of the Invention

[0005] Within the scope of this invention, a method for editing learning data records for a classifier has been developed. This method is based on an existing set of learning data records. These learning data records include learning input parameter values ​​and associated learning output parameter values, to which the classifier aims to nominally map these learning input parameter values ​​to these learning output parameter values. The learning input parameter values ​​include measurement data obtained through physical measurement processes and / or through partial or complete simulation of such measurement processes and / or through partial or complete simulation of technical systems observable using such measurement processes. By training using the learning data records, the aim is to enable the classifier to also map input parameter values ​​that are not part of the training dataset to accurate output parameter values ​​for the corresponding application.

[0006] For example, these measurement data can represent camera images, thermal images, radar data, lidar (LIDAR) data, or ultrasonic data. The output parameter values can represent classes that should be assigned to these measurement data respectively for a pre-given classification. These classes can represent, for example, objects or states that need to be detected.

[0007] Here, the terms "input parameter value" and "output parameter value" are not limited to scalars respectively, but can also include vectors and matrices, for example.

[0008] Within the scope of this method, the learning input parameter values of at least one learning data record are processed in an inconsistent manner multiple times by one or more classifiers trained up to epoch E2 (Epoche), such that these learning input parameter values can be mapped to different output parameter values. In this case, the term "epoch" should be understood, in terms of the technical term for a trainable classifier, as a stage of training in which the entire available set of learning data records is processed by the corresponding classifier and the classifier is adapted based on the comparison of the obtained output parameter values with the corresponding learning output parameter values.

[0009] As will be further elaborated below, processing the same learning input parameter value in an inconsistent manner into different output parameter values can be achieved, for example, by using different classifiers for each processing. Alternatively or in combination therewith, the processing procedure of the same classifier each time it is used can be varied, for example, by using random dropout.

[0010] Based on the deviation between the obtained output parameter values from each other, a measure of the uncertainty of these output parameter values is determined. In response to the uncertainty meeting a pre-given criterion, at least one updated learning output parameter value for the learning data record is determined based on one or more other output parameter values, where one or more classifiers map the learning input parameter values to the one or more other output parameter values after being reset to an earlier training state with epoch E1 < E2. This criterion can particularly include, for example, that the uncertainty exceeds a pre-given threshold.

[0011] It has been recognized that the updated learning output parameter values obtained in this way are generally more accurate for the corresponding application than the updated learning output parameter values determined based on the training state with epoch E2. The reason is that the training of the classifier is usually optimized so that the learning input parameter values are mapped to the corresponding learning output parameter values as accurately as possible on average. If some of these learning output parameter values are noisy or incorrect, contradictions occur in this case. Thus, for example, the same object cannot be a vehicle and a farm animal at the same time. For this type of contradiction, the optimization will look for a compromise solution. Here, a solution that was somewhat accurate at the first epoch E1 may be "made worse" during continued training, and although a good compromise can be formed at the second epoch E2 with respect to the cost function used for the optimization, this compromise cannot be used as the updated learning output parameter values.

[0012] When the classifier is trained, the parameters that characterize it and are optimized during training are usually stored at least at the end of each epoch. These parameters can include, for example, the weights of the neural network in the classifier. Thus, by reading in the stored parameters, a quick reset to an earlier training state with epoch E1 can be achieved.

[0013] In a particularly advantageous design, the learning output parameter values are mapped by multiple variants of the classifier to different output parameter values. These variants differ from each other to the extent that they do not transform into each other consistently as training progresses. That is, these variants form an ensemble. The different output parameter values determined within this ensemble allow for a statistically meaningful inference of the uncertainty of the output parameter values. At the same time, the other output parameter values obtained after resetting the ensemble classifier to epoch E1 can be settled in an understandable and statistically meaningful way to form the updated output parameter values.

[0014] In another particularly advantageous design, for example, a separate uncertainty can be determined for the output parameter values determined by each variant of the classifier. Then, only a fixed number of those variants that have produced output parameter values with the lowest uncertainty can contribute to determining the updated learning output parameter values. In other words, among the set of N variants, the first K variants have a say, where 1 < K < N, and these variants have the highest probability of accurately suggesting the output parameter values.

[0015] That is, the uncertainties among the recommendations of these integrators are effectively weighted. When considering the established process for defining official values ​​of physical constants in nature, it becomes immediately clear that updated learning output parameter values ​​determined in this way are likely more accurate for the application of the classifier than the individual recommendation of a single integrator. The values ​​determined by the Council for Data in Science and Technology (CODAT) are best estimates based on a large number of measurements from different laboratories, which are weighted against each other using the reciprocal of the square of their respective standard uncertainties. The inventors' experiments have also confirmed that, precisely in classification problems with a large number of available categories (e.g., ImageNet data records with 1000 categories), considering the first K variants rather than just the first 1 variant provides significantly more accurate updated learning output parameter values. If the learning data records used for the final classifier are upgraded with such updated values, the classifier provides output parameter values ​​that better match the corresponding target output parameter values ​​when tested with test or effective data records.

[0016] In another particularly advantageous design, separate updated learning output parameter values ​​are specified for each variant of the classifier. Among these different learning output parameter values, the uncertainty between the output parameter values ​​provided by different variants is particularly evident.

[0017] This means that when training the classifier ultimately used for classification, instead of training a single classifier, an ensemble of variants is trained. Each variant is then trained with learned output parameters, which are partly identical for all these variants and partly specified individually for each variant. Correspondingly, even when using the classifier later, the input parameter values ​​can be fed to all the trained variants of the ensemble, and the output parameter values ​​provided by these variants are interleaved.

[0018] It is quite common for the learned input parameter values ​​to be processed into output parameter values ​​multiple times in an inconsistent manner by one or more classifiers at an earlier training state with epoch E1. Then, summary statistics, such as the mean and median, can be determined about these output parameter values. Feeding the learned input parameter values ​​to different variants is just one possible way to ensure inconsistent processing over multiple iterations. Alternatively, or in combination with this, the processing itself can also be implemented probabilistically. For example, in a classifier or variant, the weights or other parameters characterizing the features of that classifier or variant can take new values ​​from the distribution each time they are used.

[0019] The summary statistic can be directly defined as the updated learning output parameter value, for example. However, it can also be assigned an updated learning output parameter value to each variant of the classifier, which assigns exactly one of the pre-defined classifiable categories to the learning input parameter value. The learning output parameter values ​​can then be distributed across these variants such that the ensemble average of these variants reproduces the summary statistic.

[0020] In a simple example, the pre-given classification includes five categories from 0 to 4, and five variants of the classifier to be trained are used so that the same learning input parameter values ​​are processed multiple times in inconsistent ways into output parameter values. These output parameter values ​​are vectors with five elements each, corresponding to the five dominable categories. Now, for example, in the training state with epoch E1, for a specific learning input parameter value, the average of the output parameter values ​​with respect to these five ensemble members should be the vector [0.6, 0.2, 0.2, 0, 0]. The vector [1, 0, 0, 0, 0] is assigned as a learning output parameter value to the first three variants. The vector [0, 1, 0, 0, 0] is assigned as a learning output parameter value to the fourth variant. The vector [0, 0, 1, 0, 0] is assigned as a learning output parameter value to the fifth variant. When processing the learned input parameter values ​​using an ensemble consisting of these five variations, the probability of each variation being active is now exactly the same. If one of the first three variations is active after training, the learned input parameter value is mapped to the vector [1, 0, 0, 0, 0], with a probability of 3 / 5 = 0.6. If the fourth variation is active, the learned input parameter value is mapped to the vector [0, 1, 0, 0, 0], with a probability of 1 / 5 = 0.2. If the fifth variation is active, the learned input parameter value is mapped to the vector [0, 0, 1, 0, 0], with a probability of 1 / 5 = 0.2. Therefore, the ensemble average for the output parameter values ​​mapped to the learned input parameter values ​​is [0.6, 0.2, 0.2, 0, 0]. The ensemble averaging is then carried out by setting the updated learning output parameter values ​​[1, 0, 0, 0, 0], [0, 1, 0, 0, 0], and [0, 0, 1, 0, 0] to other epochs.

[0021] To help understand this approach, consider its similarity to multiple imputation in mathematical statistics. In multiple imputation, multiple estimates are determined for each missing value in a statistical survey, and these estimates are then combined, for example, by averaging them, to form a single estimate.

[0022] The key advantage of this approach is that it avoids prematurely ruling out the possibility that a non-dominant class in the ensemble mean of the output parameter values ​​is actually the accurate class for learning updates to the output parameter values. That is, in the example shown above, it does not prematurely conclude from the value 0.6 in the ensemble mean of the output parameter values ​​[0.6, 0.2, 0.2, 0, 0] that the accurate class for learning updates to the output parameter values ​​can only be class 0.

[0023] In another particularly advantageous design, the learned input parameter values ​​are processed multiple times by at least one classifier. Here, one or more neurons and / or other processing units of the classifier are randomly and controllably deactivated. For example, this processing can be made probabilistic through Monte Carlo dropout.

[0024] As described above, the output parameter values ​​provided by a classifier using noisy or erroneous learned output parameter values ​​for learning input parameter values ​​may be more accurate in an earlier epoch E1 than in a later epoch E2, where the aforementioned trade-offs may have been incorporated to minimize the cost function. Therefore, in another particularly advantageous design, the accuracy with which one or more classifiers map test or effective input parameter values ​​to their corresponding test or effective output parameter values ​​is determined based on the training state of one or more classifiers. The epoch at which this accuracy is maximized is selected as epoch E1 for determining the updated learned output parameter values.

[0025] By considering the uncertainty of output parameter values ​​provided by one or more classifiers not only when identifying noisy or erroneous learned output parameter values ​​but also when determining the updated learned output parameter values, the probability that the updated learned output parameter values ​​are more accurate than the original learned output parameter values ​​in the context of applying the classifiers is increased. If inaccurate new learned output parameter values ​​are repeatedly specified for learned input parameters (such as images) (e.g., the image does not actually belong to the wrong category), the network's overall learning ability may be impaired.

[0026] As described above, the one or more classifiers can continue to be trained using learning data records that are updated with at least one updated learning output parameter value. That is, methods for editing learning data records can be integrated into the training of the classifiers.

[0027] For example, the training phase can alternate with the editing of the learning data records. For instance, training can be performed first using the original learning data records. These learning data records can then be edited as described above. Next, the training can be repeated using these new learning data records. This cycle can be repeated until any interruption criterion is met.

[0028] The interruption criterion may, for example, include: the uncertainty of the output parameter value determined based on the learning input parameter value exceeds a predetermined threshold only if it does not exceed a predetermined percentage of the learning data records.

[0029] Alternatively, or in combination with, the interruption criterion may include, for example, that the classifier, when performing tests using test or effective data records, maps the corresponding test input parameter value or effective input parameter value to the corresponding associated test output parameter value or effective output parameter value with at least a pre-given precision. That is, the test output parameter value or effective output parameter value should be considered the target output parameter value in the case of such testing.

[0030] By simultaneously considering multiple categories for the updated learning output parameter values, whether by assigning different categories to different variants or by forming a single learning output parameter value encompassing multiple categories, the tendency to completely alter the learning output parameter values ​​in the learning data record is reinforced overall. In this case, there is always a risk, in principle, that the updated learning output parameter values ​​will not be entirely accurate in the context of applying the classifier. However, especially when a classification with multiple categories is given in advance, the probability that the updated learning output parameter values ​​are at least significantly more accurate than the original learning output parameter values ​​is high.

[0031] The deeper reason lies in the fact that pre-defined classifications with multiple categories often contain semantic hierarchies. Thus, for example, in ImageNet classification with 1000 categories, there are multiple categories involving different dog breeds and multiple categories involving different plant species. If, for example, a sheepdog previously labeled with the incorrect learning output parameter value "cactus" is now classified as an "Eskimo Dog," while this is still not entirely correct, the sheepdog has at least been identified as a dog once. Furthermore, multiple categories can be specified through updated learning output parameter values, and it is quite possible that the contribution of the correct category "sheepdog" is not zero. This contribution may even increase as training continues, especially when alternating between the training phase and editing of the learning data records.

[0032] The described method for editing learning data records reduces the number of records whose learned output parameter values ​​in the context of applying the classifier do not match their corresponding learned input parameter values. This is particularly advantageous when training classifiers that are especially sensitive to such misassignments in learning data records.

[0033] Therefore, the present invention also relates to a method for training a classifier based on learning data records. The learning input data contained in these learning data records includes measurement data obtained through a physical measurement process and / or through partial or complete simulation of such a measurement process and / or through partial or complete simulation of a technical system observable using such a measurement process. Additionally, these learning data records each contain learning output parameter values, to which the classifier is designed to nominally map the learning input parameter values ​​to these learning output parameter values.

[0034] Within the scope of this method, a pre-classifier is first trained to distinguish learning data records whose corresponding learning input parameter values ​​are assigned to correct learning output parameter values ​​from learning data records whose corresponding learning input parameter values ​​are assigned to noisy or erroneous learning output parameter values. This pre-classifier can be constructed relatively simply and, for example, can divide the assignment of learning output parameter values ​​to learning input parameter values ​​into one of two or more quality categories, depending on the correctness of that assignment within the context of applying the classifier.

[0035] A pre-classifier is used to classify the learning data records. The assignment of learning output parameter values ​​to learning input parameter values ​​is determined by the pre-classifier; learning data records deemed accurate are included in a preferred set of learning data records.

[0036] The learning output parameter value is determined by the pre-classifier to be noisy or erroneous at least one learning data record, and at least one updated learning output parameter value is provided using the method described above. The learning data record processed in this manner is included in a preferred set of learning data records.

[0037] Finally, the actual classifier is trained using training data records from a preferred set of data records. This classifier can have a relatively complex architecture, which, while guaranteeing higher accuracy, is sensitive to noisy or erroneous learning output parameter values ​​in order to achieve higher accuracy.

[0038] In this context, pre-filtering by the pre-classifier has the following effect: noisy or erroneous learning output parameter values ​​can be excluded from the training of the true classifier without harming the training results. Simultaneously, editing noisy or erroneous learning output parameter values ​​using the above method results in the following: by excluding noisy or erroneous learning output parameter values, not much labeled learning data record is ultimately lost. That is, some of these learning data records can still be "saved." As mentioned above, learning data records are a scarce resource because labeling usually requires manual intervention.

[0039] In a particularly advantageous design, the learning data record with the updated learning output parameter values ​​is fed back to the pre-classifier. Then, only if the pre-classifier classifies the learning data record as having the correct learning output parameter values ​​assigned to its learning input parameter values ​​is it included in the preferred set of learning data records. That is, the success of the "relabeling" is specifically checked before using the results for further training.

[0040] In a broader sense, the combination of pre-filtering using a pre-classifier and training the real classifier can be viewed as a "teacher-student process." The similarity lies in feeding all the learning data records to the pre-classifier ("teacher"), while only a portion of these learning data records is used to train the real classifier ("student").

[0041] As described above, by editing the learning data records, the classifier trained using the edited learning data records is ultimately able to classify the measurement data more accurately within the context of its respective application. That is, if the technical system is manipulated based on the output parameter values ​​provided by the classifier, the probability of the triggered action regarding the current state of the technical system being appropriate is increased. This current state is reflected in the input parameter values ​​of the classifier.

[0042] Therefore, the present invention also relates to another method that improves the behavioral chain from training to controlling a technical system.

[0043] In this method, the classifier is first trained using learning data records edited using the method described first and / or using the training method described later.

[0044] The classifier is then run by feeding input parameter values. These input parameters include measurement data obtained through physical measurement processes and / or through partial or complete simulations of such measurement processes and / or through partial or complete simulations of the technological system that can be observed using such measurement processes. For example, realistic images of the situation can be generated using computational ray tracing or neural generator networks (such as generative adversarial networks, GANs). In this case, knowledge from the simulation of the technological system, such as the location of a specific object, can also be introduced as an auxiliary condition. The generator network can be trained to selectively generate images that satisfy these auxiliary conditions (such as conditional GANs, i.e., cGANs).

[0045] Control signals are generated based on the output parameter values ​​provided by the classifier. These control signals are then used to control vehicles and / or for quality control of mass-produced products and / or medical imaging systems.

[0046] These methods can be implemented entirely or partially by a computer. Therefore, the present invention also relates to a computer program having machine-readable instructions that, when implemented on one or more computers, cause the computers to perform one of the described methods. In this regard, control devices for vehicles and embedded systems for technical devices, which are also capable of implementing machine-readable instructions, should also be considered as computers.

[0047] Similarly, the present invention also relates to a machine-readable data carrier and / or downloadable product having the computer program. The downloadable product is a digital product that can be transmitted via a data network, that is, downloaded by a user of the data network, and such digital product may, for example, be sold in an online store for immediate download.

[0048] In addition, the computer may be equipped with the computer program, the machine-readable data carrier, or the downloadable product. Attached Figure Description

[0049] In the following description, other improvements to the invention will be presented in more detail with reference to the accompanying drawings and preferred embodiments of the invention.

[0050] in:

[0051] Figure 1 illustrates an embodiment of a method 100 for editing learning data records 2;

[0052] Figure 2 shows the relationship between classification accuracy A and the number of epochs e used for training with noise-free labels (curve a) and noisy labels (curve b).

[0053] Figure 3 shows the improvement of the classification accuracy A shown in Figure 2 by considering the first 3 integrated classifiers 1a - 1c;

[0054] Figure 4 shows a segment in the confusion matrix of the images of the ImageNet data records after training in the case of editing the learning data record 2 by method 100;

[0055] Figure 5 shows an embodiment of method 200 for training classifier 1;

[0056] Figure 6 shows an embodiment of method 300 with a complete behavior chain up to the manipulation technology systems 50, 60, 70. Detailed Description

[0057] Figure 1 is a schematic flow chart of an embodiment of method 100 for editing the learning data record 2 for classifier 1. The learning data record 2 includes learning input parameter values 11a and the associated learning output parameter values 13a, and the classifier aims to nominally map these learning input parameter values 11a to these learning output parameter values. The learning input parameter values 11a include measurement data, which are obtained by a physical measurement process and / or by partial or complete simulation of such a measurement process and / or by partial or complete simulation of a technical system that can be observed using such a measurement process.

[0058] In step 110 of method 100, the learning input parameter values 11a of at least one learning data record 2 are processed multiple times in an inconsistent manner by one or more classifiers 1, 1a - 1c trained for epoch E2. Thereby, for the same learning input parameter values 11a, multiple different output parameter values 13 are formed. For this purpose, for example, according to block 111, multiple variants 1a - 1c of the classifier can be used, which differ from each other to such an extent that they do not consistently transform into each other as the training progresses.

[0059] In step 120, based on the deviation between the output parameter values 13 from each other, a measure of the uncertainty 13b of these output parameter values 13 is determined. In response to the uncertainty 13b meeting a pre - given criterion 130 (probability value 1), at least one updated learning output parameter value 13a* for the learning data record 2 is determined based on one or more other output parameter values 14, where one or more classifiers 1, 1a - 1c map the learning input parameter values 11a to the one or more other output parameter values after resetting to an earlier training state with epoch E1 < E2.

[0060] In step 150, the one or more classifiers 1, 1a-1c can continue to be trained using learning data record 2 with at least one updated learning output parameter value 13a*.

[0061] Here, for example, according to block 121, a separate uncertainty 13c can be determined for each output parameter value 13 determined by each variant 1a-1c. Then, according to block 122, only a fixed number of variants 1a-1c that have produced output parameter values ​​13 with the lowest uncertainty 13c can contribute to determining the updated learned output parameter value 13a*. That is, the top K output parameter values ​​13 with the lowest uncertainty 13c are used to form the updated learned output parameter value 13a*. To check this criterion 130, each uncertainty 13c can be, for example, settled into a unique uncertainty 13b. However, this criterion 130 can also be expressed directly as each uncertainty 13c.

[0062] According to box 141, in particular, for example, individual updated learning output parameter values ​​13a* can be determined for each variant 1a-1c. Then, these variants 1a-1c can continue to be trained independently of each other.

[0063] According to box 142, the learned input parameter value 11a can also be processed into output parameter value 14 multiple times in an inconsistent manner by one or more classifiers 1, 1a-1c at an earlier training state with epoch E1. This can be achieved, for example, by using multiple variants 1a-1c of classifier 1, but alternatively or in combination with it, this can also be achieved through a probabilistic design of the processing itself, such as Monte Carlo dropout or by obtaining the parameters of classifiers 1, 1a-1c from a distribution. Next, according to box 143, summary statistics 14a can be determined regarding these output parameter values ​​14.

[0064] This summary statistic 14a, for example, can be directly specified as the updated learning output parameter value 13a* according to box 144.

[0065] However, for example, according to box 145, a separate updated learning output parameter value 13a* can also be determined for each variant 1a-1c, which assigns exactly one category to the learning input parameter value 11a. According to box 146, these updated learning output parameter values ​​13a* can be distributed across these variants 1a-1c such that the ensemble average of these variants 1a-1c reproduces the summary statistic 14a.

[0066] Typically, according to boxes 117 and 147, one or more neurons and / or other processing units of classifiers 1, 1a-1c can be randomly and controlled to be deactivated (Monte Carlo dropout) when the learning input parameter value 11a is processed multiple times by at least one classifier 1, 1a-1c.

[0067] To find the optimal epoch E1 for determining the updated learning output parameter value 13a*, according to box 148, the accuracy with which one or more classifiers 1, 1a-1c map the test or effective input parameter value to its corresponding test or effective output parameter value can be determined based on the training state of one or more classifiers 1, 1a-1c. Then, the epoch with the maximum accuracy is selected as the epoch E1 for determining the updated learning output parameter value 13a*.

[0068] Figure 2 illustrates the motivation behind the method used to determine epoch 1. Here, the classification accuracy A determined by classifier 1 for images in the ImageNet dataset during epoch e of the training state of classifier 1 is plotted. Curve a is created for the case where all learned output parameter values ​​(labels) 13a are correct. Curve b is created for the case where 40% of the labels are noisy or incorrect. To determine accuracy A, the class considered most likely by classifier 1 (i.e., the class with the highest Softmax score) is compared to the correct class.

[0069] Curve 'a', created for the correct labels, steadily increases until its final value at the end of training, except for a slight pullback. Curve 'b', created for noisy labels, reaches its maximum value at epoch e=25 and accuracy A is approximately 0.8 (80%), then decreases again. This demonstrates the previously discussed effect: optimization of the cost function produces compromises when there are conflicting information in the labels, causing accuracy A to "get worse with continued training."

[0070] The same trend is also qualitatively shown in Figure 3, where a similar accuracy A to that determined in Figure 2 is plotted. However, unlike Figure 2, here accuracy A is measured in terms of the degree to which the first three output parameters of the output parameters 13, provided by the ensemble classifiers 1a-1c with the lowest uncertainty 13c, specify the correct class. Quantitatively, in the noisy case (curve b), the accuracy A of approximately 0.95 (95%) measured at epoch number e=25 is significantly better than that in Figure 2. Therefore, it is highly promising to determine the updated learned output parameter values ​​13a* based not on the first prediction of a single classifier 1, 1a-1c, but on the first K predictions of multiple ensemble members 1a-1c.

[0071] Figure 4 shows a fragment of the obfuscation matrix determined based on test or effective data. In this obfuscation matrix, each row specifies the object shown on the left side of that row.

[0072] · airplane,

[0073] Passenger vehicles (PKW)

[0074] · bird,

[0075] · cat,

[0076] · Elk,

[0077] · dog,

[0078] · frog,

[0079] · horse,

[0080] · Shiphe

[0081] • Cargo trucks (LKW)

[0082] The images are assigned probabilities or confidence levels to these ten categories. The diagonal elements in the bolded box of the matrix represent correct assignments. Off-diagonal elements represent confusion.

[0083] The confusion matrix is ​​formed after the set of learning data records, which initially had 40% noisy learning output parameter values ​​(labels) 13a, is edited once using the method 100 shown in Figure 1, and then the classifier 1 is trained again.

[0084] It is noteworthy in this confusion matrix that misassignments are mainly concentrated between the categories "dog" and "cat". This is because dogs and cats are quite similar to each other.

[0085] Figure 5 is a schematic flowchart of an embodiment of a method 200 for training a classifier 1 based on a given learning data record 2 having learning input parameter values ​​11a and learning output parameter values ​​13a. Similar to method 100, the learning input parameter values ​​11a include measurement data.

[0086] In step 210, the pre-classifier 1d is trained to distinguish the learning data record 2 with the correct learning output parameter value 13a assigned to its corresponding learning input parameter value 11a (that is, the learning data record 2 with the "clean" label 13a) from the learning data record 2 with the noisy or incorrect learning output parameter value 13a assigned to its corresponding learning input parameter value 11a (the noisy label 13a).

[0087] In step 220, the trained pre-classifier is used to classify the learning data record 2, and the learning data record 2 with the “clean” label 13a is included in the preferred set 2* of learning data records 2, on which the actual classifier 1 should be trained later.

[0088] As described above, much of the original learning data record 2 is "missing". However, the labeled learning data record 2 is a scarce and expensive commodity. Therefore, in step 230, the learning data record 2 with the noisy label 13a is edited using the method 100 illustrated in more detail in conjunction with FIG1. ​​In this case, at least one learning data record 2 obtains an updated learning output parameter value 13a*, that is, the learning data record is "relabeled".

[0089] The relabeled learning data record 2 is fed back to the pre-classifier 1d in step 240. In step 250, it is checked whether the learning data record 2, after being relabeled, can now be identified as a learning data record 2 with a "clean" label 13a. If so (probability value 1), the learning data record 2 is added to the preferred set 2* of learning data records 2 in step 260. In step 270, the true classifier 1 is trained using the learning data records 2 included in set 2*, which aims to classify any input parameter value 11 into a pre-given category through the output parameter value 13 of the classifier.

[0090] Figure 6 is a schematic flowchart of an embodiment of method 300 with a complete chain of behaviors.

[0091] In step 310, at least one classifier is trained based on the learning data record 2 edited using method 100 and / or using training method 200.

[0092] In step 320, the classifier is run by feeding the input parameter values ​​11 to the trained classifier 1. These input parameter values ​​11 include measurement data.

[0093] In step 330, a control signal 330a is formed based on the output parameter value 13 provided by classifier 1. This control signal is used to control vehicle 50 and / or system 60 and / or medical imaging system 70 for quality control of mass-produced products.

Claims

1. A method (100) for editing a learning data record (2) for a classifier (1), wherein the learning data record (2) includes measurement data as learning input parameter values ​​(11a), the measurement data being obtained by a physical measurement process and / or by a partial or complete simulation of such measurement process and / or by a partial or complete simulation of a technical system that can be observed using such measurement process, wherein the measurement data represents camera images, thermal images, radar data, lidar data, or ultrasonic data, wherein the learning data record (2) additionally includes learning output parameter values ​​(13a), the classifier (1) being designed to nominally map the learning input parameter values ​​(11a) to the learning output parameter values, the method comprising the steps of: • The learning input parameter value (11a) of at least one learning data record (2) being processed multiple times in an inconsistent manner (110) by one or more classifiers (1, 1a-1c) trained to epoch E2, such that the learning input parameter value is mapped to different output parameter values ​​(13); • Based on the deviations between the output parameter values ​​(13), a measure of the uncertainty (13b) of the output parameter values ​​(13) is determined (120); · As a response to the uncertainty (13b) satisfying a pre-given standard (130), at least one updated learning output parameter value (13a*) for the learning data record (2) is determined (140) based on one or more other output parameter values ​​(14), wherein, The one or more classifiers (1, 1a-1c) map the learning input parameter value (11a) to the one or more other output parameter values ​​after resetting to an earlier training state with epoch E1 < E2, characterized in that the learning input parameter value (11a) is mapped (111) to different output parameter values ​​(13) by multiple variants of classifier (1), the multiple variants being distinguished from each other to the extent that the variants do not consistently transform into each other as training progresses, and a separate uncertainty (13c) is determined (121) for each output parameter value (13) determined by each variant, and only a fixed number of those variants that have produced output parameter values ​​(13) with the lowest uncertainty (13c) can contribute (122) to determining the updated learning output parameter value (13a*).

2. The method (100) according to claim 1, wherein a separate updated learning output parameter value (13a*) is determined for each variant (141).

3. The method (100) according to claim 1 or 2, wherein the learning input parameter value (11a) is processed (142) into output parameter value (14) multiple times by the one or more classifiers (1, 1a-1c) in an inconsistent manner at an earlier training state with epoch E1, and wherein a summary statistic (14a) is determined (143) with respect to the output parameter value (14).

4. The method (100) according to claim 3, wherein the summary statistics (14a) are defined (144) as the updated learning output parameter values ​​(13a*).

5. The method (100) according to claim 3, wherein the learning output parameter value (13a*) updated for each variant assigns exactly one category (145) to the learning input parameter value (11a), and wherein the learning output parameter value (13a*) is distributed (146) across the variant such that the ensemble average of the variant reproduces the summary statistics (14a).

6. The method (100) according to claim 1 or 2, wherein the learning input parameter value (11a) is processed multiple times by at least one classifier (1, 1a-1c), wherein one or more neurons and / or other processing units of the classifier (1, 1a-1c) are randomly and controlled to be deactivated (117, 147).

7. The method (100) according to claim 1 or 2, wherein the accuracy by which the one or more classifiers (1, 1a-1c) maps test or effective input parameter values ​​to their respective test or effective output parameter values ​​is determined (148) based on the training state of the one or more classifiers (1, 1a-1c), and wherein the epoch with the maximum accuracy is selected (149) as epoch E1 for determining the updated learning output parameter value (13a*).

8. The method (100) according to claim 1 or 2, wherein the one or more classifiers (1, 1a-1c) are further trained (150) using a learning data record (2) having at least one updated learning output parameter value (13a*).

9. A method (200) for training a classifier (1) based on a learning data record (2), wherein the learning data record (2) includes measurement data as learning input parameter values ​​(11a), the measurement data being obtained by a physical measurement process and / or by a partial or complete simulation of such measurement process and / or by a partial or complete simulation of a technical system that can be observed using such measurement process, wherein the measurement data represents camera images, thermal images, radar data, lidar data, or ultrasonic data, wherein the learning data record (2) additionally includes learning output parameter values ​​(13a), the classifier (1) being designed to nominally map the learning input parameter values ​​(11a) to the learning output parameter values, the method comprising the following steps: Training (210) a pre-classifier (1d) to assign the correct learning output parameter value (13a) to the corresponding learning input parameter value (11a) of the learning data record (2) is distinguished from the corresponding learning input parameter value (11a) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (11a) of the learning data record (13a) of the learning data record (11a) of the learning data record (13a) of the learning data record (2) of the learning data record (2) of the learning data record (2) of the learning data record (11a) of the learning data record (13a) of the learning data record (13a) of the learning data record (11a) of the learning data record (13a) of the learning data record (13a) of the learning data record (11a) of the learning data record (11a) of the learning data record (13 ...3a) of the learning data record (11a) of the learning data record (13a) of the learning data record (13a) of the learning data record (13a) of the learning data record (13a) of the learning data record (13a) of the learning data record (11a) of the learning data record (13a) of the learning data record (13a) of the learning data record (13a) of the The learning data records (2) processed in this way are incorporated (260) into a preferred set of learning data records (2*); the classifier (1) is trained (270) using the learning data records (2) in the preferred set of learning data records (2*).

10. The method (200) according to claim 9, wherein the learning data record (2) having the updated learning output parameter value (13a*) is again fed (240) to the pre-classifier (1d), and wherein the learning data record (2) is included in the preferred set (2*) of learning data records (2) only when the pre-classifier (1d) classifies (250) the learning data record (2) as a learning data record (2) whose learning input parameter value (11a) is assigned the correct learning output parameter value (13a).

11. A method (300) for controlling (340) a vehicle (50) and / or for quality control of mass-produced products and / or a medical imaging system (70), comprising the steps of: • training (310) at least one classifier (1) based on a learning data record (2) edited using the method (100) according to any one of claims 1 to 8 and / or using the method (200) according to claim 9 or 10; • running the classifier (1) by feeding (320) input parameter values ​​(11) to the classifier (1), wherein the input parameter values ​​(11) include measurement data obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a technical system that can be observed using such a measurement process; • forming (330) a control signal (330a) based on output parameter values ​​(13) provided by the classifier (1); The control signal (330a) is used to control (340) the vehicle (50) and / or the system (60) for quality control of mass-produced products and / or the medical imaging system (70).

12. A downloadable product comprising a computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100, 200, 300) according to any one of claims 1 to 11.

13. A machine-readable data carrier having a computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100, 200, 300) according to any one of claims 1 to 11.

14. A computer equipped with computer programs and / or equipped with a machine-readable data carrier according to claim 13 and / or a download product according to claim 12, wherein, The computer program contains machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100, 200, 300) according to any one of claims 1 to 11.

Images