Method for generating labeled data, in particular for training a neural network

Abstract

Method for generating labeled data, in particular for training neural networks. The present disclosure relates to a method (100, 1000) and a device (200) for generating labeled data, e.g. training data, in particular for neural networks.

Description

Technical Field

[0001] This disclosure relates to a method for generating labels, particularly for unlabeled data. The resulting labeled data can be used, for example, as training data, especially for neural networks.

[0002] Furthermore, this disclosure relates to an apparatus for performing the first method and / or other methods. Background Technology

[0003] Machine learning, especially methods utilizing neural networks, particularly deep neural networks (DNNs), outperforms classic, untrained methods for pattern recognition in many problems. Almost all of these methods are based on supervised learning.

[0004] Supervised learning requires annotated or labeled data as training data. These annotations, also referred to as labels below, serve as the target output of the optimization algorithm. Each data element is assigned at least one label.

[0005] The quality of the labels can affect the recognition performance of a trained model in a machine learning method. Manually labeling samples used to train a machine learning method is known in the prior art. Summary of the Invention

[0006] This disclosure provides an improved method for generating tags compared to existing technologies.

[0007] One implementation relates to a method for generating labels for a dataset, the method comprising: Provide an unlabeled dataset, the unlabeled dataset comprising a first subset of unlabeled data and at least one other subset of unlabeled data that does not intersect with the first subset; The first tagged subset is generated by generating labels for the first subset and the first tagged subset is provided as the nth first tagged subset, where n=1; The iterative process is executed, wherein the nth iteration of the iterative process for each n = 1, 2, 3, ... N includes the following steps: The first model is trained using the nth labeled first subset and used as the nth trained first model; The nth labeled subset is generated by predicting the labels for other subsets using the nth trained first model; Train another model using the other subset of the nth labeled model as the nth trained model; The (n+1)th labeled first subset is generated by predicting the labels for the first subset using the nth trained model.

[0008] This method is based on training a first model and other models in an iterative manner, starting from a first subset of labeled data, and thereby progressively improving the quality of the labels predicted by the models, especially the labels themselves. Here, the ability of the trained models for generalization and / or the increased accuracy of the trained models with iterations are utilized.

[0009] Unlabeled datasets, or unlabeled data from the first and other subsets, are, for example, real-world data, particularly sensor measurements, especially multimodal data. The sensors, according to an incomplete list of examples, may be radar sensors, optical cameras, ultrasonic sensors, lidar sensors, or infrared sensors. Such sensors are commonly used in autonomous and semi-autonomous functions in motor vehicles or generally in robots.

[0010] For the initially unlabeled data in the first subset, labels are generated. The advantage of the disclosed method is that it is sufficient if labels are generated incorrectly in this step. Label generation can therefore be implemented relatively simply, and thus relatively quickly and cost-effectively.

[0011] The generation of labels for the first subset is performed, for example, using an automatic method. It can be advantageous to use untrained methods, particularly classic pattern recognition algorithms, for which no training data is required. It is also possible, in particular, to use methods trained on other datasets, without adapting them to the current dataset. Alternatively, it is conceivable to implement the method in which labels for the first subset are generated manually.

[0012] To improve the generalization of the trained model during the iterative process and to avoid systematic errors in learning the initial labels, it is particularly likely that a portion of the information from the unlabeled dataset is not used initially, especially at the beginning of the iterative process. It is especially reasonable to initially omit this information, which is important for generating the initial labels through an untrained pattern recognition algorithm. In later stages of the iterative process, the information that was initially unused can be used last. An example of this is that color information from an image is used to generate the initial labels, and color information is not initially provided during the iterative process; that is, the original color image is converted to a grayscale image. In later stages of the iterative process, color information can be added, whereby the architecture of the trainable model can be adapted accordingly to handle additional information, such as the color image instead of the grayscale image.

[0013] It can be advantageous if the first subset is roughly, especially exactly, the same size as the other subsets, or includes roughly, especially exactly the same amount of unlabeled data.

[0014] The advantage of the disclosed method lies in that, during the iterative process, at each step, the training of the first model and other models is performed using data that is disjoint from the data on which the corresponding model subsequently performs predictions. This avoids the error present in the labels at the beginning of the iteration propagating during the training of the corresponding model and continuing to propagate until the end of the iteration process. The training of the corresponding model thus starts from "zero" in each iteration. Therefore, it is not just a matter of adapting the model from the previous iteration. This improves the generalization ability of the corresponding model and the entire method and thus expands the applicability of the method. Advantageously, the alternation of two or more disjoint subsets thus improves generalization and suppresses overfitting.

[0015] The labels generated using this method can be provided along with the dataset as labeled or annotated training data for training models, especially neural networks.

[0016] According to another implementation, after the nth iteration of the iterative process, the final model is trained using the first subset of the nth labeled dataset and / or other subsets of the nth labeled dataset. Combining the first subset with other subsets can increase, in particular, double, the size of the training data used to train the final model. This advantageously improves the prediction quality of the resulting final model. The resulting model can then be used to predict labels for the entire dataset. Advantageously, by using the final model for prediction, an optimized version of the labels for the dataset can be generated.

[0017] According to another implementation, the labeled dataset and / or the final labeled dataset is generated by predicting labels for the dataset using a final model. Advantageously, by making predictions using the final model, an optimized version of the labels for the dataset can be generated, and thus an optimized labeled dataset can be generated.

[0018] According to another embodiment, a first labeled subset is generated by predicting labels using an initial model. It can be advantageous that the initial model includes an untrained model, particularly a classic pattern recognition algorithm, for which no training data is required.

[0019] According to another implementation, the initial model is trained in previous steps using a labeled initial subset, and this initial subset does not intersect with the first subset or other subsets. The labeling of the initial subset is performed, for example, manually.

[0020] According to another implementation, the initial subset is smaller than the first subset or includes a smaller amount of data than the first subset, and / or smaller than other subsets or includes a smaller amount of data than other subsets. This can be advantageous, for example, as the initial subset can be manually labeled at a relatively low cost. The generation of labels for the first subset and other subsets is then performed during the iteration process as described above.

[0021] According to another implementation, the iterative process steps are repeated as long as the quality criteria and / or termination criteria are not met. Quality criteria include, for example, the quality of the generated labels or the predictive quality of the model. Termination criteria include, for example, exceeding or falling below a threshold, particularly the number of iterations to be performed, or the value for the change in label quality from one iteration to the next, or the quality level of the label. Evaluation of the predictive quality and / or label quality can be performed, for example, based on a reference sample of good quality labeled labels. Alternatively, quality can be evaluated based on the model's confidence score, which is output in addition to the predicted labels. For this purpose, the confidence score can be normalized, for example, by using a portion of the labeled dataset obtained through prediction for this step. In this case, this portion is not used for training.

[0022] According to another embodiment, the first model and / or other models and / or the initial model and / or the final model include neural networks, especially deep neural networks.

[0023] According to another embodiment, the method further includes increasing the complexity of the first model and / or other models. It can be specified that the complexity of the first model and / or other models is increased in each iteration n, where n = 1, 2, 3, ... N. It can also be specified that the complexity of the final model is increased compared to the complexity of the first model and / or other models in the last iteration of the iterative process.

[0024] Advantageously, it can be specified that at the beginning of the iterative process, i.e., in the first iteration and in a determined number of other iterations relative to the beginning of the iterative process, the first model and / or other models are simpler in terms of the type of mathematical model and / or simpler in terms of model complexity and / or contain a smaller number of parameters to be estimated in the training range. Furthermore, it can be specified that during the iterative process, i.e., after a determined number of other iterations of the iterative process, the first model and / or other models are trained, the first model and / or other models are more complex in terms of the type of mathematical model and / or more complex in terms of model complexity and / or contain a higher number of parameters to be estimated in the training range.

[0025] Another implementation relates to a method for generating labels for a dataset, the method comprising: Provide an unlabeled dataset comprising a first subset of unlabeled data and at least one other subset of unlabeled data that does not intersect with the first subset, wherein the subsets include at least one first subset and a second subset, particularly k sub-subsets, where k = 1, 2, 3...K, and k indicates the index of the corresponding sub-subset; The initial tagged subset is generated by generating labels for the first subset; The model is trained using the initial labeled subset as the nth trained model, where n=1; The iterative process is executed, wherein the nth iteration of the iterative process for each n = 1, 2, 3, ... N includes the following steps: The nth labeled subset is generated by using the nth trained model to predict the labels for the kth subset; The model is trained using the second subset of the nth labeled set as the (n+1)th trained model. In another implementation, the (n+1)th trained model is trained using the second subset of the kth labeled set and the initial labeled set.

[0026] In one implementation, the number of iterations N in the iterative process is equal to the number of sub-subsets K. In this implementation, the iterative process will therefore continue until a label has been predicted exactly once for each sub-subset and training has been performed exactly once using each of these labeled sub-subsets.

[0027] In another implementation, the number of iterations N in the iterative process is less than the number of sub-subsets K. In this case, not all sub-subsets are used. This may be reasonable if the iterative process is terminated based on a termination criterion before applying all sub-subsets.

[0028] In another implementation, the number of iterations N in the iterative process is greater than the number of sub-subsets K. In this case, some or all sub-subsets are used more than once. For example, the (((n-1) modulo K)+1)th sub-subset can be used in the nth iteration of the iterative process.

[0029] This method is based on starting with an initial subset of labeled data, training a model iteratively, and progressively improving the quality of the labels predicted by the model, particularly the labels themselves. Here, the ability of the trained model to generalize and / or the accuracy of the trained model increasing with each iteration of the iterative process are utilized.

[0030] Unlabeled datasets, or unlabeled data from the first and other subsets, are, for example, real-world data, particularly sensor measurements, especially multimodal data. The sensors, according to an incomplete list of examples, may be radar sensors, optical cameras, ultrasonic sensors, lidar sensors, or infrared sensors. Such sensors are commonly used in autonomous and semi-autonomous functions in motor vehicles or generally in robots.

[0031] For the unlabeled data in the first subset, initial labels are generated. The generation of initial labels for the first subset is performed, for example, manually. Alternatively, it is conceivable that initial labels are generated using semi-automatic or automatic methods, particularly when using typical pattern recognition algorithms.

[0032] It can be advantageous if the first subset is smaller than the other subsets or includes data that is smaller than the data in the other subsets. Furthermore, it can be advantageous if the sub-subsets of the other subsets are approximately, especially exactly, the same size, or include approximately, especially exactly the same amount of unlabeled data.

[0033] The advantage of the disclosed method is that, in each step of the iterative process, the model is trained using data that is at least partially disjoint with the data on which the model subsequently makes predictions. This prevents errors present in the labels at the beginning of the iterative process from propagating during model training and persisting until the end of the iterative process. An unseen sub-subset is used in each iteration of the iterative process. The trained model in the corresponding iteration is thus applied to the sub-subset, which was not used to train the model in previous iterations.

[0034] To produce an improved model in each iteration, retraining is performed, using a sub-subset of the labeled data from the corresponding iteration. In another implementation, the initial subset of labeled data can additionally be used for training.

[0035] The labels generated using this method can be provided along with the dataset as labeled or annotated training data for training models, especially neural networks.

[0036] According to another implementation, after the nth iteration of the iterative process, another labeled subset is generated by predicting labels for other subsets using a trained model. This step can advantageously be performed by applying the nth trained model to the unlabeled dataset.

[0037] According to another implementation, a labeled dataset is generated, which includes an initial labeled subset and other labeled subsets. This step is advantageously performed after the nth iteration of the iterative process, especially after the end of the iterative process.

[0038] According to another implementation, a final model is trained using the labeled dataset and / or the labeled dataset is generated by predicting labels for the dataset using the final model. Advantageously, by making predictions using the final model, an optimized version of the labels for the dataset can be generated, and thus an optimized labeled dataset is generated.

[0039] According to another implementation, the iterative process steps are repeated as long as the quality criteria and / or termination criteria are not met. Quality criteria include, for example, the quality of the generated labels or the predictive quality of the model. Termination criteria include, for example, exceeding or falling below a threshold, particularly the number of iterations to be performed, or the value for the change in label from one iteration to the next, or the quality level of the label. Evaluation of the predictive quality and / or label quality can be performed, for example, based on a reference sample of good quality labeled labels. Alternatively, quality can be evaluated based on the model's confidence score, which is output in addition to the predicted labels. For this purpose, the confidence score can be normalized, for example, by using a portion of the labeled dataset obtained through prediction for this step. In this case, this portion is not used for training. The number of iterations to be performed can be predetermined, for example, by the number of sub-subsets of other subsets. According to one implementation, the iterative process can be re-executed using the already used sub-subsets after undergoing a number of iterations based on the number of sub-subsets, such that the total number of iterations is greater than the number of sub-subsets.

[0040] According to another embodiment, the model and / or the final model includes neural networks, especially deep neural networks.

[0041] According to another embodiment, the method further includes increasing the complexity of the model. It can be specified that the complexity of the model is increased in each iteration n, where n = 1, 2, 3, ... N. Advantageously, it can be specified that at the beginning of the iteration process, i.e., in the first iteration and in a determined number of other iterations relative to the beginning of the iteration process, the model is simpler in terms of the type of mathematical model and / or simpler in terms of model complexity and / or contains a smaller number of parameters to be estimated within the training range. Furthermore, it can be specified that during the iteration process, i.e., after a determined number of other iterations of the iteration process, the model is trained to be more complex in terms of the type of mathematical model and / or more complex in terms of model complexity and / or contains a higher number of parameters to be estimated within the training range.

[0042] Another embodiment relates to an apparatus configured to implement one or more methods according to the embodiments described above.

[0043] According to another embodiment, the device includes a computing device and a storage device, particularly for storing at least one model, especially a neural network.

[0044] Another embodiment relates to a computer program, wherein the computer program includes computer-readable instructions that, when executed by a computer, implement one or more methods according to the embodiments described above.

[0045] Another embodiment relates to a computer program product, wherein the computer program product includes a computer-readable storage medium on which a computer program according to the embodiment is stored.

[0046] Another embodiment relates to an application for generating training data, which is used to train a model, particularly a neural network, according to at least one method and / or device according to the embodiments described above and / or computer program and / or computer program product according to the embodiments.

[0047] Another embodiment relates to the application of labeled datasets, wherein the labeled datasets are generated using the method and / or the device and / or the computer program and / or the computer program product according to the embodiment, for training models, particularly neural networks.

[0048] For example, it can be envisioned for use in classification methods, recognition methods, especially biometric methods, especially speech recognition, facial recognition, iris recognition, or especially object detection, object tracking, etc.

[0049] This approach can also be envisioned for use in "lifelong learning" programs. A key feature of this approach is the ability to conduct further training during the course of the method's use.

[0050] This method is particularly suitable for labeling data recorded by sensors, such as cameras, lidar sensors, radar sensors, and ultrasonic sensors. Preferably, the data labeled using this method is used to train pattern recognition algorithms, especially object recognition algorithms. These pattern recognition algorithms can be used to manipulate different technological systems and, for example, to achieve medical advancements in diagnostics. In particular, object recognition algorithms trained with labeled data are suitable for use in control systems, especially driving functions, and in at least semi-automated robots. Thus, the object recognition algorithm can be used, for example, in industrial robots to selectively handle objects, transport objects, or activate safety functions, such as deactivation, based on specific object categories. For automated robots, especially automated vehicles, such object recognition algorithms can be advantageously used to improve or implement driving functions. In particular, lateral and / or longitudinal guidance of robots, especially automated vehicles, can be performed based on object recognition algorithms. Different driving functions, such as emergency braking or lane keeping functions, can be improved by using these object recognition algorithms. Attached Figure Description

[0051] Other features, applications, and advantages of the invention will emerge from the subsequent description of embodiments of the invention, illustrated in the accompanying drawings. Here, all described or presented features, either alone or in any combination, constitute the subject matter of the invention, regardless of their combination in the claims or their reference thereto, or their representation or illustration in the specification or drawings. Here, method steps are schematically represented as rectangles, data as cylinders, transitions between method steps and data as arrows, and data flows as dashed arrows.

[0052] In the attached diagram: Figure 1 A flowchart illustrating the steps of the first method; Figure 2 A block diagram illustrating a first method according to a preferred first embodiment is provided. Figure 3 A block diagram illustrating the method according to another preferred embodiment is provided. Figure 4 A flowchart illustrating the steps of another method; Figure 5 A block diagram illustrating another method according to one embodiment is provided. Figure 6 A block diagram illustrating the other method according to another embodiment; and Figure 7 A simplified block diagram illustrates a device according to a preferred embodiment. Detailed Implementation

[0053] Figure 1 A schematic diagram illustrates the steps of a method 100 for generating labels, particularly the final label L_f, for a dataset S. The method 100 includes the following steps: Step 110 for providing an unlabeled dataset S, the dataset comprising a first subset SA of unlabeled data and at least one other subset SB of unlabeled data that does not intersect with the first subset; Step 120 is used to generate the labeled first subset SA_L_1 by generating a label L_A_1 for the first subset SA. And step 130, which is used to provide the first subset SA_L_1 of the labels as the first subset SA_L_n of the nth label, where n=1; Step 140 for performing the iterative process, wherein for each n = 1, 2, 3, ... N, the nth iteration of the iterative process includes the following steps: Step 141n is used to train the first model MA using the nth labeled first subset SA_L_n as the nth trained first model MA_n; Step 142n is used to generate the nth labeled subset SB_L_n by predicting the label L_B_n for the other subset SB using the trained model MA_n; Step 143n is used to train another model MB using the nth labeled subset SB_L_n as the nth trained other model MB_n; Step 144n is used to generate the (n+1)th labeled first subset SA_L_n+1 by predicting the label L_A_n+1 for the first subset SA using the nth other trained model MB_n.

[0054] According to the presented implementation, the method further includes a step of training a final model M_f using a first subset SA_L_n with nth labels and / or another subset SB_L_n with nth labels. Step 150 is performed after the nth iteration of the iterative process.

[0055] According to the presented implementation, the method further includes step 160 for generating a labeled dataset S_L_f by predicting the label L_f for the dataset S using the final model M_f.

[0056] The advantage of Method 100 is that, during the iterative process 140, in each iteration, training of the untrained first model and other models MA and MB is performed using data that is disjoint from the data on which the corresponding trained models MA_n and MB_n are subsequently used for prediction. This avoids the errors present in labels L_A_1 and L_B_1 to L_A_N and L_B_N, especially at the beginning of the iteration, from propagating during the training of the corresponding models MA and MB and continuing to propagate until the end of the iteration process. The training of the corresponding models MA and MB therefore starts from "zero" in each iteration. Thus, it is not just a matter of adapting the models MA_n-1 and MB_n-1 from the previous iterations. This improves the generalization ability of the corresponding models MA and MB and the entire Method 100 and thus expands the applicability of the Method 100. Advantageously, the alternation of two disjoint subsets SA and SB thus improves generalization and suppresses overfitting.

[0057] A first embodiment of the method 100 is referred to below. Figure 2 Let me explain.

[0058] The unlabeled dataset S consists of a first subset SA of unlabeled data and other subsets SB of unlabeled data that do not intersect with the first subset.

[0059] According to the presented implementation, the first subset SA and other subsets SB are approximately, especially exactly the same size, or include approximately, especially exactly the same amount of unlabeled data.

[0060] In step 120, the first tagged subset SA_L_1 is generated by generating a label L_A_1 for the first subset SA, and is provided in step 130 as the nth tagged first subset SA_L_n, where n=1.

[0061] The generation of labels L_A_1 for the first subset SA is performed, for example, using an automatic method. This automatic method, according to this embodiment, may include untrained methods, particularly classic pattern recognition algorithms, for which no training data is required.

[0062] As long as the unlabeled data S is time-dependent, this automatic method can, for example, perform offline processing. Here, for example, single-frame methods operating on individual frames, particularly deep learning-based object recognition and methods utilizing temporal consistency, such as tracking combinations, can also be used.

[0063] Alternatively, the generation of the label L_A_1 for the first subset SA can also be done manually.

[0064] For better understanding, the steps of the iterative process 140 are enclosed in boxes with reference numerals 140.

[0065] Each nth iteration of the iterative process includes the following steps for each n = 1, 2, 3, ... N: In step 141n, the first model MA is trained using the nth labeled first subset SA_L_n, which is then used as the nth trained first model MA_n; In step 142n, the nth labeled subset SB_L_n is generated by predicting the label L for the other subset SB using the trained model MA_n; In step 143n, model MB is generated using the nth labeled subset SB_L_n generated in step 142n, which is used as the nth trained model MB_n. In step 144n, the (n+1)th labeled first subset SA_L_n+1 is generated by predicting the label L_n+1 for the first subset SA using the nth trained other model MB_n.

[0066] During the iterative process 140, in each step, the training of the first model and other models MA and MB is performed using data that does not intersect with the data for which the corresponding models MA_n and MB_n subsequently perform predictions.

[0067] The quality of the labels is improved or the error rate is reduced by going through multiple iterations of the process 140.

[0068] For example, the iterative process 140 is implemented as long as the quality criteria and / or termination criteria are not met. Quality criteria include, for example, the quality of the resulting labels L_A_n and L_B_n, or the prediction quality of the first model and / or other models MA, MB. Termination criteria include, for example, exceeding or falling below a threshold, particularly the number of iterations to be performed, or, for example, the value calculated based on the interval (Abstandsmaß) for the change in label L_A_n and / or L_B_n from one iteration to the next, or the quality level for label L_A_n and / or L_B_n. The evaluation of prediction quality and / or the quality of label L_A_n and / or L_B_n can be performed, for example, based on a well-quality labeled reference sample. Alternatively, quality can be evaluated based on the confidence level of the first model and / or other models MA, MB, which is output in addition to the predicted labels L_A_n and / or L_B_n. For this purpose, the confidence score can be normalized, which can be performed, for example, by using a portion of the labeled dataset obtained through prediction for this step. In this case, this portion is not used for training.

[0069] Following this iterative process, the final training process can be performed. The final training process includes, for example, the following: Figure 1 Steps 150 and 160. According to the presented implementation, the first subset SA_L_N+1 of the (N+1)th label from the last Nth iteration and the other subsets SB_L_N of the Nth label are used to train the final model M_f. The final model M_f is trained, for example, on the labels L_A_N and L_B_N of the last Nth iteration.

[0070] Next, in step 160, the labeled dataset S_L is generated by predicting the label L_f for dataset S using the final model M_f.

[0071] Figure 3 Another implementation of method 100 is shown. The unlabeled dataset S includes an initial subset SC in addition to the first and second subsets SA and SB. The initial subset SC may be disjoint from the first and other subsets SA and SB, but it is also possible that SC is a subset of SA and / or SB.

[0072] In step 112, the label L_C for the initial subset SC is generated, in particular, by means of a manual method.

[0073] In step 114, the initial model MC is trained using the labeled initial subset SC_L.

[0074] According to this implementation, the first subset SA_L_n is generated by predicting labels using the trained initial model MC 120.

[0075] Advantageously, the initial subset SC is smaller than the first subset SA and / or smaller than other subsets SB, or includes a smaller amount of data than the first subset SA and / or other subsets SB. This can be demonstrated as advantageous, for example, that the initial subset SC can be manually labeled at a relatively low cost.

[0076] Labels for the first and other subsets SA and SB are generated using references. Figure 1 and Figure 2 The described iterative process 140 is performed.

[0077] When SC is a subset of SA and / or SB, new labels are predicted during iterations of the iterative process for those dataset elements (samples of samples) that appear not only in SC but also in SA and / or SB. In one implementation, these predicted labels can replace the initial label L_C, so that the initial label is used only at the beginning. In another implementation, some of the initial labels can be used in the iterative steps of the iterative process, while some of the initial labels are replaced by the predicted labels. Which of these labels are used unchanged and which are replaced can be changed during the progress of the iterative process, in particular the ratio of labels used unchanged to labels replaced can be changed. This yields the advantage that if the initial label is erroneous and the quality of the predicted labels exceeds the quality of the initial iterations during the progress of the iterative process, a larger share of the unchanged initial label is used at the beginning of the iterative process and a larger share of the predicted label is used in further progress. Conversely, if the quality of the initial label is high, the initial label can generally be used in all iterative steps of the iterative process, so that predictions for those dataset elements that appear not only in SC but also in SA and / or SB do not need to be performed during iterations. In this case, the labeled sample SC_L can be referred to as the reference sample.

[0078] Figure 4 A schematic diagram illustrates the steps of another method 1000 for generating labels L_f for dataset S. Method 1000 includes the following steps: Step 1100 for providing an unlabeled dataset S, wherein the unlabeled dataset comprises a first subset SA of unlabeled data and at least one other subset SB of unlabeled data that does not intersect with the first subset. The subset SB includes at least one first subset SB_1 and a second subset SB_2, and in particular K sub-subsets SB_k, where k = 1, 2, 3...K; Step 1200 is used to generate the initial labeled subset SA_L by generating a label L_A for the first subset SA. Step 1300 is used to train model M as the nth trained model M_n using the initial labeled subset SA_L, where n=1; Step 1400 for performing the iterative process, wherein the nth iteration of the iterative process for each n = 1, 2, 3, ... N includes the following steps: Step 1410n is used to generate the nth labeled subset SB_n_L by using the nth trained model M_n to predict the label L_n for the nth subset SB_n; Step 1420n is used to train model M as the (n+1)th trained model M_n+1 using only the nth labeled sub-subset SB_n_L and the initial labeled subset SA_L. In another alternative implementation, step 1420n includes training model M as the (n+1)th trained model M_n+1 using only the nth labeled sub-subset SB_n_L but without the initial labeled subset SA_L.

[0079] According to the presented implementation, method 1000 further includes a step 1500 for generating a labeled subset SB_L by predicting a label L for the other subset SB. This step is advantageously performed by applying the nth trained model M_n to the unlabeled subset SB, which advantageously includes sub-subsets SB_1, SB_2, ..., SB_n. Step 1500 is performed after the nth iteration of iterative process 1400. The final Nth iteration of the iterative process produces model M_N+1 in step 1420N.

[0080] According to the presented implementation, step 1500 further includes generating a labeled dataset S_L, which includes an initial labeled subset SA_L and other labeled subsets SB_L.

[0081] According to the presented implementation, the method 100 further includes, particularly optionally, a step 1600 for training a final model M_f using a labeled dataset S_L, and particularly optionally, a step 1700 for generating a final labeled dataset S_L_f by predicting labels L for dataset S using the final model M_f. Advantageously, an optimized version of the labels for dataset S can be generated by prediction using the final model M_f, and thus an optimized labeled dataset S_L_f is generated.

[0082] A first embodiment of the method 1000 is referred to below. Figure 5 Let me explain.

[0083] The unlabeled dataset S includes a first subset SA of unlabeled data and other subsets SB of unlabeled data. Subsets SB include n subsets SB_n, where n = 1, 2, 3...N, especially the first subset SB_1, the second subset SB_2, and so on.

[0084] It can be shown to be advantageous if the first subset SA is smaller than the other subsets SB or if the first subset SA includes a smaller number of data than the other subsets SB. Furthermore, it can be shown to be advantageous if the sub-subsets SB_1, SB_2, ..., SB_n of the other subsets SB are approximately, especially exactly, the same size, or include approximately, especially exactly the same number of unlabeled data.

[0085] In step 1200, an initial labeled subset SA_L is generated by generating labels L for the first subset SA. The initial labels for the first subset SA are generated, for example, manually. Alternatively, it is conceivable that the initial labels L are generated semi-automatically or automatically, particularly when using classic untrained pattern recognition algorithms or when using trained models.

[0086] In step 1300, model M is trained using the initial labeled subset SA_L, which is the nth trained model M_n, where n=1.

[0087] For better understanding, the steps of the iterative process 1400 are enclosed in boxes with reference numeral 1400.

[0088] The iterative process 1400 includes the following steps for each n=1, 2, 3, ... N: Step 1410n is used to generate the nth labeled subset SB_n_L by using the nth trained model to predict the label L for the nth subset SB_n; Step 1420n is used to train model M as the (n+1)th trained model M_n+1 using the nth labeled subset SB_n_L and the initial labeled subset SA_L.

[0089] For example, the iterative process 1400 is implemented as long as the quality criteria and / or termination criteria are not met. The quality criteria include, for example, the quality of the resulting label L or the prediction quality of the model M_n. The termination criteria include, for example, exceeding or falling below a threshold, particularly the number of iterations to be performed or the value of the change in label L from one iteration to the next, or the quality level of label L. The evaluation of the prediction quality and / or the quality of label L can be performed, for example, based on a reference sample with good quality labels. Alternatively, the quality can be evaluated based on the confidence score of model M_n, which is output in addition to the predicted label L. For this purpose, the confidence score can be normalized, where normalization can be performed, for example, by using a portion of the labeled dataset obtained through prediction for this step. In this case, this portion is not used for training. The number of iterations n to be performed can be given in advance, for example, via the number n of sub-subsets SB_n of other subsets SB.

[0090] Figure 6 Show Figure 5 Method 1000, in which Figure 6 The implementation of method 1000 presented herein extends the final training process. Final training can be performed after the iterative process. The final training process includes, for example, […]. Figure 4 Steps 1500, 1600, and 1700 are described below. In step 1500, a labeled subset SB_L is first generated by predicting the label L for the other subset SB. According to the presented implementation, step 1500 further includes generating a labeled dataset S_L, which includes the initial labeled subset SA_L and the other labeled subset SB_L.

[0091] In step 1600, the final model M_f is trained using the labeled dataset S_L. In step 1700, the final labeled dataset S_L_f is generated by predicting the label L for dataset S using the trained final model M_f. Advantageously, an optimized version of the label for dataset S can be generated by predicting the label L using the final model M_f, and thus an optimized labeled dataset S_L_f can be generated.

[0092] Optionally, in Figures 4 to 6 The implementation of the method presented herein may include weighted training of model M.

[0093] For example, it can be stipulated that the data of the nth labeled subset SB_n_L and the data of the initial labeled subset SA_L are weighted so that the model M is trained as the (n+1)th trained model M_n+1.

[0094] For example, it can be advantageous to weight the data in the initial labeled subset SA_L more heavily than the data in the nth labeled subset SB_n_L. The weighting can also be changed during iterations of iteration process 1400. In this context, it can be advantageous to weight the data in the initial labeled subset SA_L more heavily at the start of iteration process 1400 than in later iterations.

[0095] Alternatively, it can be specified that model M is suitable for and used to generate confidence scores. These confidence scores can then be used to weight the data in the nth labeled subset SB_n_L. For this purpose, the confidence scores can be normalized, for example by using a portion of the labeled dataset obtained through prediction. In this case, this portion is not used to train model M.

[0096] Advantageously, by weighting, incorrectly predicted labels have a less severe impact, especially at the beginning of iteration 1400, making it possible to achieve a high quality of the labeled sub-subset SB_n_L with a small number of iterations.

[0097] Optionally, in Figures 1 to 6 In the implementations of methods 100 and 1000 presented herein, steps may be included to increase model complexity.

[0098] Advantageously, it can be specified that the complexity of models M, MA, and MB is increased during iterations. Advantageously, it can be specified that the complexity of models M, MA, and MB is increased in each iteration n, where n = 1, 2, 3, ... N.

[0099] According to one implementation, it can be specified that at the beginning of iteration processes 140, 1400, i.e., in the first iteration and in a certain number of further iterations relative to the beginning of iteration processes 140, 1400, models M, MA, MB are trained, said models being simpler in terms of the type of mathematical model and / or simpler in terms of model complexity and / or containing a smaller number of parameters to be estimated in the training range.

[0100] One specific implementation is illustrated by applying methods 100 and 1000 to a classification problem using the Expectation-Maximization (EM) algorithm. The EM algorithm estimates the class-specific distribution of data in dataset S, or the class-specific distribution of features computed from the data in dataset S. For example, when using Bayes' theorem, classification is based on maximizing class-specific probabilities. The EM algorithm can, for example, be used to estimate the parameters of a Gaussian mixture distribution. When using a Gaussian mixture distribution, model complexity can be increased by increasing the number of Gaussian distributions estimated for each mixture (and therefore for each class). In this example, a relatively small number of Gaussian distributions will therefore be used at the beginning of the iteration process, and this number will always be further increased during iterations.

[0101] Another specific implementation is illustrated by applying this method 100, 1000 in cases where a deep neural network (DNN) is used as model M, MA, MB. In this case, the model complexity can be altered by the architecture of the neural network. The larger the number of layers and the larger the number of neurons in each layer, the more parameters are estimated during training, and therefore the higher the complexity of the neural network. In specific cases, the type of association between layers can also play a role.

[0102] Generally, increasing the complexity of models M, MA, and MB can be improved, particularly by increasing the number of parameters to be estimated during training and the model's ability to fit the training data, i.e., the distribution of the learning data. This advantageously leads to better recognition performance. In some cases, high complexity of models M, MA, and MB can also lead to poorer generalization ability and so-called overfitting to the training data. While recognition performance on training data increases further with increased model complexity, recognition performance on unseen test data decreases. The less data available for training, the more likely overfitting is to be a problem.

[0103] In the methods 100 and 1000 disclosed herein, this effect may be significant because the labels L used for training, such as L_A_n, L_B_n, L_f, have more errors at the beginning of the iterative process compared to after repeated iterations. Therefore, the recognition performance obtained at the beginning of the process may be worse than the recognition performance at the end of the process. Thus, it may be advantageous, for example, to achieve good generalization ability and avoid overfitting at the beginning of the process. It may also be advantageous, if necessary, to tolerate a certain error rate due to the relatively low complexity of models M, MA, and MB. As the iterative process progresses, the quality of labels L improves, making more training data of better quality available. After a certain quality of labels L is achieved, it can then be advantageous to continuously increase the complexity of models M, MA, and MB. The higher complexity of models M, MA, and MB, given training data of a certain quality, generally also leads to further improvements in recognition performance.

[0104] In the determining steps of the iterative method, the error rate can be used as a criterion for determining the appropriate complexity of models M, MA, and MB. In particular, comparing the error rate of the predicted label L with the error rate of the determined training samples can be advantageous. If the error rate of the predicted label L is worse, it may be advantageous to adjust the complexity of models M, MA, and MB accordingly.

[0105] at last, Figure 7 The device 200 is shown, wherein the device 200 is configured to implement the method 100 and / or method 1000 according to the described embodiments.

[0106] The device 200 includes a computing device 210 and a storage device 220, particularly for storing models, especially neural networks. In this example, the device 210 includes an interface 230 for inputting and outputting data, particularly for inputting data from a dataset S and / or labels L and / or a first labeled subset SA_L_1, and for outputting the generated labels L and / or the final labeled dataset S_L_f. The computing device 210, storage device 220, and interface 230 are connected via at least one data line 240. The computing device 210 and storage device 220 can be integrated into a microcontroller. The device 200 can also be configured as a distributed system within a server infrastructure.

[0107] According to the embodiment, computing device 210 can access storage device 220a, on which computer program PRG1 is stored, wherein computer program PRG1 includes computer-readable instructions that, when implemented by a computer, particularly by computing device 210, implement method 100 and / or method 1000 according to the embodiment.

[0108] The exemplary use of methods 100 and 1000 is explained below using their application in a system for biometric speech recognition as an example. Application in other biometric methods, particularly facial recognition devices, iris recognition, etc., is equally conceivable.

[0109] First, a small set of speech material in the form of a dataset S, initially unknown to the user, is provided. Based on this set, the system can learn the user in a first learning process (the so-called "registration"). For example, the labeling of a subset SA_L, or SA_L_1 or SC_L depending on the implementation, can be performed by the user themselves. By applying this method 100, 1000, subsequently labeled subsets SA_L_n, SB_L_n, or SB_n_L are generated, and if necessary, a finally labeled dataset S_L or S_L_f is generated.

[0110] Application example: Environmental perception of autonomous or partially autonomous vehicles Another specific example of the application of methods 100 and 1000 is environmental perception for autonomous or partially autonomous driving. In this case, the vehicle is equipped with at least one sensor that detects stationary (i.e., immovable) and dynamic (i.e., movable) objects in the vehicle's environment. Advantageously, the vehicle can be equipped with multiple sensors, especially sensors of different modalities, such as a combination of cameras, radar sensors, lidar sensors, and / or ultrasonic sensors. This involves a multi-mode sensor array. The vehicle equipped in this way is used to record an initial unlabeled sample of sensor data and store it as a dataset D. The purpose of perception or environmental perception is to identify and locate stationary and dynamic objects in the vehicle's environment, and thus generate symbolic representations of these objects, including temporal variations. These symbolic representations are typically given by partially temporally relevant attributes of these objects, such as object type (e.g., bus, truck, pedestrian, cyclist, guardrail, drivable or non-drivable objects), lane marking attributes, and other attributes such as the number of axles, size, shape, position, orientation, speed, acceleration, state of the driving direction indicator, etc.

[0111] The trained models MA and MB used for object identification and determination of relevant attributes can be a combination of at least one sub-model for one of the sensor modalities and another fusion sub-model. The fusion sub-model can include a deep neural network architecture to fuse data over time and / or fuse data from different sensors and / or modalities. In particular, the models MA and MB can be an architecture comprising one or more single-frame sub-networks, each of which performs single-frame recognition. The fusion sub-networks that can be included in the overall architecture can fuse the outputs of the single-frame sub-networks and achieve fusion over time and / or multi-modal fusion. In particular, recurrent neural networks (RNNs) can be used for the architecture of the fusion sub-networks used for fusion over time. LSTM (Long Short-Term Memory) architectures can also be used in this case to better incorporate information into the recognition over longer time intervals. Fusion over time and / or fusion of different sensors and / or fusion of different sensor modalities can be implemented in deep neural networks as early fusion, in which input variables are combined into the neural network in a suitable manner, such as as multiple channels (similar to the concept of an RGB image). Alternatively, fusion can be implemented as late fusion, in which the outputs of multiple independent neural networks are combined in a suitable manner, such as by averaging. Fusion can also be achieved through an intermediate form between these two types (middle fusion), where more complex features of multiple subnetworks are generally combined in a suitable manner, such as as multiple channels or by addition. Alternatively, fusion over time and / or fusion of different sensors and / or fusion of different sensor modalities can also be achieved using untrained schemes, particularly using Kalman filters.

[0112] A single-frame subnetwork for recognizing a single camera frame, i.e., a single image from a camera, can be, for example, a convolutional deep neural network. Similarly, a single-frame subnetwork for recognizing point clouds, such as a single sensor scan from a LiDAR sensor or a radar sensor scan, can also be a convolutional deep neural network, which either takes a 2D projection of the point cloud as input data or performs 3D convolution in the case of a 3D-CNN, where the point cloud is then represented in a regular 3D mesh. Alternatively, a deep neural network with an architecture of PointNet or PointNet++ can be involved, where the point cloud can be processed directly. The training of model MA in step 141n can be performed according to the label L_A_n. The training of model MB in step 143n can be performed according to the label L_B_n. In this case, when training a subnetwork independently of the fusion network or separately from the fusion network, attribute transformations can be performed according to the corresponding modality. For example, the 3D position of the tracked object can be projected into the camera image, i.e., projected into a 2D bounding box.

[0113] Instead of using a trained fusion method over time, objects detected in individual frames can be tracked over time, for example, using a Kalman filter or an extended Kalman filter. In this case, at least one single-frame sub-model can be included in the overall model MA, MB. The single-frame sub-model can be used to predict the relevant properties of the objects. Based on these predicted properties, the identified objects can be associated with each other by comparison with the predicted properties of objects already known in previous time steps, where the prediction of known objects can be made at the corresponding measurement time. This prediction can be based on a physical motion model. The properties predicted by the single-frame sub-model can be used to update the properties of the associated objects predicted at the measurement time. The result of the update is the predictions 142n and 144n of the overall model MA, MB, which in this example include a sub-model with a Kalman filter or an extended Kalman filter and a trained single-frame model, respectively.

[0114] If the fusion over time utilizes a trainable model, particularly a DNN, the architecture can be chosen such that sensor data not only before but also after a specific moment is available to the DNN for recognizing the environment at that moment. This thus involves an offline version of the recognition system. The additional use of information about the future state of the estimated data can improve the accuracy and reliability of the recognition, which is an advantage of methods implemented in this form.

[0115] If the fusion is achieved using an untrained model instead, methods for offline processing are also possible. For example, a Kalman smoother, such as a Rauch-Tung-Striebel filter, can be used instead of a Kalman filter.

[0116] After the iterative process is fully implemented, a perception system exists in two scenarios—not only when using a trained fusion architecture but also when using an untrained method—that is trained at least partially using the labeled data from the last iteration. This system can be used for offline perception to label other sensor data not used in the iterative process. In this way, additional labeled samples can be generated automatically. If the offline tracking or trained offline architecture used for fusion in this perception system is replaced by tracking with online capabilities or a trainable architecture with online capabilities, then the online-capable perception system can be used in vehicles for environmental perception to achieve autonomous driving functions.

[0117] For example, the Rauch-Tung-Striebel smoother can be replaced by a Kalman filter without a smoother, and the same trained model can continue to be used at the single-frame level. To reduce the computational requirements, a trained single-frame model with reduced complexity can also be used for the online version of the perception system. This single-frame model can be trained based on the labels generated in the final iteration of the iterative process or the final labels of the dataset S_L_f, and / or can be compressed and pruned by the trained model MN or MB_N from the final iteration or by the final trained model M_f.

[0118] When using a trained fusion architecture, the architecture can be modified so that it uses only sensor data from the past to identify the state of an object at a given moment, thus creating a system with online capabilities in this case.

[0119] The iterative process described for offline and online perception in achieving autonomous driving capabilities can also be similarly applied to other robots. For example, the iterative process can be applied to achieve environmental perception in home robots, care robots, construction robots, or garden robots.

[0120] Other application examples: medical image recognition and biometric identification Another application of method 100 and / or method 1000 and / or the tags generated by method 100 or method 1000 can be particularly found in systems used for pattern recognition, especially object detection, object classification and / or segmentation, especially in the field of medical image recognition, such as segmenting or classifying medical images, and / or in the field of biological person identification. This application is illustrated below with two separate examples: on the one hand, classifying medical diseases based on X-ray images, computed tomography (CT) images, or magnetic resonance imaging (MRT) images; and on the other hand, locating faces in images as elements of a biological system for verifying or identifying persons.

[0121] In these examples, the method is applied in the following manner: First, samples of images of the corresponding domain are recorded, which are initially unlabeled datasets S. Thus, for example, samples of CT images of specific human organs are obtained; in the second example, samples of photographs with faces are obtained. In the case of samples of facial images, it may be advantageous to use video sequences instead of individual photographs, because the method can then be used in conjunction with tracking over time, as described in application examples for environmental perception of autonomous or semi-autonomous robots.

[0122] Step 120, generating initial, erroneous labels, can be performed using relatively simple heuristics in both application examples to obtain initial labels for image segmentation and / or classification. Specific examples include pixel-level segmentation based on simple thresholds of corresponding luminance and / or chrominance values ​​and / or rule-based classification based on the distribution of all luminance or chrominance values ​​across the entire image and / or the segmented image. In the case of face localization, rule-based segmentation of the image can be performed based on typical skin color. Alternatively, manual labeling is possible in both application cases, where this can be performed relatively quickly and at low cost due to the low requirements for the quality of the initial labels.

[0123] The models MA and MB, trained during the iterative process and used for prediction, can be convolutional deep neural networks. In classification applications, one-hot encoding of the output layer can be used. For face recognition applications, which are specifically for object detection, one of the following deep neural network architectures can be used for models MA and MB: YOLO ("You Only Look Once"), R-CNN ("Region Proposal CNN"), Fast R-CNN, Faster R-CNN, and / or RetinaNet.

[0124] Since the initial labels are generated based on color information, generalization can be improved by removing color information from the image at the beginning of the iteration process; that is, training and prediction are initially performed based solely on grayscale images in the iterative steps. Further in the iteration process, especially when the initially incorrectly labeled portions of the image as "faces" no longer cause false positives in the CNN, color information can be added again, allowing the entire set of information to be utilized.

[0125] In applications involving face localization in images, when video sequences exist in dataset S, the data can be analyzed based on... Figure 4 The method is combined with tracking over time.

Claims

1. A method (100) for generating labels (L) for a dataset (S), wherein the dataset is image data from a sensor, the method (100) comprising: Provide (110) an unlabeled dataset (S), the unlabeled dataset comprising a first subset (SA) of unlabeled data and at least one other subset (SB) of unlabeled data that does not intersect with the first subset; A first tagged subset (SA_L_1) is generated by generating a label (L_A_1) for the first subset (SA) and the first tagged subset (SA_L_1) is provided (130) as the first tagged subset, wherein the label is used to classify and / or segment the image data; Performing the (140) iterative process, wherein the nth iteration of the iterative process for each n = 1, 2, 3, ... N includes the following steps: Use the nth labeled first subset (SA_L_n) to train (141n) the first model (MA) as the nth trained first model (MA_n); (142n) The nth labeled subset (SB_L_n) is generated by predicting the label (L_B_n) for the other subset (SB) using the nth trained first model (MA_n); Use the nth labeled subset (SB_L_n) to train (143n) other models (MB) as the nth trained other model (MB_n); The (144n)th (n+1)th labeled first subset (SA_L_n+1) is generated by predicting the label (L_A_n+1) for the first subset (SA) using the nth trained other model (MB_n).

2. The method (100) according to claim 1, wherein after the nth iteration of the iterative process (140), the final model (M_f) is trained (150) using the first subset (SA_L_n) of the nth labeled and / or the other subset (SB_L_n) of the nth labeled.

3. The method (100) of claim 2, wherein a labeled dataset (S_L) and / or a final labeled dataset (S_L_f) are generated (160) by predicting a label (L_f) for the dataset (S) using the final model (M_f).

4. The method (100) according to any one of claims 1 to 3, wherein the generation (120) of the first labeled subset (SA_L_1) is performed by predicting the labels using an initial model (MC).

5. The method (100) of claim 4, wherein the initial model (MC) is trained (114) in a previous step using a labeled initial subset (SC_L), wherein the initial subset (SC) does not intersect with the first subset and the other subsets (SA, SB).

6. The method (100) according to claim 5, wherein the initial subset (SC) is smaller than the first subset (SA) and / or smaller than the other subsets (SB).

7. The method (100) according to any one of claims 1 to 3, wherein the steps (141n, 142n, 143n) of the iterative process (140) are repeatedly performed as long as the quality standard and / or termination standard are not met.

8. The method (100) according to claim 2 or 3, wherein the first model (MA) and / or the other models (MB) and / or the initial model (MC) and / or the final model (M_f) comprises a neural network.

9. The method (100) according to claim 2 or 3, wherein the method (100) further comprises: Increase the complexity of the first model and / or the other models and / or the final model.

10. The method (100) of claim 8, wherein the neural network is a deep neural network.

11. A method (1000) for generating labels (L) for a dataset (S), wherein the dataset is image data from a sensor, the method comprising: Provide (1100) an unlabeled dataset (S), the unlabeled dataset comprising a first subset (SA) of unlabeled data and at least one other subset (SB) of unlabeled data that does not intersect with the first subset, wherein the subset SB comprises k sub-subsets SB_k, where k = 1, 2, 3…K; The initial labeled subset (SA_L) is generated by generating a label (L_A) for the first subset (SA), and (1200) the initial labeled subset (SA_L). The initial labeled subset (SA_L) is used to train (1300) model (M) as the first trained model, wherein the labels are used to classify and / or segment the image data; Performing the (1400) iterative process, wherein the nth iteration of the iterative process for each n = 1, 2, 3, ... N includes the following steps: The nth labeled subset (SB_n_L) is generated by using the nth trained model (M_n) to predict the label (L_B_n) for the kth subset SB_K. The model (M) is trained (1420n) using the nth sub-subset of the labels (SB_n_L) and / or the initial subset of the labels (SA_L) as the (n+1)th trained model (M_n+1).

12. The method (1000) of claim 11, wherein after the Nth iteration of the iterative process (1400), the final labeled subsets SB_L_f and / or SA_L_f are generated (1500) by predicting labels L_B_f and / or L_A_f for the other subsets (SB) and / or for the first subset (SA) using the trained model (M_N).

13. The method (1000) of claim 11 or 12, wherein generating (1500) a labeled dataset S_L comprising the initial labeled subset (SA_L) and other labeled subsets (SB_L_f).

14. The method (1000) of claim 13, wherein the final model (M_f) is trained (1600) using the labeled dataset (S_L) and / or the labeled dataset (S_L_f) is generated (1700) by predicting labels (L_S_f) for the dataset (S) using the final model (M_f).

15. The method (1000) according to claim 11 or 12, wherein the steps (1410n, 1420n) of the iterative process (1400) are repeatedly performed as long as the quality standard and / or termination standard are not met.

16. The method (1000) of claim 14, wherein the model (M) and / or the final model (M_f) comprises a neural network.

17. The method (1000) according to claim 14, wherein the method (1000) further comprises: Increase the complexity of the model (M) and / or the final model (M_f).

18. The method (1000) according to claim 16, wherein the neural network is a deep neural network.

19. An apparatus (200) configured to perform the method (100) according to any one of claims 1 to 10 and / or the method (1000) according to any one of claims 11 to 18.

20. The device (200) according to claim 19, wherein the device (200) includes a computing device (210) and a storage device (220).

21. The device (200) according to claim 20, wherein the storage device (220) is used to store at least one model (M, MA, MB, MC, M_f).

22. The device (200) according to claim 20 or 21, wherein the storage device (220) is used to store a neural network.

23. A computer program product comprising computer-readable instructions that, when implemented by a computer, perform the method (100) according to any one of claims 1 to 10 and / or the method (1000) according to any one of claims 11 to 18.

24. A computer-readable storage medium on which a computer program product according to claim 23 is stored.

25. The use of the method (100) according to any one of claims 1 to 10 and / or the method (1000) according to any one of claims 11 to 18 and / or the apparatus (200) according to any one of claims 19 to 22 and / or the computer program product according to claim 23 and / or the computer-readable storage medium according to claim 24 for generating training data, wherein the training data is used to train a model.

26. The use according to claim 25, wherein the training data is used to train a neural network.

Images