Methods for improving synthetic ground truth data

Abstract

Method and apparatus (100) for improving synthetic ground truth data by a data generator and for training a target machine learning model, the method comprising the steps: - Providing (S1) ground truth data samples synthetically generated by the data generator to ground truth source data; - Comparing (S2) a performance of the data generator for the provided ground truth data samples with a performance threshold; - Regeneration (S3) of ground truth data samples for the same ground truth source data by the data generator if the performance threshold for the provided ground truth data samples is exceeded; - replacing (S4) the ground truth data samples for which the performance threshold is exceeded with the newly generated ground truth data samples; and - Training (S5) the target machine learning model based on the replaced and provided ground truth data samples.

Description

[0001] The invention relates to a method and a device for improving synthetic ground truth data by a data generator and for training a machine target learning model. State of the art

[0002] In the world of artificial intelligence (AI), the quality and diversity of training data is of central importance. To maximize the performance of AI networks, it is crucial that they are trained with a broad range of data that covers both real and unforeseen situations. Due to cost reasons and the need to include rare events (corner cases) in training datasets, researchers and developers are increasingly turning to synthetic data. Traditionally, such data is created by modeling scenes as 3D models, then randomizing these models and generating images using ray tracing or rasterizing. Although this process can be effective, it has its limitations. The quality of the generated data can be limited. Furthermore, the process may not be as accurate as the original data.cannot be scaled well, which seems particularly problematic given the growing demands of AI development.

[0003] In response to these challenges, a paradigm shift in training data generation is emerging. Traditional methods are increasingly being replaced by AI-assisted approaches, in which AI models themselves generate ground truth data from existing ground truth target data. This new approach offers the possibility of generating data with unprecedented diversity and at scale. However, it also entails uncertainties regarding data quality, as AI-generated data cannot offer the same quality assurances as conventionally generated data. Added to this is the problem that manual quality control is not a practical solution due to the volume of data generated.

[0004] It is an object of the invention to provide an improved method and / or an improved device.

[0005] The object is achieved by a method according to the features of patent claim 1. The object is achieved by a device according to the features of patent claim 10. Disclosure of the invention

[0006] According to a first aspect, a method is proposed for improving synthetic ground truth data by a data generator and for training a target machine learning model, the method comprising the steps: - Providing ground truth data samples synthetically generated by the data generator to ground truth source data; - Comparing the performance of the data generator for the provided ground truth data samples with a performance threshold; - Regenerating ground truth data samples for the same ground truth source data by the data generator if the performance threshold for the provided ground truth data samples is exceeded; - Replacing the ground truth data samples for which the performance threshold is exceeded with the newly generated ground truth data samples; and - Training the target machine learning model based on the replaced and provided ground truth data samples.

[0007] It is understood that the steps according to the invention, as well as other optional steps, do not necessarily have to be performed in the order shown, but can also be performed in a different order. Furthermore, additional intermediate steps can be provided. The individual steps can also comprise one or more substeps without thereby departing from the scope of the method according to the invention.

[0008] According to a second aspect, a device is proposed for improving synthetic ground truth data by a data generator and for training a machine target learning model, wherein the device comprises an evaluation and computing device which is designed to carry out the following steps: - Providing ground truth data samples synthetically generated by the data generator to ground truth source data; - Comparing the performance of the data generator for the provided ground truth data samples with a performance threshold; - Regenerating ground truth data samples for the same ground truth source data by the data generator if the performance threshold for the provided ground truth data samples is exceeded; - Replacing the ground truth data samples for which the performance threshold is exceeded with the newly generated ground truth data samples; and - Train the target machine learning model based on the replaced and provided ground truth data samples.

[0009] When "providing synthetic ground truth data," the data generator generates synthetic ground truth data samples. This data serves as the basis or "ground truth" for training the target machine learning model. It is, for example, an initial attempt to replicate reality or the characteristics of real data sets. The performance or quality of this generated data is then evaluated by comparing it to a predefined performance standard or performance threshold. This verification is intended to ensure that the generated data has a certain quality or accuracy. If the generated data does not meet the specified performance threshold, i.e., the quality is insufficient, it is regenerated. This means that the data generator attempts again to create improved versions of the ground truth data samples that correspond to the same ground truth source data.The newly generated data then preferably replaces the original data that did not meet the quality standard. This step ensures that only data of acceptable quality is used for further training. Furthermore, the data generator can optionally be trained with the improved ground truth data samples, i.e., the replaced and originally provided ones. This iterative process of evaluation, improvement, and, if necessary, training promotes the development of a more effective and precise data generator. The statements made for the method apply accordingly to the device. It is understood that linguistic variations of procedurally formulated features can be reformulated for the device according to common linguistic practice, without such formulations having to be explicitly listed here.

[0010] The target machine learning model can, in principle, be designed to solve a wide variety of tasks, for which it can be trained using the generated data. The design of the target machine learning model can vary depending on the specific inference case.

[0011] The method provides an approach for ensuring the quality of AI-generated training data without compromising the efficiency and / or scalability of the generation process. This overcomes the limitations of conventional data production and ensures the integrity of the training data. The present method can distinguish good from poor synthetic ground truth data without human input or any other discriminator. This allows a machine learning model, such as a neural network, to be trained robustly. The performance of the machine learning model is improved.

[0012] The term "ground truth sample data" describes an input for the machine learning model to be trained. Input can be, for example, image and / or video data, text data, audio data, or similar.

[0013] The term “Ground Truth Target Data” describes an expected output of the machine learning model to be trained, for example a visible object and / or a visible subject, such as a human skeleton, in an image or video file, a word or a sequence of words in a text file, a sound or a sequence of sounds in an audio file, etc.

[0014] The term "synthetic ground truth source data" describes the input of a generator that generates consistent synthetic ground truth data samples. The synthetic ground truth source data includes at least the ground truth target data, although additional inputs may be preferred. For example, the generator can process a depth map of an image scene as additional input in addition to the object and / or subject. This distinction is preferred because the generator makes it possible to generate any number of synthetic ground truth data samples that all share the same synthetic ground truth source data and thus also the associated ground truth target data.

[0015] This method is based on the realization that the machine learning model being trained itself represents a type of discriminator. Using an object detector as an example, poor detection of a sample is due either to poor performance of the machine learning model underlying the object detector or to a faulty synthetic ground truth sample. This method makes it possible to react to poor detection or to an output that deviates from the ground truth target by replacing the synthesized ground truth sample in the training dataset with several new synthesized samples with the same precondition or ground truth source. This makes it irrelevant what exactly led to the poor performance. If the synthesized sample was poor, it can be replaced using this method.If, on the other hand, the machine learning model's performance was poor, especially in the specific situation, replacing it with several new samples will enrich the training dataset with precisely the type of data that the machine learning model is still struggling with. Thus, pinpointing the exact cause of the underlying problem becomes irrelevant.

[0016] In a further aspect, providing ground truth data synthetically generated by the data generator comprises samples of ground truth source data: - Providing ground truth source data, in particular comprising depth maps and skeleton pose images; - Generating ground truth data samples from the provided ground truth source data by the data generator, in particular a stable diffusion control net, to synthesize consistent data; - preferably pre-training the target machine learning model with the generated ground truth data samples.

[0017] The method preferably includes providing ground truth source data, focusing in particular on depth maps and skeletal pose images. Depth maps provide spatial information about the distances of objects in the scene to the camera, while skeletal pose images visualize the arrangement and orientation of human or animal figures in the scene. These specific types of data are particularly valuable for applications requiring spatial understanding and pose analysis. Based on the provided source data, the data generator generates synthetic ground truth data samples. This process is preferably performed by a specialized type of data generator, namely a Stable Diffusion ControlNet.This type of network is designed to synthesize consistent, high-quality data by leveraging advanced artificial intelligence and machine learning techniques. The focus on consistency means that the generated data should match the source data in terms of visual and spatial properties. Before the data generator begins the iterative process of evaluation and improvement, it can preferably be pre-trained using the previously generated ground truth data samples. However, this is purely optional. This step preferably serves to establish the baseline performance of the data generator by training it on an initial dataset.Pretraining helps to improve the efficiency of the subsequent iterative training process by ensuring that the data generator has already gained a certain understanding and adaptation to the characteristics of the ground truth data.

[0018] The data generator is preferably designed to generate synthetic ground truth data samples based on ground truth source data, possibly defective. In image processing, the data generator can be a "Stable Diffusion ControlNet." For example, the data generator can synthesize consistent images that form the ground truth sample from depth maps and skeleton pose images that serve as the ground truth source. During training of the data generator, ground truth data samples that exhibit poor performance, i.e., for which the data generator has a performance that lies below the performance threshold, are replaced with newly generated ground truth data samples, preferably after pretraining as described.

[0019] In a further aspect, the method further comprises comparing a performance of the ground truth data samples with a median of the performance of all ground truth data samples for the same ground truth source, in particular to identify defective samples.

[0020] Performance comparison with the median is used to test the performance of individual ground truth data samples not only against a fixed performance threshold, but also against the median of the performance of all ground truth data samples for the same ground truth source. The median serves as a representative value of the central tendency of all generated data, which is robust to outliers and provides a balanced measure of overall data quality. The reason for this comparative approach is to identify defective or substandard ground truth data samples. Defective samples are preferably those whose performance characteristics deviate significantly from the median, indicating problems with data quality or consistency. These could preferably be caused by errors in the generation process, inadequacies in the source data, or other disturbances.By identifying defective or problematic data, the method enables targeted improvements in data quality and consistency. Comparing the data quality with the median performance of all samples introduces a more nuanced and dynamic criterion for assessing data quality than a fixed performance threshold alone could provide. This approach helps further increase the effectiveness and accuracy of the data generator by specifically addressing the elements most needed for improvement.

[0021] In a further aspect, ground truth data samples are regenerated based on the deviation of their performance from the median of all ground truth data samples of the respective ground truth source.

[0022] The resampling strategy can be further improved in various ways. For example, if a sufficient number of ground truth data samples with the same ground truth source are already present in the provided data set of (initial) ground truth data samples, it can be determined whether a ground truth sample is defective by comparing the performance of the data generator. In this case, the performance of defective ground truth data samples deviates significantly from the median of all ground truth data samples, whereas for the ground truth source data, a generally poor performance of the data generator indicates that all ground truth data samples are equally poor.

[0023] The method involves analyzing how much the performance of an individual ground truth data sample deviates from the median performance of all samples for the same ground truth source. This evaluation allows for a precise assessment of the quality of each sample and the specific identification of those that perform below average with respect to the specified performance standards. The decision to regenerate ground truth data samples is preferably based directly on the observed deviation of their performance from the median. Samples that exhibit a significant negative deviation are preferentially selected as candidates for regeneration. This criterion preferably ensures that the focus is on improving data quality by specifically addressing samples that can be improved the most relative to their peers.By preferentially regenerating ground truth data samples based on their relative performance deviation, specific problems or deficiencies in the data can be effectively addressed. This methodological approach aims to increase the overall quality of the dataset by ensuring that all samples exhibit a consistent level of performance that is closely aligned with the median of the entire sample group. Because the median performance is preferably used as a comparison standard, the method dynamically adapts to the changing quality of the dataset. This means that as the average data quality improves, the requirements for each individual sample also increase, leading to a continuous improvement in data quality over time.

[0024] In a further aspect, a number of newly generated ground truth data samples is chosen proportional to the median performance of all synthetic ground truth data samples of the respective ground truth source, in particular to prevent overfitting and / or promote consistent network performance across all ground truth target data.

[0025] Instead of replacing a ground truth sample with a fixed number of new samples, the number of ground truth data samples can also be chosen proportionally to the median performance of all synthetic ground truth data samples of the respective ground truth source data. This approach is particularly effective in preventing overfitting and helps achieve similar data generator performance for all ground truth target data.

[0026] The number of newly generated ground truth data samples is chosen to be proportional to the median performance of all synthetic ground truth data samples. This means, in particular, that the decision on how much data to regenerate is based on the average quality of the current dataset. A higher quality median can lead to fewer regenerations, while a lower median suggests a larger number. By proportionally adjusting the number of newly generated samples to the median data quality, a mechanism is implemented to counteract overfitting. Overfitting occurs when a model is too specifically tailored to the training data and thus loses its ability to generalize to new, unknown data.Adjusting the number of newly generated samples to ensure consistent quality across different ground truth target data sets promotes stable and predictable network performance. Balancing data quality and quantity ensures that the network is not affected by fluctuations in data quality, resulting in more consistent and reliable performance. Similar to previous aspects, this approach allows for dynamic adaptation of the training strategy based on changing data quality.

[0027] In another aspect, replacing poorly performing ground truth data samples is performed by automatically identifying poor samples and regenerating ground truth data samples without manual intervention.

[0028] The process preferably begins with the automatic detection of ground truth data samples whose performance lies below a specified threshold or compared to a performance median. This step uses algorithms and evaluation criteria to efficiently identify data that does not meet quality requirements. Automating this step increases the objectivity and speed with which poor samples can be identified. After identification, the ground truth data samples rated as inadequate are preferably regenerated automatically, without the need for manual intervention. This process involves reusing the data generator to create improved versions of the data in question. The automatically regenerated ground truth data samples preferably replace those with poor performance in the database, without the need for manual intervention.

[0029] In a further aspect, the method further comprises: limiting a number of ground truth data samples per ground truth source in order to prevent poor performance of all samples of a ground truth source due to systematic reasons and thus in particular to avoid bloating of the training data.

[0030] In this preferred additional step, the number of ground truth data samples for the same ground truth source is limited to prevent systematically poor performance of the data generator on all ground truth data samples from a ground truth source from bloating the dataset. In this case, the problem lies neither in the data synthesis nor in the convergence of the training of the target machine learning model. Rather, it is necessary to investigate whether the ground truth source is faulty and / or whether the network structure of the target machine learning model has any weaknesses.

[0031] A fixed upper limit on the number of ground truth data samples generated from a single ground truth source is preferably implemented. This limit should preferably prevent the database from being dominated by an excessive number of samples from individual sources, which could impair the diversity and generalizability of the training dataset. By limiting the number of samples per source, the risk is mitigated that systematic errors or limitations of a specific source lead to consistently poor performance of all associated samples. Such systematic problems could result, for example, from low-quality source data or from inherent limitations of the data generation method. Furthermore, bloat of the training dataset is avoided.Too much data, especially if it is of limited diversity, can slow down training and reduce the efficiency of the learning process without necessarily improving model performance.

[0032] In a further aspect, a computer program with program code is claimed for carrying out at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, a computer program (product) comprising instructions that, when executed by a computer, cause the computer to carry out the method(s) of the method in one of its aspects.

[0033] In a further aspect, a computer-readable data carrier with program code of a computer program is proposed for executing at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium comprising instructions that, when executed by a computer, cause the computer to execute the method(s) of the method in one of its aspects.

[0034] The described designs and further training courses can be combined as desired.

[0035] Further possible embodiments, further developments and implementations of the invention also include combinations of features of the invention described previously or below with regard to the embodiments that are not explicitly mentioned. Short description of the drawings

[0036] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain principles and concepts of the invention.

[0037] Other embodiments and many of the aforementioned advantages will become apparent upon review of the drawings. The elements illustrated in the drawings are not necessarily drawn to scale.

[0038] They show: Fig. 1 is a schematic flow diagram of an aspect of the present method; and Fig. 2 is a schematic block diagram of an aspect of the present method.

[0039] In the figures of the drawings, the same reference symbols designate the same or functionally equivalent elements, parts or components, unless otherwise stated.

[0040] Fig. Figure 1 shows a schematic flow diagram of a method for enhancing synthetic ground truth data by a data generator and for training a target machine learning model.

[0041] In any embodiment, the method can be carried out at least partially by a device 100, which for this purpose can comprise several components not shown in detail, for example, one or more provision devices and / or at least one evaluation and computing device. It is understood that the provision device can be designed jointly with the evaluation and computing device or can be different from it. Furthermore, the device 100, which can be part of a system, can comprise a storage device and / or an output device and / or a display device and / or an input device.

[0042] The computer-implemented method comprises at least the following steps: In a step S1, ground truth data samples synthetically generated by the data generator are provided to ground truth source data. In a step S2, a performance of the data generator for the provided ground truth data samples is compared with a performance threshold In a step S3, the data generator regenerates ground truth data samples for the same ground truth source data if the performance threshold for the provided ground truth data samples is exceeded. In step S4, the ground truth data samples for which the performance threshold is exceeded are replaced by the newly generated ground truth data samples. In step S5, the target machine learning model is trained based on the replaced and provided ground truth data samples.

[0043] Optionally, the method may include step S6, namely limiting the number of ground truth data samples per ground truth source, in order to prevent poor performance of all samples of a ground truth source due to systematic reasons and thus, in particular, to avoid bloating the training data. This step is preferably performed after step S2 and before step S3.

[0044] Fig.Figure 2 shows a schematic block diagram of one aspect of the present method. It illustrates the provision S1 of ground truth data samples synthetically generated by the data generator to ground truth source data. The provided ground truth data samples form a pool of ground truth data samples. In an optional step S1A, ground truth data samples are preferably selected from the pool of ground truth data samples. The subsequent steps are performed using the selected or thus provided ground truth data samples.

[0045] The data generator's performance for the ground truth data samples provided or selected in step S1A is compared (S2) with a performance threshold. A check (S2A) is performed to determine whether the data generator's performance for the respective ground truth data samples is below the performance threshold.

[0046] Furthermore, after step S2 or S2A, the number of ground truth data samples per ground truth source is limited (S6) to prevent poor performance of all samples of a ground truth source due to systematic reasons and thus, in particular, to avoid bloating the training data. The limitation is performed by comparing whether a number of ground truth data samples that have the same ground truth source exceeds a predetermined or selectable limit. It should be noted that if the limit is exceeded in the optional step S6, the ground truth source can be marked for manual review in a step S7.

[0047] The data generator then regenerates or synthesizes S3 ground truth data samples for the same ground truth source data if the performance threshold for the provided ground truth data samples is exceeded. The same ground truth source data is used as a basis for this.

[0048] Furthermore, the ground truth data samples for which the performance threshold is exceeded are replaced S4 by the newly generated ground truth data samples.

[0049] The ground truth data samples are replaced in the initial data pool of ground truth data samples. This revised data pool can then be used to train the target machine learning model.

Claims

[1] Method for improving synthetic ground truth data by means of a data generator and for training a machine learning target model, the method comprising the steps: - Providing (S1) samples of Ground Truth data synthetically generated by the data generator to Ground Truth Source data; - Compare (S2) the performance of the data generator for the provided ground truth data samples with a performance threshold; - Regenerating (S3) Ground Truth data samples for the same Ground Truth Source data by the data generator if the performance threshold for the provided Ground Truth data samples is undershot; - Replace (S4) the ground truth data samples for which the performance threshold is undershot with the newly generated ground truth data samples; and - Training (S5) the machine learning target model based on the replaced and provided ground truth data samples. [2] The method of claim 1, wherein the provision (S1) of Ground Truth data samples synthetically generated by the data generator comprises Ground Truth Source data: - Providing Ground Truth Source data; - Generating Ground Truth data samples from the provided Ground Truth source data using the data generator to synthesize consistent data; - preferably pretraining the machine learning target model with the generated ground truth data samples. [3] Method according to claim 1 or 2, further comprising comparing a performance of the Ground Truth data samples with a median of the performance of all Ground Truth data samples for the same Ground Truth Source. [4] Method according to claim 3, wherein the regeneration of Ground Truth data samples is based on the deviation of their performance from the median of all Ground Truth data samples of the respective Ground Truth Source. [5] Method according to claim 3 or 4, wherein a number of the newly generated Ground Truth data samples is selected proportionally to the median of the performance of all synthetic Ground Truth data samples of the respective Ground Truth Source. [6] Method according to any of the preceding claims, wherein the replacement of poor-performing ground truth data samples is carried out by automated identification of poor samples and regeneration of ground truth data samples without manual intervention. [7] Method according to any of the preceding claims, wherein the method further comprises: limiting (S6) a number of Ground Truth data samples per Ground Truth Source to prevent poor performance of all samples of a Ground Truth Source due to systematic reasons. [8] Computer program with program code to execute at least parts of a method according to any one of claims 1 to 7 when the computer program is executed on a computer. [9] Computer-readable data carrier containing program code of a computer program for executing at least parts of a method according to any one of claims 1 to 7 when the computer program is executed on a computer. [10] Device (100) for improving synthetic ground truth data by means of a data generator and for training a machine learning target model, wherein the device (100) has an evaluation and computing unit configured to perform the following steps: - Providing samples of Ground Truth data synthetically generated by the data generator to Ground Truth Source data; - Comparing the performance of the data generator for the provided ground truth data samples against a performance threshold; - Regenerating Ground Truth data samples for the same Ground Truth source data by the data generator if the performance threshold for the provided Ground Truth data samples is not met; - Replacing the ground truth data samples for which the performance threshold is undershot with the newly generated ground truth data samples; and - Training the machine learning model based on the replaced and provided ground truth data samples.

Images