Method for evaluating a degree of realism of synthetic sensor data

Abstract

The invention relates to a method (100) for evaluating a degree of realism of synthetic sensor data (2), comprising the following steps: - providing (101) captured sensor data (1), wherein the captured sensor data (1) result from a capture by at least one sensor (5), - providing (102) the synthetic sensor data (2), wherein the synthetic sensor data (2) result from a synthesis of the captured sensor data (1), - applying (103) a method for attribute-based explanation of machine model predictions on the basis of the sensor data (1) in order to ascertain at least one assignment result (4) for the captured sensor data (1), - applying (104) the method for attribute-based explanation of machine model predictions on the basis of the synthetic sensor data (2) in order to ascertain at least one assignment result (4) for the synthetic sensor data (2), - comparing (105) the at least one ascertained assignment result (4) for the synthetic sensor data (2) with the at least one ascertained assignment result (4) for the captured sensor data (1), in particular each ascertained assignment result (4) for the synthetic sensor data (2) with all ascertained assignment results (4) for the captured sensor data (1), in order to evaluate the degree of realism of the synthetic sensor data (2). The invention also relates to a computer program, to a device and to a storage medium for this purpose.

Description

[0001] Description

[0002] title

[0003] Method for evaluating the degree of realism of synthetic sensor data

[0004] The invention relates to a method for evaluating the degree of realism of synthetic sensor data. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose.

[0005] State of the art

[0006] In automated optical inspection use cases, problems can arise in capturing defect images (NOK) due to high production quality. This can complicate the training of a reliable, robust, and high-quality machine learning model, or reduce its subsequent detection performance. However, even at the start of an automated optical inspection (AOI) use case, the unavailability of NOK images in sufficient quantity and variation can be a barrier. Collecting this representative dataset, including sufficiently underrepresented images such as the NOK class, can lead to significant delays. With generative artificial intelligence, it is possible to create synthetic images that help develop robust, high-quality inspection methods and reduce waiting times.

[0007] The state of the art provides a variety of quantitative metrics for evaluating generated synthetic images. These metrics measure the quality of the images, for example, in terms of domain fidelity, i.e., the ability of the generating model to remain faithful to the domain for which it was trained. For example, a model for vehicles should generate images of vehicles and not of birds. Furthermore, diversity can be measured, i.e., the extent to which the domain space is covered. At the extreme end of low diversity, this means that only the training images have been stored.

[0008] Examples of these quantitative metrics are the inception score, the Frechet inception distance, the likelihood score or the maximum mean discrepancy.

[0009] Explainable Artificial Intelligence (XAI) plays an increasingly important role in building trust in and visualizing black-box deep learning models. In deep learning-based automated optical inspection (AOI), XAI methods can be used for visualization for domain users. Post-hoc XAI methods are particularly important for industrial applications because they can be applied to a previously trained model. In particular, post-hoc pixel attribution methods generate predicted attribution values ​​that can be visualized as a heatmap. Heatmaps illustrate the significance of each input pixel with respect to the predicted output class.

[0010] Disclosure of the invention

[0011] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9 and a computer-readable storage medium having the features of claim 10. Further features and details of the invention emerge from the respective subclaims, the description and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that a mutual reference is always possible with regard to the disclosure of the invention.The invention particularly relates to a method for evaluating the degree of realism of synthetic sensor data, comprising the following steps, wherein the steps can be performed repeatedly and / or sequentially. The degree of realism corresponds, in particular, to a similarity between the synthetic sensor data and the acquired sensor data. This can be represented by domain fidelity, e.g., the extent to which the synthetic sensor data represent the same objects or the same features.

[0012] In a first step, preferably acquired sensor data is provided, wherein the acquired sensor data results from the detection of at least one sensor. Providing this data may, for example, comprise data transmission or retrieval from a data storage device. The acquired sensor data may include images, such as camera images. Furthermore, acquired sensor data from a radar, LiDAR, or ultrasonic sensor is also conceivable, so that the images may accordingly be radar, LiDAR, or ultrasonic images.

[0013] In a further step, the synthetic sensor data is preferably provided. The synthetic sensor data can be a simulated imitation of the acquired sensor data and, in particular, result from a simulation or synthesis of the acquired sensor data. The fact that the synthetic sensor data is a simulated imitation of the acquired sensor data describes, in particular, that it was generated based on the acquired sensor data and is similar to it. For example, a generative machine learning model can be trained based on the acquired sensor data to generate the synthetic sensor data. The provision of the synthetic sensor data can also include, for example, data transmission or retrieval from a data storage device.

[0014] In a further step, a method for attribute-based explanation of machine model predictions based on the acquired sensor data is preferably applied to determine at least one assignment result of the acquired sensor data, preferably for a respective element of the acquired sensor data. An element of the acquired sensor data can be an image, for example. The method for attribute-based explanation of machine model predictions is, in particular, an "Explainable Artificial Intelligence" (XAI) method. This refers, in particular, to techniques and approaches within artificial intelligence that aim to make the decision-making processes of machine learning models transparent and comprehensible. XAI methods can ensure that the functioning of a machine learning model is understandable.This can be achieved by using simpler or interpretable models or by explaining more complex models. Some examples of XAI methods are described below. According to a first example, it is possible to determine which input features contribute most to the machine learning model's prediction. This can be visualized, for example, as a heatmap representing the matching result for a particular piece of synthetic sensor data. Furthermore, there are Local Interpretable Modelagnostic Explanations (LIME), which provide local explanations for the predictions of any classification or regression model. Finally, there are SHapley Additive exPlanations (SHAP), which explain the effect of each feature on the prediction of a machine learning model using game-theoretic Shapley values.

[0015] In a further step, the method for attribute-based explanation of machine model predictions based on the synthetic sensor data is preferably applied to determine at least one assignment result of the synthetic sensor data, preferably for a respective element of the synthetic sensor data. An element of the synthetic sensor data can be, for example, an image.

[0016] In a further step, the at least one determined assignment result of the synthetic sensor data is preferably compared with the at least one determined assignment result of the acquired sensor data in order to evaluate the degree of realism of the synthetic sensor data. The comparison can be carried out mathematically, for example, based on data points in the assignment results. In particular, it can be provided that each determined assignment result of the synthetic sensor data is compared with all determined assignment results of the acquired sensor data. In particular, a comparison is made of the determined assignment result of a respective element of the synthetic sensor data with all respective assignment results of the elements of the acquired sensor data. This allows a more precise assessment of the degree of realism.

[0017] Optionally, it is conceivable that applying the method for attribute-based explanation of machine model predictions includes the following step:

[0018] Providing a visual and / or mathematical description of a result of a classification in the respective element of the acquired sensor data and the synthetic sensor data as the assignment result, preferably in the form of a heatmap, to explain the classification.

[0019] This step is preferably carried out on the basis of both the acquired sensor data and the synthetic sensor data. The description preferably represents a relationship between the input data, i.e. the respective element of the acquired sensor data and the synthetic sensor data, and the resulting classification. For example, a heat map is used to show the extent to which features and areas have contributed to the resulting classification. Alternatively, a mathematical or textual description of this relationship is also conceivable. In the context of an automatic optical inspection, for example, the heat map may show in an element of the acquired sensor data that a defect was detected in a component during the classification.

[0020] It may be further advantageous for the comparison to further include the following step:

[0021] Performing a similarity analysis between the at least one determined assignment result of the synthetic sensor data with the at least one determined assignment result of the acquired sensor data.

[0022] The similarity analysis can, for example, be a mathematical method in which data points from the assignment results are compared with each other, e.g., by calculating the distance between the data points. It is also conceivable to perform the similarity analysis visually, for example, by a specialist, in which case the specialist can be advantageously supported by the visualization in evaluating the synthetic sensor data.

[0023] Optionally, the provision of synthetic sensor data may include the following step:

[0024] Generating synthetic sensor data using a generative machine learning model.

[0025] Generative machine learning models are specifically designed to generate new data, i.e., synthetic sensor data, that is similar to training data, i.e., the acquired sensor data. Some examples of such models are described below. A first example is Generative Adversarial Networks (GANs). GANs typically comprise two networks, a generator and a discriminator. The generator preferentially generates data, while the discriminator attempts to distinguish real data from the generated data. There are also Conditional Generative Adversarial Networks (cGANs): This variant of GANs specifically enables the generation of data that depends on certain conditions, such as a specific type of sensor data. This can be useful for generating synthetic data under specific conditions that may be underrepresented in real datasets. Another example is Variational Autoencoders (VAEs).VAEs are a type of autoencoder that enables probabilistic treatment of data input. They are suitable, for example, for generating new data that matches the properties of the training dataset. VAEs can be used to generate synthetic sensor data, especially when modeling the uncertainties in the data is important. Another example is Long Short-Term Memory Networks (LSTMs). Although LSTMs are often used for sequential data such as text or time series, they can also be used to generate synthetic sensor data, especially when this data exhibits temporal dependencies. Diffusion networks can also be used to generate synthetic sensor data. Diffusion networks are based on the idea that information diffuses through a network by moving from one node to another.Each node in the network preferentially receives information from its neighbors and passes this information on to its own neighbors. In this way, complex information patterns can be advantageously constructed.

[0026] In a further possibility, the method may further comprise the following step:

[0027] Adjusting the generative machine learning model based on a comparison result.

[0028] The adaptation can include changing parameters and / or weights of the generative machine learning model. For a regeneration of the synthetic sensor data, at least one condition resulting from the comparison result can also be taken into account. It is conceivable, for example, that a user provides this condition to the generative machine learning model via an input. By adapting the generative machine learning model, the synthetic sensor data can subsequently be generated with greater precision and improvement.

[0029] Furthermore, within the scope of the invention, it is conceivable that the adaptation of the generative machine learning model comprises the following steps:

[0030] Determining a sample of the at least one determined assignment result of the acquired sensor data, defining an additional condition for the generative machine learning model based on the determined sample by applying a guided generative approach, performing a fine-tuning of the generative model taking into account the defined additional condition.

[0031] The sample can comprise at least one, in particular exactly one, assignment result of the acquired sensor data. The guided generative approach refers in particular to a methodology in artificial intelligence and machine learning, for example in the field of generative machine learning models. These approaches aim in particular to control or direct the generation of data or content, such as text, images, or music, to meet specific properties or criteria. ControlNet is an example of such a system capable of directing the generation of content to achieve certain desired results. One element of the guided generative approach, such as ControlNet, is in particular the ability to control the generation. This can mean that users can specify specific parameters or guidelines that influence the type of content generated.For example, when generating a text, a user could specify that it should have a formal style or address specific topics. This can advantageously provide greater flexibility and control compared to traditional generative machine learning models. As part of the fine-tuning process, parameters and / or weights of the generative machine learning model can be adjusted.

[0032] According to a further possibility, it can be provided that the acquired sensor data comprise images of components that are processed in a production plant and that the synthetic sensor data is used to train a machine learning model for detection and / or classification as part of an automatic optical inspection in the production plant. The method according to the invention can be particularly useful for an automatic optical inspection of a production plant, since few images of defective components may be available due to high production quality. The images of the components are therefore in particular images of components with at least one production defect. These images with the at least one production defect can advantageously be supplemented with further images in the form of the synthetic sensor data using the method according to the invention.

[0033] The machine learning model, in particular the predictive rather than the generative machine learning model, is therefore preferably trained for classification and / or detection. Accordingly, the training can result in a trained machine learning model that can be used for classification and / or detection. The use and thus the inference can be provided, for example, in the production plant. The data points of the input data can, for example, be pixels of image data or be based on them in order to carry out the classification and / or detection of the data points based on the pixels. The input data can comprise sensor and / or image data that result at least partially from acquisition with a sensor, preferably a camera sensor, and / or that have been at least partially synthesized, i.e., in particular, simulate the real data of a sensor.Specifically, it can be provided that a respective component of the production plant is represented by the values ​​of image points, preferably pixels, of the image data. A classification, preferably image classification and / or detection, can be provided on the basis of these values. This makes it possible, for example, to detect defects in the components. The classification can also be provided in the form of semantic segmentation (i.e., a pixel- or area-wise classification) and / or detection. The image data can, for example, be images from a camera and / or a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera. Accordingly, the images can also be embodied as radar images and / or ultrasonic images and / or thermal images and / or LiDAR images.

[0034] The invention also relates to a computer program, in particular a computer program product, comprising instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with reference to a method according to the invention.

[0035] The invention also relates to a data processing device configured to carry out the method according to the invention. The device can be, for example, a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0036] The invention may also include a computer-readable

[0037] The storage medium can be a storage medium that contains the computer program according to the invention and / or includes instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0038] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0039] Further advantages, features, and details of the invention will become apparent from the following description, which describes embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show:

[0040] Fig. 1 shows a schematic visualization of a method, a device, a storage medium and a computer program according to embodiments of the invention,

[0041] Fig. 2 is a schematic representation of a method according to embodiments of the invention.

[0042] In Fig. 1, a method 100, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the invention are shown schematically.

[0043] Fig. 1 shows in particular a method 100 for evaluating a degree of realism of synthetic sensor data 1. In a first step 101, acquired sensor data 1 is provided, wherein the acquired sensor data 1 results from a detection of at least one sensor 5. In a second step 102, the synthetic sensor data 2 is provided, wherein the synthetic sensor data 2 is a simulated imitation of the acquired sensor data 1 and, for this purpose, results from a synthesis of the acquired sensor data 1. In a third step 103, a method for the attribute-based explanation of machine model predictions based on the acquired sensor data 1 is applied in order to determine at least one assignment result 4 for a respective element of the acquired sensor data 1.In a fourth step 104, the method for attribute-based explanation of machine model predictions based on the synthetic sensor data 1 is applied to determine at least one assignment result 4 for a respective element of the synthetic sensor data 2. In a fifth step 105, the at least one determined assignment result 4 of the synthetic sensor data 2 is compared with the at least one determined assignment result 4 of the acquired sensor data 1 in order to evaluate the degree of realism of the synthetic sensor data 2. In this exemplary embodiment, each determined assignment result 4 of the synthetic sensor data 2 is compared with all determined assignment results 4 of the acquired sensor data 1.

[0044] In automatic optical inspection (AOI), the evaluation of synthetic sensor data 2 should preferably not necessarily be limited to the quality of the synthetic sensor data 2. The synthetic sensor data 2 should also be validated in the context of an entire detection application.

[0045] With the present invention, a hurdle for launching a machine learning model-based application can be lowered, particularly in automatic optical inspection. Therefore, it is preferably evaluated to what extent the amount of sensor data used for model training, which are particularly real images, can be reduced while simultaneously achieving a similar performance to that achieved when training with the original dataset.

[0046] The present invention describes, according to exemplary embodiments, an approach for a qualitative evaluation of generated synthetic sensor data 2. In particular, it is not necessary to run through an entire loop of evaluating the detection performance, i.e., for example, no respective training and subsequent evaluation of a respective machine learning model is necessary. Furthermore, an interpretable evaluation of how realistic the generated synthetic sensor data 2 is can advantageously be provided. The method according to exemplary embodiments can advantageously also provide the possibility of evaluating synthetic sensor data 2 according to its distance from the real data set. This can be important, for example, for the further training of machine learning models or for monitoring the trained machine learning models when clear outliers are to be selected for detection.

[0047] Furthermore, the interpretable approach according to embodiments with a human in the loop can provide insights into potential caveats during the generation or training process of the generative machine learning model. These insights can be used as input for further improving the training process of the generative machine learning model that generates the synthetic sensor data 2.

[0048] In the following, an embodiment of the method is described with reference to Fig. 2. An input for the method according to embodiments is, in particular, a model-specific XAI method 3 created based on a classification model. A further input is, as shown in the upper area of ​​Fig. 2, preferably a set of acquired sensor data 1, in particular real images x 1; ... , x N. Another input, which is shown in the lower part of Fig. 2, is in particular a set of synthetic sensor data 2 s 1; ..., s L generated by a generative machine learning model G.

[0049] In particular, assuming a well-established and well-tested model-specific XAI method 3 derived from a well-established and well-tested classification model, a number of matching results 4 a 1; ..., a N from the recorded sensor data 1 x 1; ..., x NThese assignment results 4 can be visualized as a heatmap, as shown in Fig. 2 for the acquired sensor data 1 and the synthetic sensor data 2, respectively. This allows multiple assignment results 4 to be provided, ie, in particular, a respective assignment result 4 per element of the acquired sensor data 1 and the synthetic sensor data 2. The assignment results can provide a distribution of the assignment results of the real data.

[0050] For each element of the synthetic sensor data 2 s 1; ..., s L In particular, the same model-specific XAI method 3 is applied to obtain an assignment result 4 a, per element, as shown in Fig. 2, so that a set of assignment results 4 as 1; ..., as L For simplicity, only one element of the synthetic sensor data 2 is shown in Fig. 2.

[0051] Furthermore, it is preferably determined whether the individual assignment results 4 as 1; as L of the synthetic sensor data 2 are real. This can be done by checking whether the assignment results 4 as 1; ..., as L be drawn from the same distribution as the allocation results 4 a 1; ..., a N of the real images. One approach would be, for example, a similarity analysis or search, for example, if the assignment results 4 can be visualized as heatmaps (as in the embodiment in Fig. 2), but a statistical test would also be conceivable, for example, nonparametric or parametric.

[0052] The elements of the synthetic sensor data 2 which are found not to correspond to the same distribution as the acquired sensor data 1 are preferably marked accordingly, for example as “inconsistency”.

[0053] According to embodiments, various possibilities can be provided for a next step.

[0054] According to a first possibility without optimizing the generative approach, the labeled elements of the synthetic sensor data 2, i.e., those identified as having a discrepancy, can be filtered. This filtering can advantageously guarantee that only elements of the synthetic sensor data 2 from the distribution are used for further training of the machine learning model. According to a second possibility, manual instructions for optimizing the generative machine learning model can be provided. By visually inspecting the heatmap of those elements of the synthetic sensor data 2 that have been labeled, i.e., for which the above-mentioned approach has identified a discrepancy, a domain expert can derive improvements for the next training iteration of the generative machine learning model. According to the embodiment in Fig.2, this could be a text prompt such as "only one head" if two dog heads are mistakenly generated. A restriction on the image size can also be imposed, as larger image sizes used when fine-tuning a generative machine learning model, as opposed to the original size, can lead to duplication in the elements of the synthetic sensor data 2.

[0055] According to a third possibility, automatic guidance is provided for optimizing the generative machine learning model. Preferably, assuming that the generative model G is conceptually correct, the percentage of labeled elements of the synthetic sensor data 2 with inconsistencies cannot exceed a certain threshold. The threshold is preferably set specifically for each use case, since the quality of the elements of the synthetic sensor data 2 can vary greatly depending on the use case. If this threshold is exceeded, it can be concluded that the generative model G is conceptually incorrect and is not learning the actual assignments correctly. The following steps can be provided here. First, a sample of an assignment result 4 a c from the distribution of the assignment results 4 a 1; ..., a N from the recorded sensor data 1 x1; ..., x N Subsequently, the original generative machine learning model G can be conditioned on a c by applying a guided generative approach, e.g., via ControlNet. In a further step, the generative

[0056] Machine learning model G with this additional condition.

[0057] The above explanation of the embodiments describes the present invention exclusively by way of examples.

[0058] Of course, individual features of the embodiments can be freely combined with one another, provided that this is technically reasonable, without departing from the scope of the present invention.

Claims

Claims 1. A method (100) for evaluating a degree of realism of synthetic sensor data (2), comprising the following steps: Providing (101) detected sensor data (1), wherein the detected sensor data (1) result from a detection of at least one sensor (5), Providing (102) the synthetic sensor data (2), wherein the synthetic sensor data (2) result from a synthesis of the acquired sensor data (1), Applying (103) a method for attribute-based explanation of machine model predictions on the basis of the acquired sensor data (1) in order to determine at least one assignment result (4) of the acquired sensor data (1), Applying (104) the method for attribute-based explanation of machine model predictions based on the synthetic sensor data (2) to determine at least one assignment result (4) of the synthetic sensor data (2), Comparing (105) the at least one determined assignment result (4) of the synthetic sensor data (2) with the at least one determined assignment result (4) of the acquired sensor data (1), in particular each determined assignment result (4) of the synthetic sensor data (2) with all determined assignment results (4) of the acquired sensor data (1), in order to evaluate the degree of realism of the synthetic sensor data (2).

2. Method (100) according to claim 1, characterized in that applying the method for attribute-based explanation of machine model predictions comprises the following step: Providing a visual and / or mathematical description of a classification result in a respective element of the recorded sensor data (1) and the synthetic sensor data (2) as the assignment result (4), preferably in the form of a heat map, to explain the classification.

3. Method (100) according to one of the preceding claims, characterized in that the comparing (105) further comprises the following step: performing a similarity analysis between the at least one determined assignment result (4) of the synthetic sensor data (2) with the at least one determined assignment result (4) of the acquired sensor data (1).

4. Method (100) according to one of the preceding claims, characterized in that the provision (102) of the synthetic sensor data (2) comprises the following step: Generating the synthetic sensor data (2) using a generative machine learning model.

5. The method (100) according to claim 4, characterized in that the method (100) further comprises the following step: adapting the generative machine learning model based on a result of the comparison.

6. The method (100) according to claim 5, characterized in that adapting the generative machine learning model comprises the following steps: Determining a sample of the at least one determined assignment result (4) of the acquired sensor data (1), defining an additional condition for the generative machine learning model based on the determined sample by applying a guided generative approach, Fine-tune the generative model taking into account the defined additional condition.

7. Method (100) according to one of the preceding claims, characterized in that the acquired sensor data (1) comprise images of components (7) which are processed in a production plant (6) and the synthetic sensor data (2) are used for training a machine learning model for detection and / or classification in the context of an automatic optical inspection in the production plant (6).

8. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to any one of the preceding claims.

9. Device (10) for data processing which is arranged to carry out the method (100) according to one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) according to any one of claims 1 to 7.

Images