Method for ensuring quality of synthetic data for at least one perceptual function using distributed system

Abstract

The invention relates to a method for ensuring the quality of synthetic data for at least one perceptual function using a distributed system, comprising the following steps: providing at least one interface of the distributed system, at least one interface is provided for providing access to an evaluation component of the distributed system by at least two instances of the distributed system via the at least one interface, a use case for synthesizing data being specified by one of the at least two instances of the distributed system, wherein the use case comprises at least one perceptual function, a description of the synthetic data and at least one metric for evaluating the quality of the synthetic data, the synthetic data being generated by said one or the other of the at least two instances of the distributed system on the basis of the described use case, the synthetic data is evaluated with respect to the reference data by an evaluation component of the distributed system using at least one metric in order to ensure the quality of the synthetic data. The invention further relates to a computer program, a device and a storage medium for this purpose.

Description

Technical Field

[0001] The invention relates to a method for ensuring the quality of synthetic data for at least one perceptual function using a distributed system. The invention further relates to a computer program, a device and a storage medium for this purpose. Background Art

[0002] In the prior art there are various examples of methods for comparing images in a reference manner and evaluating image data. In particular, these examples generally have the following focus: for example, quantifying the similarity between two images in a compressed and uncompressed manner with respect to human perception, that is, how noticeable compression artifacts are, for example. There are also examples of methods for evaluating synthetic data, which are, for example, rendered by simulation tools or generated by generative adversarial network (GAN) methods. Here, the focus is especially on the direct comparison of data (e.g. real and synthetic) with corresponding metrics (e.g. FID) in order to generate a metric number for comparing various methods.

[0003] In addition, there are methods that use computer vision methods (e.g. semantic segmentation) to compare real and synthetic images in order to assess the suitability of the data for a specific method. However, in order to apply these methods, for example, suitable computer vision algorithms must be selected, implemented, applied to the appropriate data corresponding to the specific application case and evaluated. This can take place solely on the user side of the data.

[0004] In order to develop and verify perceptual functions such as computer vision based on deep neural networks, annotated image data or videos are preferably required. For example, reconstruction of scenes, recognition of objects, recognition of events, recognition of activities, video tracking, object recognition, 3D position estimation, motion estimation and 3D scene modeling belong to sub-fields of computer vision. In addition, there are different types of tools for creating synthetic image data, images or videos. The problem here is especially to evaluate the quality and suitability of these data for training or testing perceptual functions. The problem of suitably and efficiently evaluating data and tools, for example, involves the provider of tools or data and the users of these tools or the consumers of data. The quality of synthetic data must preferably be good enough so that the data can be applied to training or testing perceptual functions designed for real data. So far, quality inspection is very expensive, which usually limits or hinders the availability of synthetic data or generation tools.

[0005] The general applicability of synthetic data or synthetic tools for training or testing perceptual functions may be questionable, which may create a cost / benefit risk for developers or testers of perceptual functions, so that methodologies that utilize synthetic data to train or test DNN-based perceptual functions are generally not used.

[0006] The evaluation of data or tools by users or data consumers is mostly done individually, qualitatively and manually. Therefore, a large effort may usually be required to conduct a quantitative evaluation in an experiment.

[0007] For example, separate solutions have been created for data, models, toolchains, and experiments to date. However, due to the complexity of these components, interfaces, and manual steps, iterations can be too slow, too inefficient, and too complex to enable an economical and scalable way of working, and thus to enable a plannable and continuous business relationship between data and tool developers and their users in perceptual development. Summary of the invention

[0008] The subject matter of the invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9 and a computer-readable storage medium having the features of claim 10. Further features and details of the invention are derived from the respective dependent claims, the description and the drawings. Features and details described in conjunction with the method according to the invention naturally also apply in conjunction with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, respectively, so that the disclosure of the individual inventive aspects is always referred to or can be referred to one another.

[0009] The subject matter of the invention is in particular a method for ensuring the quality of synthetic data for at least one perceptual function using a distributed system, comprising the following steps:

[0010] - providing at least one interface of the distributed system so as to provide at least two instances of the distributed system with access to an evaluation component of the distributed system via the at least one interface,

[0011] - specifying, by one of the at least two instances of the distributed system, a use case for the synthesized data, wherein the use case comprises at least at least one perceptual function, a description of the synthesized data and at least one metric for evaluating the quality of the synthesized data,

[0012] - generating said synthetic data based on the described use case by said one or the other of said at least two instances of said distributed system,

[0013] - evaluating the synthetic data relative to reference data by an evaluation component of the distributed system using at least one metric in order to ensure the quality of the synthetic data.

[0014] In the scope of the present invention, synthetic data is in particular the simulated or emulated data of a sensor or camera. An interface may be provided as a wired or wireless interface, for example for connection via the Internet. An instance may be a corresponding data processing device or a corresponding user or user role, wherein it is also conceivable that multiple users, that is, in particular multiple instances, use the same data processing device. For example, an instance with a data processing device may describe a use case, and another instance with the same or another data processing device may generate synthetic data. With a prescribed use case, for example, at least one perceptual function may be selected for detecting traffic signs, and accordingly, synthetic data and reference data may include different traffic signs in a traffic scene in this case. Reference data may be real data, which is derived, for example, from the acquisition of a sensor or camera. At least one metric may be based, for example, on a direct comparison between synthetic data and reference data, for example, by pixel-by-pixel comparison. Synthetic data may be generated by a tool, wherein the tool may be, for example, a rendering tool or an enhancement tool or a machine learning model or a neural network. The generation may be performed based on at least a portion of the reference data. In addition, it may be provided that other reference data is collected based on the evaluation of the synthetic data. Reference data may be provided and / or collected by one of at least two instances. If at least one metric does not exceed a defined limit value, the quality of the synthesized data can be ensured. The method particularly advantageously enables faster iteration cycles for improving or ensuring the quality of the synthesized data, as well as cooperation and knowledge exchange between different users of the distributed system. The distributed system advantageously enables cooperation of different entities without the need to disclose the technology (e.g., perception functions, tools for generating synthetic data), "know-how" and intellectual property of the respective instance.

[0015] Furthermore, it is possible that the method may further comprise the following steps:

[0016] - adapting the generating by said one or the other of said at least two instances of said distributed system based on a result of the evaluation.

[0017] In the scope of the adaptation, for example, the synthetic data and / or the tools used for generating the synthetic data are changed. This can advantageously improve the synthetic data in terms of its proximity to the reference data. It can be provided that the synthetic data is frequently adaptively generated and evaluated with respect to the reference data until the result of the evaluation indicates that the quality of the synthetic data is sufficient. Preferably, the generation and the adaptive generation are performed by the same instance.

[0018] Preferably, within the scope of the present invention, it can be provided that the adaptation generation comprises the following steps:

[0019] - changing at least one parameter of a tool, wherein said tool is configured to generate said synthetic data,

[0020] - generating said synthetic data based on the described use case while using said tool.

[0021] The tool may be, for example, a rendering tool or an enhancement tool or a machine learning model or a neural network. An automated optimization tool may be provided, wherein the change of at least one parameter of the tool and the evaluation of the synthetic data relative to the reference data are repeated successively until the quality of the synthetic data is ensured.

[0022] Furthermore, it is advantageous if the evaluation includes the following steps:

[0023] - comparing similarities between said synthesized data and said reference data so as to provide a first metric of said at least one metric.

[0024] In order to compare the similarity between the reference data and the synthetic data, a large number of metrics or comparison methods can be applied. For example, these metrics or comparison methods can be derived from the established operating mode (for example, research on compression algorithms), or can also be developed specifically for use cases. The comparison can be carried out at different granularity levels with respect to the synthetic data and the reference data. For example, the complete data set / partial data set can be compared as the distribution of characteristics, features or metric values. In addition, each paired image or paired image part or semantic image area (for example, a person or a vehicle) composed of the synthetic data and the reference data can be compared. The applied method for comparing in different granularity levels can be based on, for example, pixel-by-pixel comparison, comparison based on features or image characteristics in local neighborhoods, comparison of feature spaces, potential space of convolutional neural networks or feature embedding (for example, CLIP embedding). Similarly, a machine learning model specially trained for such comparison is conceivable.

[0025] In another possibility it can be provided that the evaluation comprises the following steps:

[0026] - performing at least one perceptual function using the synthetic data and the reference data, respectively,

[0027] - comparing the results of the respective executions to provide a second metric of the at least one metric.

[0028] Thereby, an assessment can advantageously be made with respect to a target application of the at least one perceptual function. Preferably, the at least one perceptual function is executed in order to provide a forecast. A respective forecast of the at least one perceptual function based on the synthetic data and the reference data can be a result of the respective execution and compared in order to provide a second measure of the at least one measure.

[0029] According to another advantage, it can be provided that communication between at least two instances and / or use cases and / or configuration of at least one perception function and / or at least one metric is also provided via the at least one interface. This advantageously allows distributed processing by different users, which can accelerate the improvement of the synthetic data based on the evaluation. For example, communication and processing can take place via the at least one interface via a cloud-based solution.

[0030] Furthermore, it is advantageous within the scope of the invention if the synthetic data imitate sensor data and / or image data and / or reference data resulting from measurements of at least one sensor, in particular recordings of at least one camera and / or if at least one perception function is an image classification function of a machine learning model.

[0031] It is possible to use at least one perception function in a vehicle. The vehicle can be designed, for example, as a motor vehicle and / or a passenger vehicle and / or an autonomous vehicle. The vehicle can have, for example, a vehicle device and / or a driver assistance system for providing autonomous driving functions. The vehicle device can be implemented for at least partially automatically controlling and / or accelerating and / or braking and / or steering the vehicle based on at least one perception function.

[0032] The synthetic data and / or the reference data may, for example, include image data derived from the acquisition using a camera sensor. Here, the environment and / or the traffic scene may be represented by the values ​​of the image points, preferably pixels, of the image data. Objects of the environment and / or the traffic scene may be detected if necessary by classification and preferably image classification based on these values. Classification and image classification may also be provided in the form of semantic segmentation (i.e., pixel-by-pixel or region-by-region classification) and / or object detection. The image data may, for example, be images of a radar sensor and / or an ultrasonic sensor and / or a lidar sensor and / or a thermal imaging camera. Accordingly, the image may also be implemented as a radar image and / or an ultrasonic image and / or a thermal image and / or a lidar image.

[0033] The subject of the present invention is also a computer program, in particular a computer program product, which comprises instructions which, when executed by a computer, cause the computer to perform the method according to the present invention. Thus, the computer program according to the present invention can bring the same advantages as have been described in detail with respect to the method according to the present invention.

[0034] The subject matter of the present invention is also a device for data processing, which is set up to carry out the method according to the invention. For example, a computer can be provided as the device, which executes the computer program according to the invention. The computer can include at least one processor, which is used to execute the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored and from which the computer program can be read out by the processor for execution.

[0035] The subject of the present invention may also be a computer-readable storage medium having a computer program and / or instructions according to the present invention, which, when executed by a computer, cause the computer to perform the method according to the present invention. The storage medium is, for example, configured as a data storage device, such as a hard disk and / or a non-volatile memory and / or a memory card. For example, the storage medium can be integrated into a computer.

[0036] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Further advantages, features and details of the invention are apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the accompanying drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. Among them:

[0038] Figure 1 Schematic visualization showing a method, a device, a storage medium and a computer program according to embodiments of the present invention,

[0039] Figure 2 A schematic diagram showing an evaluation unit according to an embodiment of the present invention is shown,

[0040] Figure 3 A schematic diagram showing a technical structure of a method according to an embodiment of the present invention,

[0041] Figure 4 A schematic diagram showing a distributed system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] exist Figure 1 Schematically shown in FIG. 1 are a method 100 , an apparatus 10 , a storage medium 15 and a computer program 20 according to an embodiment of the present invention.

[0043] Figure 1In particular, an embodiment of a method 100 for ensuring the quality of synthetic data 5 for at least one perception function 7 using a distributed system 1000 is shown. In a first step 101, at least one interface 21, 22, 23, 24 of the distributed system 1000 is provided in order to provide access to an evaluation component 1 of the distributed system 1000 by at least two instances 30, 40, 50 of the distributed system 1000 via the at least one interface 21, 22, 23, 24. In a second step 102, a use case 31 for the synthetic data 5 is specified by one of the at least two instances 30, 40, 50 of the distributed system 1000. In this case, the use case 31 comprises at least at least one perception function 7, a description of the synthetic data 5 and at least one metric for evaluating the quality of the synthetic data 5. In a third step 103, synthetic data 5 is generated by one or the other of the at least two instances 30, 40, 50 of the distributed system 1000 based on the described use case 31. In a fourth step 104 , the synthetic data 5 are evaluated by the evaluation component 1 of the distributed system 1000 with respect to the reference data 4 using at least one metric in order to ensure the quality of the synthetic data 5 .

[0044] According to an embodiment, the invention evaluates the quality of synthetic sensor data, i.e., for example, image or video data or synthetic tools, for training or testing at least one perception function 7, which is to be applied to real data. For example, the invention describes a software solution to enable simple adaptation and execution of comparison methods, connection of data and perception function 7, fast iteration and feedback when developing synthetic data 5, and the sum of these advantages to enable novel collaborative work processes. This solution can distribute tasks in different ways, for example, so that data or tool suppliers can advantageously work more autonomously and more efficiently.

[0045] Advantages of the present invention according to embodiments are, for example, reducing application complexity for users, enabling fast iteration cycles and flexibility in application evaluation, and facilitating collaboration and knowledge exchange between different users.

[0046] According to an embodiment, the present invention describes a special validation software / architecture / platform for synthesizing image and video data, such as cameras, radars or lidars for developing or testing perception functions, especially DNN-based perception functions.

[0047] The present invention, according to an embodiment, automates complex experimentation and qualitative evaluation, and in particular enables an iterative improvement and evaluation process between data or tool clients and corresponding suppliers.

[0048] The present invention, according to embodiments, may further enable data / tool ​​suppliers to obtain fast and automatic feedback during the development of data / tools in order to improve the data or tools.

[0049] According to embodiments, the present invention can be used as a communication basis between data / tool ​​suppliers and customers by recording and exchanging the development status of data quality and tool functionality. According to embodiments, the present invention can be used as a tracking tool for the current development status of synthetic data.

[0050] The present invention can also enable agile work. In an agreement with a data / tool ​​supplier, the customer can further update the perceptual function or deep neural network or modify the evaluation criteria. The new evaluation criteria or perceptual function can be immediately used for automatic evaluation at the data / tool ​​developer.

[0051] According to the embodiments of the present invention, the design studio or data provider can plan and perform the necessary reference data or real data recording and other measurements by itself, so that these are optimally adapted to the workflow and tool operation of the design studio. As a result, the design studio can work more flexibly, more effectively and more efficiently, and achieve better results more quickly.

[0052] According to the embodiments, the present invention can furthermore enable continuous processes and business relationships and thus achieve scaling. As a communication basis, the present invention in particular standardizes, automates and accompanies data / modeling / tool ​​development processes and preferably enables continuity of processes and business relationships.

[0053] According to an embodiment, the present invention preferably compares a synthetic image from synthetic data 5 with a real image from reference data 4 with respect to the prediction quality of a machine learning model or a perceptual function, and may allow determination of parameters of a tool for data generation in order to obtain better consistency between the real or reference data 4 and the synthetic data 5.

[0054] According to an embodiment, the present invention describes a software system that enables automated evaluation of the suitability and quality of synthetic data 5 for a specific use case not only for data providers of synthetic data 5 but also for consumers.

[0055] To this end, the required requirements and prerequisites for data production can be managed for the defined use cases. In addition, real data can be provided or specially generated as reference data 4. In addition, synthetic data 5 that maps the reference data 4 to a twin can be generated by a simulation tool or provider. In addition, the reference data 4 and the synthetic data 5 can be compared and evaluated in an automated manner by software components with the help of suitable methods. In addition, automated assessment and evaluation can be used to verify the suitability of the simulation and iteratively improve it.

[0056] According to an embodiment, the present invention is, for example, a technical framework for evaluating and iteratively improving synthetic sensor data (e.g., image / video data) that should be used to test or train a perception function 7 (e.g., based on neural networks, deep learning, artificial intelligence methods). The synthetic data 5 can be generated here, for example, by generative artificial intelligence methods (e.g., GAN, diffusion), rendering or ray tracing methods from computer graphics, or by enhancement methods. Here, the goal can be to ensure the quality of the synthetic data 5 for use cases, such as validation and testing, and to enable efficient collaboration between users (e.g., data consumers and data providers).

[0057] The following description includes the basic mathematical principles used for the evaluation, the technical structure, and an interaction scheme that shows how different people and organizations can interact based on the role assignments.

[0058] According to an embodiment, the evaluation unit 111 comprises an evaluation component 1 and various interfaces 21, 22, 23, 24. Figure 2 In the embodiment of the present invention, the evaluation unit 111 comprises in particular a sequence of functions and methods 2, 3, 6, 7, 14, 11, 13, which can be flexibly exchanged and expanded in the form of plug-ins and are used in particular for comparing reference data 4 and synthetic data 5 and evaluating the quality based on this comparison. In this case, the comparison of synthetic data 5 and reference data 4 is preferably carried out in a reference form (direct comparison) 6 and with respect to the output of a computer vision (CV) function by applying a perception function 7, wherein the computer vision function is, for example, the perception function 7 according to an advantageous embodiment. The selection and configuration of the plug-in and the retrieval of the results and visualizations can be carried out via various interfaces 21, 21, 22, 24.

[0059] The flow of evaluation according to the embodiment may be performed as follows.

[0060] Data from different domains (e.g. real and synthetic) for different use cases (e.g. object detection or traffic sign recognition) may have different data structures and annotation formats. These data sets may be loaded by a suitable data loader 2, which is implemented in the form of a plug-in. The data loader 2 preferably loads sensor or image data, ground truth data and other attributes characterizing sensor data. The data loader 2 and the selection of data may be controlled by an associated interface 21.

[0061] The data are preferably suitably preprocessed (eg normalization for computer vision functions) by means of a preprocessing plug-in 3 for comparison and application, and aligned with each other (eg time synchronization of data, geometric registration or assignment of various properties and parameters).

[0062] The loaded and preprocessed reference data 4 and synthetic data 5 are used, in particular, as input for a reference-based comparison 6 and for applying relevant computer vision (CV) functions or perception functions 7 .

[0063] Both the evaluation of the results of the execution of computer vision functions, in particular perception functions 7, and the evaluation of the results of the execution of computer vision functions, in particular perception functions 7, can be flexibly implemented by means of plug-ins 6, 14 and thus adapted to specific use cases. The selection and configuration of plug-ins 6 and 7 preferably takes place via corresponding interfaces 22 and 24.

[0064] In order to make a direct comparison 6 between the reference data 4 and the synthetic data 5, a large number of metrics or comparison methods can be applied. For example, these metrics or comparison methods can be derived from established operating methods (for example, studies on compression algorithms), or can be developed specifically for the use case. The reference comparison between the reference data 4 and the synthetic data 5 can be performed at different granularity levels with respect to the data. That is to say, for example, complete data sets / partial data sets can be compared as distributions of characteristics, features or metric values. In addition, separate paired images (for example, extracted from the reference data / synthetic data, respectively) or paired image parts or semantic image areas (for example, people, vehicles, etc.) can be compared. The methods used for comparison at different granularity levels can be based on, for example, pixel-by-pixel comparisons, comparisons based on features / image characteristics in local neighborhoods, comparisons of feature spaces, latent spaces of CNNs, feature embeddings (for example, CLIP embeddings). Similarly, artificial intelligence methods trained in accordance with the data, especially for such comparisons, are conceivable.

[0065] The perception or computer vision function 7 can be applied not only to the reference data 4 but also to the corresponding synthetic data 5 in order to assess the data with respect to the target application. The results of the computer vision function for each data domain 8, 9 (i.e. the results when applied to the reference data 8 and the results when applied to the synthetic data 9) are preferably used as inputs for a component implemented as a plug-in for comparing the results 14, which can be configured via an interface 24. The comparison of the corresponding predictions by the perception function 7 can be carried out in a use case-specific or domain-specific manner. The comparison 14 is in particular a measure of the extent to which the synthetic data 5 leads to the same positive and negative situations as the reference data 4 with respect to the perception function 7. For this purpose, domain-specific metrics and operating methods can be applied, i.e. in particular the evaluation methodology established for the corresponding type of computer vision function or perception function 7, however, the evaluation is preferably carried out for the reference data 4 / synthetic data 5 pair. Thus, for example, in the case of object detection by means of bounding boxes, the predicted boxes for the real image of the reference data 4 can be compared with the predicted boxes for the synthetic image of the synthetic data 5. For bounding boxes, this is done, for example, by matching the boxes (reference data to synthetic) and calculating the Intersection-over-Union (IoU) or the number of matches. For other use cases / CV functions, such as optical flow, for example, the angular error or endpoint error can be calculated between the output of the method used to estimate optical flow for the reference data 4 and the synthetic data 5.

[0066] The calculated metrics and comparisons and, if necessary, intermediate results from the components 6 and 14 are preferably fed to a quality assessment plug-in 11, which can be configured via a metrics and assessment interface 24. For example, these plug-ins 11 evaluate the similarity of the corresponding predictions 14 of the perception or computer vision function 7, taking into account the similarity based on a direct comparison 6 of the synthetic data 5 and the reference data 4. In the case of very similar pairings of images (reference data 4 and synthetic data 5, respectively), the predictions can be expected to be similar or identical. Based on the calculated metrics and results, error situations (such as poor agreement between reference data and synthetic data) can additionally be identified, quantified and evaluated, and correlations between image characteristics, metrics and deviations can be correlated and quantified in the perception or computer vision function 7. Technically, the metrics calculated by the components 6 and 14 for different granularity levels (e.g., data sets, single images, image parts, objects in images) and all attributes / parameters characterizing the synthetic and reference data can also be aggregated in a suitable data structure 12.

[0067] The result 12 of the quality assessment 11 can be converted into a suitable graphical evaluation by means of a visualization plug-in 13. The results, intermediate results and visualizations can be made available, for example, via an interface 23.

[0068] According to an embodiment of the present invention, the following further describes Figure 3 The technical structure of the embodiment in the embodiment represents interfaces, configurations and technical components. The structure in particular enables multiple users 75 or partners to instantiate the evaluation unit 111 and interact with each other via user interfaces 60-62, wherein the evaluation unit comprises an evaluation component 1 and interfaces 21, 22, 23, 24 (in Figure 3 24 for clarity). The evaluation unit 111 allows the user 75 to work on common resources 65-70 and use cases 74 via configuration 64, respectively. The evaluation unit 111 can be adapted to the technical requirements of the various use cases by programming plug-ins 72 using an open source API 73, which are loaded by the evaluation component 1 at startup.

[0069] In order to prepare one or more instances of the evaluation unit 111, the user 75 can define and prepare local data storage 65-69, computing resources (including GPUs, servers, etc.) 70, 71, users and access rights, and agree on which data is where and which processes are executed. Information about data and computing resources is exchanged between users 75, and users 75 define configuration files 64 for their corresponding evaluation components 1. Resources 65-71 can be internal or external for users 75, that is, for example, data consumers 30, synthetic data and / or tool suppliers 50 or reference data suppliers 40, and can be opposite for other users 75. For example, the local storage of user A can be an external storage for user B, and the same is true for computing resources. If necessary, data storage and computing resources 65-71 can be provided by cloud operators and used by users 75.

[0070] The test can then be accessed and executed. The evaluation unit 111 now preferably has access to local and external data stores and has the possibility of executing processes locally or externally.

[0071] The user 75 can perform the instantiation of the evaluation unit 111 as a web server 60 , as a CLI or GUI application 61 or in a programming language as an API 62 .

[0072] When the evaluation unit 111 is started, a plug-in 72 is preferably loaded, which plug-in can be programmed via an open source API 73. The plug-in can operate the programming technology interface expected by the evaluation unit 111 and access the data and computing resources 65, 66.

[0073] The data loader plugin 2 in particular accesses memories 65, 66 for synthetic data 5 and reference data 4. The data loader plugin 2 can implement virtual generation of subsampling and subsets in order to generate virtual partial data sets according to specific criteria. Different data subsets and data sampling methods may be required for different metric plugins 6, 14, 11. In particular, a deep neural network (DNN) plugin 7 for at least one selected perceptual function 33 of the use case 31 uses, for example, a DNN from a memory for a DNN 69, and can execute the DNN on a computing resource 70, which may require a GPU. In particular, there is a plugin 76 for controlling a tool 52 for synthetic data 5, wherein the tool 52 is executed on a corresponding computing resource 71 for the tool 52. With this plugin 76, the evaluation unit 111 can automatically change and optimize the parameters of the tool 52 for synthetic data 5 in order to generate better synthetic data 5 or a specific required situation and in order to improve the tool 52.

[0074] For the use case definition 74, all properties of the use case and use case management can be defined by the data consumer 30 via the interfaces 21, 22, 23, 24, which are mapped to elements of the user interface 60-62. Figure 3 For the sake of clarity, the elements 31 - 37 of the use case definition 74 have been shortened to 31 . . . 37 . The user 75 or the partner can read and, if necessary, edit the data via the evaluation unit 111 . The use cases are preferably stored in a use case registry 67 .

[0075] Via the user interface 60-62, the user 75 or the partner can use the evaluation component 1 of the evaluation unit 111 to evaluate the quality of the synthetic data 5 or the quality of the tool 52 for synthetic data 5. In this case, individual image pairs or image sets can be selected and evaluated, or the evaluation component 1 automatically calls the tool 52 for synthetic data generation with the help of the plug-in 76 and optimizes the parameters of data generation. The results 12 of the measurement plug-ins 6, 14, 11 are stored in particular in a memory for the evaluation data 68 related to the use case and all variation parameters. The result and visualization plug-in 13 preferably stores the data in the memory 68 and can allow visualization via the user interface 60-62.

[0076] Figure 4 An example is shown of different user roles or instances 30, 40, 50 for a person or an organization and how they interact with one another by means of the evaluation unit 111. A person or an organization may, if necessary, utilize a plurality of these roles or instances 30, 40, 50. Other interaction flows are also possible.

[0077] The data consumer & use case manager 30 preferably defines use cases 31. In particular, this includes the selection 33 of DNNs or computer vision or perception functions 7 (based on the available DNNs in the DNN memory 69, which are available via DNN plugins) and the selection 32 of metrics based on the available plugins 6, 14, 11. In addition, for example, the requirements 34 of the use case 31 are detected and agreed with the reference data supplier 40 and the synthetic data and / or tool supplier 50. In addition, other data 35 can be provided, such as existing measurement data such as drone recordings or light probes and, if necessary, cameras or other hardware 36. This information is detected, in particular, via the interfaces 21, 22, 24 of the user interface 60-62 of the evaluation unit and can be made available to the user 75 via the user's instantiated evaluation unit 111. This information can help to distinguish and organize the use cases 31.

[0078] The role of the reference data supplier 40 can be implemented multiple times if necessary and can also overlap with the data consumer & use case manager 30 or the synthetic data and / or tool supplier 50. In this role, in particular, real reference data 4 are recorded by sensors 41 for comparison with synthetic data 5 and provided to the evaluation component via the interface 21 for evaluation. For the user, the interface 21 is depicted in particular in the scope of the user interface 60-62; technically, the interface for the data can be implemented by the data plug-in 2. The data can be stored in a memory 66 for reference data. The memory can be located locally for some users 75 or roles 30, 40, 50 and externally for others.

[0079] The synthetic data and / or tool supplier 50 uses in particular information from the use case 31 and additional data 35 for modeling 51. The data generation tool 52 and the modeling 51 can be implemented differently. For example, when using classic 3D rendering tools, 3D models and materials (textures) are created. When using enhancement tools, modeling consists in particular in programming the enhancement method and setting its parameters. When using DNN-based solutions (such as NeRF, GAN, diffusion networks, etc.), in particular DNNs are trained. Training data may be required for this purpose, and the programming and parameterization of these tools are adapted.

[0080] The synthetic data and / or tool supplier 50 now preferably uses the evaluation component 1 in order to evaluate its synthetic data 5 with respect to the real reference data 4 and get fast feedback 53. The evaluation is performed with respect to the selected DNN and metrics of the use case 31. The result and visualization plug-in 13 in particular calculates the results 12, stores and visualizes the results and maps the results via the result and visualization interface 23 of the user interface 60-62. The results 12 are stored in the memory 68 for evaluation data, for example in the context of the use case 31 and all the variation parameters.

[0081] The synthetic data and / or tool supplier 50 can use the feedback 53 in order to improve its data generation tool 52. In this case, the synthetic data and / or tool supplier has the possibility to request measurement data from the reference data supplier 40 on demand for modeling 51 (e.g. 3D modeling, tool programming or DNN training). If necessary, the roles of the synthetic data and / or tool supplier 50 and the reference data supplier 40 can overlap. Therefore, the synthetic data and / or tool supplier 50 can optimize its tool 52 and its working steps independently of the data consumer & use case manager 30. If the parameters of the data generation tool 52 should be optimized automatically, an optimizer 76 can be used, which changes the parameters of the tool 52 and automatically executes the tool on the computing resources 71 set up for this purpose via the optimizer & tool control plug-in 76.

[0082] The data consumer & use case manager 30 and the synthetic data and / or tool supplier 50 are preferably in exchange and use the results of the evaluation unit 111 and the visualization interface 23 as a basis for discussion for collaboration. The data consumer & use case manager 30 can use these results to evaluate the usability 37 of the synthetic data 5 and, if necessary, adapt the use case 31 in an agile manner.

[0083] The possible goal of this working method is not only to generate qualified synthetic data 5 once. The synthetic data and / or tool supplier 50 can advantageously improve its working steps, the required additional data 35 and its tools 52 in order to provide high-quality synthetic data 5 in a cost-effective manner in the future. The requirements for reference data suppliers 40 and working steps can be optimized using the synthetic data and / or tool supplier. The data consumer & use case manager 30 can establish a structured working method and thereby enable the evaluation of the required synthetic data 5 and tools 52 and the definition of requirements 34.

[0084] The above explanations of the embodiments describe the invention only within the scope of examples. Of course, the individual features of the embodiments can be freely combined with one another as far as technically reasonable, without departing from the scope of the invention.

Claims

1. A method (100) for ensuring the quality of synthetic data (5) for at least one perception function (7) using a distributed system (1000), comprising the following steps: - providing (101) at least one interface (21, 22, 23, 24) of the distributed system (1000) in order to provide at least two instances (30, 40, 50) of the distributed system (1000) with access to an evaluation component (1) of the distributed system (1000) via the at least one interface (21, 22, 23, 24), - specifying (102) a use case (31) for the synthetic data (5) by one of the at least two instances (30, 40, 50) of the distributed system (1000), wherein the use case (31) comprises at least at least one perceptual function (7), a description of the synthetic data (5) and at least one metric for evaluating the quality of the synthetic data (5), - generating (103) said synthetic data (5) based on a specified use case (31) by said one or the other of said at least two instances (30, 40, 50) of said distributed system (1000), - evaluating (104) the synthesized data (5) with respect to reference data (4) by an evaluation component (1) of the distributed system (1000) using at least one metric in order to ensure the quality of the synthesized data (5).

2. The method (100) according to claim 1, characterized in that: The method (100) further comprises the following steps: - adapting the generation (103) by said one or the other of the at least two instances (30, 40, 50) of said distributed system (1000) based on the result of the evaluation (104).

3. The method (100) according to claim 2, characterized in that: Adaptation generation (103) includes the following steps: - changing at least one parameter of a tool (52), wherein the tool (52) is designed to generate the synthetic data (5), - generating the synthetic data (5) based on the described use case (31) by using the tool (52).

4. The method (100) according to any one of the preceding claims, characterized in that The evaluation (104) includes the following steps: - comparing similarities between said synthetic data (5) and said reference data (4) in order to provide a first measure (6) of said at least one measure.

5. The method (100) according to any one of the preceding claims, characterized in that The evaluation (104) includes the following steps: - performing at least one perceptual function (7) using the synthetic data (5) and the reference data (4), respectively, - comparing the results of the respective executions in order to provide a second metric (14) of the at least one metric.

6. The method (100) according to any one of the preceding claims, characterized in that Furthermore, communication between at least two instances (30, 40, 50) and / or configuration of a use case (31) and / or at least one perception function (7) and / or at least one metric is provided via the at least one interface (21, 22, 23, 24).

7. The method (100) according to any one of the preceding claims, characterized in that The synthetic data (5) simulates sensor data and / or the reference data (4) is obtained from measurements of at least one sensor (41), and / or at least one perception function (7) is an image classification function of a machine learning model.

8. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to perform the method (100) according to any one of the preceding claims.

9. A device (10) for data processing, which is configured to carry out the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images