Method for providing synthesis data
By combining simulation environment and semantic image synthesis technology, the labeled realistic synthetic data is generated, which solves the problem of insufficient image quality in the existing technology, realizes high-quality image data generation, and improves the training effect of machine learning models.
Patent Information
- Application Number
- CN202411332927.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-27
- Filing Date
- 2024-09-24
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art is difficult to generate high-quality realistic image data, especially in the field of autonomous driving, where the image quality provided by traditional methods is insufficient to train and verify AI models.
By combining simulated environments and semantic image synthesis (SIS) technology, a realistic synthetic data with labels is generated. This method uses a simulation environment to generate annotated simulation data, and uses semantic image synthesis technology to convert semantic label maps into realistic images, providing high-quality annotated image data.
It realizes the generation of image information with improved realistic quality, similar to the real world, improves the effectiveness of synthetic data for machine learning and deep learning, and reduces the workload of manual annotation.
Smart Images
Figure CN119723226A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for providing synthetic data. The invention also relates to a computer program, a device and a storage medium for this purpose. Background Art
[0002] In the field of computer vision, generating realistic synthetic data for machine learning is an increasingly important task. Several methods known in the art, such as generative adversarial networks (GANs) and diffusion models, can be used to generate realistic images.
[0003] One category of synthetic image generation is semantic image synthesis (abbreviated as SIS). In SIS, a semantic label map for each pixel describes which class the pixel belongs to. Then, a generative model learns how to map this semantic label map that describes the scene at the pixel level (e.g., a car on the road) to a realistic image. Thus, the semantic label map describes the scene to be generated by such a generative model.
[0004] Driving simulators are an important tool in the field of autonomous driving. They support the development, training, and validation of autonomous driving systems. Such tools provide open digital assets (city maps, buildings, vehicles) for creating customizable driving environments. Simulation platforms such as the open source CARLA (see "Dosovitskiy, Alexey, et al." CARLA: An open urban driving simulator." Conference on robot learning. PMLR, 2017 (hereafter referred to as [1])) support flexible specification of sensor suites, environmental conditions, full control of all static and dynamic actors, map creation, etc. All simulation-related elements such as pedestrian behavior, weather, sensors, and traffic generation can also be controlled. Thus, desired and problematic edge cases (e.g., "a street full of pedestrians") can be generated. Another option is to simulate various sensor annotations. Users can configure a variety of sensor systems, including global positioning systems (GPS), depth sensors, laser radars (LIDAR), and numerous cameras.
[0005] Known solutions can provide images rendered using 3D engines, but they often lack photorealistic quality. In addition, such images are also different from the image data domain used to train or validate downstream AI models. Summary of the invention
[0006] In this context, features and details described in conjunction with the method according to the invention may obviously also be applied in conjunction with the computer program according to the invention, the device according to the invention and the computer-readable storage medium according to the invention, and vice versa, so that reference is always made to or can be made to the individual aspects of the present invention.
[0007] An object of the present invention is in particular a method for providing synthetic data for use in machine learning, wherein the synthetic data preferably comprises synthetic realistic image information and may therefore also be referred to as realistic synthetic data in the following.
[0008] In the method according to the invention, the following method steps can be provided and carried out in particular in a computer-assisted and / or automated manner:
[0009] - generating simulation data, wherein the simulation data is in the form of annotated data and / or can be generated by a simulation environment such as CARLA (see [1]),
[0010] - determining annotations from the generated simulation data, in particular by extracting annotations from the generated simulation data, wherein the annotations may be automatically generated in the context of a simulation by a simulation environment for rendering image information about the generated simulation data and / or provided as part of the generated simulation data,
[0011] - generating, in particular synthesizing, the synthetic data by means of semantic image synthesis (also called SIS) using the determined annotations as input for semantic image synthesis, wherein the semantic image synthesis preferably comprises at least one generative model, preferably a neural network, in particular a GAN (Generative Adversarial Network),
[0012] - annotating the generated synthetic data based on the output from the simulated environment, in particular in order to provide the generated annotated synthetic data, ie including associated annotated (synthetic) realistic image information.
[0013] Therefore, one advantage of the present invention is that it can not only synthesize realistic image information, but also automatically annotate it, thus creating an option for generating annotated realistic synthetic data and reducing the workload of manual annotation.
[0014] Furthermore, the simulation environment may be designed as a driving simulator in order to provide annotated synthetic data that can be used for machine learning in the field of at least partially autonomous driving.
[0015] Optionally, it is also provided that the output from the simulation environment includes the determined annotations, whereby the determined annotations are preferably at least partially carried into the generated synthetic data for the purpose of annotation. Synthetic data from such a simulation environment, in particular an autonomous driving simulation environment, can be used to train and verify machine learning models for various tasks (such as at least partially automatic and / or autonomous driving). This type of data provides a variety of traffic situations that are very difficult to observe and capture in the real world, and can play a key role in modeling boundary situations of at least partially automatic and / or autonomous driving. However, in traditional solutions, these data usually lack the required realistic quality and are therefore easily distinguished from actual data. Therefore, training based on such data is usually not very successful. The advantage that can be achieved according to the present invention is that image information with improved realistic quality can be generated, which is similar to the real world, and can achieve more effective "synthetic data expansion" for machine learning, in particular deep learning. According to the present invention, a combination of a simulation environment and semantic image synthesis (SIS) can be provided for this purpose, which is a category of generative modeling for deep learning. This enables semantic label maps (which are extracted, for example, from a simulation environment) to be converted into realistic images.
[0016] It is also conceivable that the generated synthetic data comprises at least one realistic image (i.e., in particular has realistic image information), and based on the annotation of the generated synthetic data, the at least one realistic image can be classified in an image element manner, in particular in a pixel manner, wherein the classification preferably comprises object detection and / or semantic segmentation. This enables the synthesis of training data suitable for training a machine learning model to be applied to image data captured by an actual sensor.
[0017] For example, it can be provided that the determined annotation comprises at least one semantic label map and / or at least one bounding box. Thus, the annotation can be suitable for training and / or validating classification tasks and can be used in particular for object detection and / or semantic segmentation of image information in the context of machine learning.
[0018] It can also be provided that the determination of the annotation based on the generated simulation data is performed according to at least one situation condition, wherein the situation condition is specific to the type of situation represented in the generated simulation data, preferably a traffic situation of the represented traffic scenario. An example of such a condition is as follows: "street full of pedestrians". This enables the capture of predetermined edge cases and extreme cases.
[0019] It is also conceivable within the scope of the invention to repeatedly perform these method steps in order to obtain a data volume of annotated generated synthetic data and / or to gradually increase this volume in each method step, whereby the data volume used in machine learning can represent a variety of different situations and objects. This enables artificial synthesis of data, which can be used for training and / or validating these situations and objects. For this purpose, these situations and objects can be simulated in advance by a simulation environment, in particular a driving simulator, in order to obtain appropriate annotations.
[0020] Preferably, provision can be made that the annotated generated synthetic data is used for machine learning by being used to train and / or validate a machine learning model, wherein the machine learning model is preferably trained for image classification (especially pixel-wise), in particular object detection and / or semantic segmentation based on the annotations, preferably for application in at least partially autonomous driving. Based on the output from the machine learning model, the machine learning model trained in this way can be used to at least partially automatically control, and / or accelerate, and / or brake, and / or steer a vehicle such as a motor vehicle, and / or a passenger car, and / or an at least partially automated vehicle. For this purpose, the machine learning model can use, for example, sensor and / or camera data from the vehicle as input.
[0021] The generated simulated data may also include simulated image information, which preferably has no or less realism than the realistic image information of the synthesized data. In this case, the annotation of the generated simulated data may annotate the simulated image information, in particular classify it pixel-wise and semantically, whereby, among the annotation and the simulated image information, preferably only the annotation is used as input for semantic image synthesis. In other words, this enables less realistic simulated image information to be exchanged with realistic image information while still obtaining the annotation.
[0022] The object of the present invention is also a computer program, in particular a computer program product comprising instructions which, when the computer program is executed by a computer, prompt the computer to perform the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with respect to the method according to the invention.
[0023] The present invention also aims at a device for data processing, which is configured to perform the method according to the present invention. For example, a computer can be provided as a device for executing a computer program according to the present invention. The computer can include at least one processor for executing the computer program. Non-volatile data storage means can also be provided, in which the computer program is stored, and from which the computer program can be read by the processor for execution.
[0024] The object of the present invention may also be a computer program according to the present invention and / or a computer-readable storage medium comprising instructions which, when executed by a computer, prompt the computer to perform the method according to the present invention. The storage medium is, for example, designed as a data storage device, such as a hard disk, and / or a non-volatile memory, and / or a memory card. The storage medium may, for example, be integrated into a computer.
[0025] The method according to the invention can also be designed as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Other advantages, features and details of the invention are apparent from the following description, in which embodiments of the invention are described in detail with reference to the accompanying drawings. In this context, each of the features mentioned in the claims and the description may be essential to the invention either alone or in any combination. In the drawings:
[0027] Figure 1 is a schematic diagram of a method, an apparatus, a storage medium, and a computer program according to an exemplary embodiment of the present invention.
[0028] Figure 2 is a schematic diagram illustrating semantic image synthesis (SIS) according to an exemplary embodiment of the present invention.
[0029] Figure 3 A simulated traffic scenario according to an exemplary embodiment of the present invention is shown as an example.
[0030] Figure 4 The integration of a driving simulator into a SIS according to an exemplary embodiment of the present invention is shown as an example.
[0031] Figure 5 is a simplified illustration of a process flow according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0032] Realistic synthetic images are an important tool for data augmentation and validation of artificial intelligence (AI) models for computer vision (CV). Using this tool to train machine learning algorithms for at least partially automated and / or autonomous driving can enable safety and reliability during the operation of at least partially automated and / or autonomous robots or vehicles.
[0033] In order to ensure the necessary safety, it is sometimes necessary for such AI models to be generalizable in various scenarios (e.g., rare objects in tunnels or on roads) and robust to various edge cases (e.g., unusual weather or road conditions). In order to achieve favorable generalization and robustness of the model, these rare cases must be sufficiently present in the training data. However, these cases are traditionally quite difficult and complex to collect.
[0034] One option for increasing the representation of these rare cases in the training data is to enrich the data with synthetic data samples. Synthetic data can also be used to validate computer vision AI models with edge cases such as severe weather.
[0035] Generative models typically provide "only" synthetic images. For many downstream tasks in which synthetic data is used, more annotations (hereinafter also referred to as "labels") may be provided and optionally required in some cases, such as two-dimensional bounding boxes (for specific objects), and / or three-dimensional bounding boxes, and / or radar point clouds, and / or lidar point clouds, and / or density depth, and / or the like. Object detectors require, for example, images with two-dimensional bounding boxes for detected objects for training. However, by default, SIS only provides images (and semantic label maps used as input), which is not sufficient in this case to train two-dimensional object detectors. On the other hand, the images provided by simulation platforms are usually insufficient in quality and in particular not realistic. In addition, such images are generally not suitable for training real AI systems due to a reason known as the "domain gap", as these systems require real-looking, realistic data points in training and validation.
[0036] Figure 1 Schematically shown in the method 100, the device 10, the storage medium 15 and the computer program 20 according to an exemplary embodiment of the present invention. The method 100 is used to provide synthetic data for use in machine learning. In this article, the synthetic data should include synthetic realistic image information, but this information can still be annotated in an automatic manner. For this purpose, simulation data can first be generated according to a first method step 101, wherein the simulation data is in the form of annotated data and is generated by a simulation environment. According to a second method step 102, the annotation can then be determined from the generated simulation data. According to a third method step 103, the synthetic data is generated by semantic image synthesis (SIS) using the determined annotation as an input of semantic image synthesis (SIS). It is made possible to annotate such generated synthetic data by a fourth method step 104, in which the annotation is based on the output from the simulation environment.
[0037] Therefore, one contribution of embodiments of the present invention is the ability to provide realistic synthetic images using all (or a desired subset) of the annotations described above. In particular, exemplary embodiments of the present invention provide an option for generating annotated realistic synthetic data, preferably by combining semantic image synthesis (SIS) with a simulation platform.
[0038] Simulation platforms such as driving simulators can use ground truth annotations such as semantic label maps, 2D bounding boxes, and optionally can also generate other annotations (labels) as specified above. The determined annotations (e.g. semantic label maps) can then be used as input for semantic image synthesis (SIS). This is Figure 2 As shown in , realistic images can be synthesized in this way using this input. The realistic images can then benefit from all other annotations delivered by the simulation platform, such as 2D bounding box annotations, semantic segmentation, depth information (depth map), etc. Figure 3 Examples of simulated image information 301 , depth information 302 , and semantic segmentation 303 that may be obtained by a simulation platform are shown.
[0039] refer to Figure 5 An example of a process for providing realistic synthetic data is described. Simulation parameters (e.g., camera parameters, maps, traffic conditions, etc.) may be initially selected and established (see 501). A simulation platform (e.g., CARLA) may then be started. Based on the simulation environment of the simulation platform, labeled data may be sampled at a specific frequency (see 502). Optionally, data is sampled only when a specific condition (trigger condition) is met. An example of such a condition is as follows: "a street full of pedestrians." This enables the capture of predetermined edge cases and extreme cases. In addition, all or a preselected set of real annotations from the simulation environment may be processed. Semantic label maps may then be extracted from the labeled data. The extracted semantic label maps may then be used as input to a pretrained SIS (e.g., OASIS). Realistic images (random appearance, i.e., random noise / latent vectors; see 503) may be generated in this manner. Optionally, instead of using random selection, a preselected latent vector defining the appearance of the image (e.g., thunderstorm, weather, etc.) may be provided to the SIS. The generated realistic images may then be annotated using information from the simulation environment. This is Figure 4 The steps specified above can be repeated until a representative number of data samples have been collected as the data volume for machine learning. The generated synthetic data patterns can then be used to train or validate AI models in downstream tasks (e.g., object detection).
[0040] One advantage of exemplary embodiments of the present invention is the ability to train a SIS model using exactly the same image dataset used to train a downstream AI model, thereby being able to provide synthetic images that are very similar to the same downstream task area. In addition, the workload for labeling and data collection can be reduced. The annotated data can, for example, include annotations in the form of two-dimensional and / or three-dimensional bounding boxes, and / or lidar and radar points. It is also possible to control the desired scene parameters (e.g., traffic density, number of pedestrians, etc.) for desired realistic synthetic data. It is also possible to extract "only" scenes that meet certain desired conditions (called "trigger conditions") from the simulation environment, such as desired edge cases, such as "many pedestrians crossing the road." This helps to focus on specific edge cases or other desired scenarios of the data generation process by filtering out these specified conditions.
[0041] The above explanations of the embodiments describe the invention only by way of example. The specific features of the embodiments may obviously be combined with one another as desired within the technically feasible range without departing from the scope of the invention.
Claims
1. A method (100) for providing synthetic data for use in machine learning, wherein: The synthetic data includes synthetic realistic image information, and the method includes the following method steps: - generating (101) simulation data, wherein the simulation data is in the form of annotated data and is generated by a simulation environment, - determining (102) annotations from the generated simulation data, - using the determined annotations as input to a semantic image synthesis (SIS), generating (103) the synthesized data by means of the semantic image synthesis (SIS), - Annotating the generated synthetic data based on output from the simulation environment (104).
2. The method (100) according to claim 1, characterized in that: The output from the simulation environment includes the determined annotations, wherein the determined annotations are at least partially carried over into the generated synthetic data for annotation (104).
3. The method (100) according to claim 1 or 2, characterized in that: The generated synthetic data comprises at least one realistic image, wherein the at least one realistic image is classified in an image element manner based on the annotation (104) of the generated synthetic data, wherein the classification comprises object detection and / or semantic segmentation.
4. The method (100) according to claim 1, characterized in that: The determined annotations include at least one semantic label map and / or at least one bounding box.
5. The method (100) according to claim 1, characterized in that: The determination (102) of the annotation based on the generated simulation data is performed according to at least one situation condition, wherein the situation condition is specific to a situation type represented in the generated simulation data.
6. The method (100) according to claim 1, characterized in that: The method steps are repeatedly performed to obtain a data volume of annotated generated synthetic data, which represents a variety of different situations and objects for machine learning.
7. The method (100) according to claim 1, characterized in that: The annotated generated synthetic data is used for machine learning by being used to train and / or validate a machine learning model, wherein the machine learning model is trained for image classification based on the annotations.
8. The method (100) according to claim 1, characterized in that: The generated simulated data includes simulated image information, which is characterized by having no or less realism than the realistic image information of the synthetic data, wherein the annotations of the generated simulated data annotate the simulated image information, wherein, among the annotations and the simulated image information, only the annotations are used as input for the semantic image synthesis (SIS).
9. A computer program product comprising instructions which, when executed by a computer (10), prompt the computer (10) to perform the method (100) according to any one of claims 1 to 8.
10. A device (10) for data processing, the device being configured to perform the method (100) according to any one of claims 1 to 8.
11. A computer-readable storage medium (15), comprising instructions which, when executed by a computer (10), prompt the computer (10) to perform the method (100) according to any one of claims 1 to 8.