Method for automatically generating synthetic scene
By combining the generative model and scene parameters, the problem of insufficient object insertion control in the synthetic scene in the prior art is solved, real life-like and physically reasonable image generation is achieved, and the generation efficiency and quality are improved.
Patent Information
- Application Number
- CN202411158479.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-24
- Filing Date
- 2024-08-22
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art is difficult to effectively generate realistic synthetic scenes, especially the lack of control and automation when inserting rare objects, resulting in the generated images being not realistic and physically reasonable enough.
By providing template scenes and object data, generating intermediate representations using generative models (such as GAN or diffusion models), and adjusting data based on scene parameters to ensure physical rationality, ultimately generating realistic synthetic scenarios through machine learning models.
Automatic control of the position, direction and size of objects in the synthesis scene is realized, the generated images are more realistic, physically reasonable, and the process can be fully automated, improving the generation efficiency and quality.
Smart Images
Figure CN119515695A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for automatically generating a synthetic scene, and also relates to a computer program, a device and a storage medium for the purpose. Background Art
[0002] Generative models such as GANs or diffusion models are increasingly being used to generate realistic synthetic images that can be used for model training or validation. In areas such as autonomous driving, it is useful to integrate "rare" objects into synthetic scenes, so as to, for example, consider a non-operating motorcycle in the middle of the road. However, acquiring such rare scenes is quite difficult, making it an important task to generate them via generative models.
[0003] The technique of so-called "image inpainting" is known from the prior art. In particular, in text image diffusion models, image inpainting can be achieved by selecting a binary mask that describes the area where the object should be inserted. The object can then be described by a textual hint (see, for example, Rombach, Robin et al., "High-resolution image synthesis with latent diffusion models", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022). But the disadvantage of this case is usually the lack of control over how the desired object is generated. The use of a single text hint and a mask gives the generative model too much freedom when performing image inpainting. As a result, it is usually not possible to generate the desired object in a sufficiently realistic way.
[0004] Related research focuses on defining shape masks for improved controlled object placement (see, for example, Xie, Shaoan, et al., “SmartBrush: Text and Shape Guided Object Inpainting with DiffusionModel,” arXiv preprint, arXiv:2212.05034, 2022).
[0005] Another challenge and limitation of traditional methods is that of automation. For example, if a "non-working motorcycle on the road" is to be inserted, the size, orientation, position and other characteristics of the motorcycle must be created in the original scene. This ensures that the motorcycle remains physically plausible when inserted without human intervention. Automation of such processes raises some complex issues that have not yet been solved, or at least not solved enough, especially with regard to the industrialization of these practices. Summary of the invention
[0006] The present invention relates to a method having the features of the first aspect of the present invention, a computer program having the features of the eighth aspect of the present invention, a device having the features of the ninth aspect of the present invention, and a computer-readable storage medium having the features of the tenth aspect of the present invention. Other features and details of the present invention are derived from the specification and the drawings. In this regard, the features and details described in conjunction with the method of the present invention are obviously also applicable to the computer program of the present invention, the device of the present invention, and the computer-readable storage medium of the present invention, and vice versa, so they are always or can be mutually referenced with respect to various aspects of the present invention.
[0007] The present invention particularly relates to a method for automatically generating a synthetic scene, comprising the following steps:
[0008] - providing scene data representing a plurality of template scenes,
[0009] - providing object data specifying various objects particularly for insertion into the synthetic scene,
[0010] - providing the template scene with at least one scene parameter describing at least one characteristic of the template scene,
[0011] - generating an intermediate representation for the synthesized scene, wherein, for each of the intermediate representations, at least one selected object is inserted into the selected template scene, the insertion of the at least one selected object being parameterized based on the provided scene parameters to take into account physical plausibility in the respective intermediate representation,
[0012] - determining adjustment data from the generated intermediate representation in order to also take into account physical plausibility in the synthesized scene,
[0013] - Initiating synthetic scene generation based on the determined adjustment data, in particular by generating a (machine learning) model.
[0014] The present invention provides an improved solution for automatically inserting objects into synthetic scenes, and in particular aims to improve the generation of synthetic scenes and make them more realistic, efficient and versatile.
[0015] Hereinafter, the task of inserting an object into a scene (preferably a template scene) is also referred to as "image inpainting". In this case, the scene can be a synthetic scene or a real scene and / or based on a simulated scene and / or a real scene. In addition, the scene can be described by scene parameters such as Canny edge representation, depth map, semantic label map or a combination thereof.
[0016] The generative model has the ability to create realistic images representing synthetic scenes. The advantage is that the image generation process can be controlled so that the desired object (e.g., a non-working motorcycle in the middle of the road) is "placed" (painted) into the scene. Therefore, the shape, position and orientation of the placed object can be controlled, and the placement can be adapted to the image surroundings and can be objectively reasonable, for example in terms of size and distance. Another advantage is that this process can be automated, so absolutely no human input is required. In particular, this is used to obtain the specific task of generating synthetic images with and / or without the desired objects painted into the synthetic scene (rather than changing the real image). "Semantic image synthesis" is a possible way to achieve this task, especially when using GAN (GAN is an abbreviation for "Generative Adversarial Network"). In this case, the scene can be described by a semantic label map. For image restoration, objects can be painted in by modifying the semantic label map. But optionally, only objects such as those described by a limited number of classes in the semantic label map can be painted in. In contrast, the solution proposed by the present invention can have the ability to automatically introduce objects into the desired synthetic scene without this restriction, and in this way generate multiple images including the desired painted-in objects.
[0017] As another option, it can be provided that the synthetic scene is generated by a generative model, preferably a diffusion model. The machine learning model can generate the synthetic scene based on a scene definition, preferably a text prompt and preferably using at least one of the objects according to the object data. In this case, the generation can be adjusted by adjustment data, wherein the adjustment data is preferably used for this purpose when training the model to influence the model output. For example, scene parameters can be used to adjust and thus improve the generation of synthetic images with the help of conditional inputs in this way. The generative model can also be designed as a text image diffusion model such as stable diffusion. For example, ControlNet can then control the model output by processing the adjustment data and using this information to influence the model output. The text request can also be called a text prompt and, for example, includes a description (optionally a complete description) of the desired scene (such as the environment and the desired objects in the scene) in text form.
[0018] It is conceivable within the scope of the invention that the provided scene parameters include at least a depth map, a semantic label map and / or a Canny edge representation for use in the automatic determination of the position, orientation and / or size of objects for insertion into the template scene. This allows for physically reasonable insertion into the template scene. It is also possible that the provided scene parameters include at least one of the following parameters for describing the scene: normal map, soft edges, M-LSD lines, scribbles, OpenPose.
[0019] For example, it can be provided that the physical plausibility of the intermediate representation is at least taken into account and maintained in such a way that the insertion of at least one selected object is parameterized with respect to the object position using the provided scene parameters, preferably for determining the spatial placement of the object in the template scene as a function of the driving situation and / or the template scene environment. This optionally allows a fully automated and therefore more efficient generation of synthetic scenes, for example for training a machine learning model.
[0020] Furthermore, it can be provided that the object data comprises at least one data set with different objects, wherein the objects comprise at least a tire and a motorcycle, and the at least one data set provides each of said objects in two-dimensional form as a 2D image and / or in three-dimensional form as a 3D model, in the case of 3D models, the 3D models differ from one another in parameterization of size, angle, pose and / or orientation. Many scene variations can be generated, for example, in order to allow comprehensive training of a machine learning model based on synthetic scenes.
[0021] It is also possible within the scope of the present invention that the template scene includes a plurality of virtual and / or real driving scenes for autonomous driving, and the synthetic scene includes a plurality of virtual driving scenes, which are generated with reference to the determined adjustment data, in particular by influencing the weighting parameters of the generated model, so that information about the inserted objects and preferably about the layout of the respective intermediate representations are taken into account, wherein training data for training a machine learning model for autonomous driving is preferably provided based on the generated synthetic scene. The machine learning model trained in this way can be in a vehicle. The vehicle can be designed, for example, as a motor vehicle, a car and / or an autonomous driving vehicle. The vehicle can include a vehicle device for, for example, providing an autonomous driving function and / or a driver assistance system. The vehicle device can be designed to control the vehicle and / or accelerate and / or brake and / or maneuver the vehicle in an at least partially automatic manner.
[0022] For example, ControlNet can use generative models such as stable diffusion, and by influencing or changing the model with weighted parameters, it can also take layout information into account (otherwise stable diffusion cannot take layout information into account). Therefore, synthetic scenes can be used to generate synthetic images, and synthetic scenes are characterized by higher quality due to greater physical plausibility.
[0023] Determining adjustment data from the generated intermediate representation may also include at least the following steps: extracting at least one scene parameter from the respective intermediate representation, preferably a Canny edge representation, the Canny edge representation including edges from the template scene and the inserted object. By using the intermediate representation, important adjustment data can be generated for the final composite image generation. For example, the Canny edge containing edges from the original source image and the integrated object can be used as a guide for synthesizing a new image by stable diffusion. This allows for better synthesis of objectively reasonable scenes. The intermediate representation may also be referred to as a synthetic intermediate image.
[0024] The invention also relates to a computer program, in particular a computer program product comprising instructions which, when a computer runs the computer program, cause the computer to perform the method of the invention. The computer program of the invention thus provides the same advantages as those described in detail with respect to the method of the invention.
[0025] The invention also relates to a data processing device configured to perform the method of the invention. For example, a computer may be provided as a device for performing the computer program of the invention. The computer may include at least one processor for running the computer program. A non-volatile data storage device may also be provided, in which the computer program is stored and from which the computer program can be read by the processor for running.
[0026] The invention also relates to a computer-readable storage medium comprising a computer program according to the invention and / or comprising instructions which, when run by a computer, cause the latter to perform the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into a computer.
[0027] The method according to the invention can also be designed as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Further advantages, features and details of the invention are derived from the following detailed description of embodiments of the invention with reference to the drawings. In this regard, each of the features mentioned in the claims and the description may be essential to the invention either alone or in any combination, wherein:
[0029] Figure 1 Schematic diagrams showing methods, devices, storage media and computer programs according to embodiments of the present invention, and
[0030] Figure 2 Schematic diagram showing composite image and adjustment data. DETAILED DESCRIPTION
[0031] Figure 1A method 100 , an apparatus 10 , a storage medium 15 and a computer program 20 according to an embodiment of the present invention are schematically shown. Figure 1 A method 100 for automatically generating a synthetic scene 175 according to an embodiment of the present invention is shown. According to a first method step 101, scene data 110 is provided, which may represent a number of template scenes 120. The scene data 110 may be provided digitally, for example, via an interface of a computer program. According to a second method step 102, object data 115 is provided, which may specify various objects 125 for insertion into the synthetic scene 175. In this case, the object data 115 may be a data set specifying an object 125 (intended to be provided in a subsequent synthetic scene 175). In a third method step 103, at least one scene parameter 130 may be provided for the template scene 120, which describes at least one feature of the template scene 120. According to a fourth method step 104, at least one of the template scenes 120 may then be selected, and at least one of the objects 125 may be selected according to a fifth method step 105. The selection may optionally be done in an automatic and / or random manner. Furthermore, in a sixth method step 106, provision may be made for the generation of an intermediate representation 140 for the synthetic scene 175. With regard to the intermediate representation 140, at least one selected object 125 can be respectively inserted into the selected template scene 120. In this case, the insertion of the at least one selected object 125 can be parameterized based on the provided scene parameters 130 to take into account the physical plausibility of the respective intermediate representation 140. According to a seventh method step 107, adjustment data 150 can be determined from the generated intermediate representation 140 in order to also take into account the physical plausibility in the synthetic scene 175. Then in an eighth method step 108, the generation of the synthetic scene 175 can be initiated based on the determined adjustment data 150. In this case, the previously selected object 125 can advantageously be inserted into the scene 175 by a corresponding input in the text prompt.
[0032] Embodiments of the present invention may be based on "ControlNet", a known method described in "Adding conditional control to text-to-image diffusion models" by Zhang, Lvmin, and Maneesh Agrawala (arXiv preprint, arXiv:2302.055432023). In this case, stable diffusion can be used to create so-called "rare object image restoration". Stable diffusion is a deep learning text-to-image generator and can be used to generate detailed images based on text descriptions. ControlNet can be referred to as a neural network that is suitable for learning task-related conditions in an end-to-end manner.
[0033] Using ControlNet, the generation of synthetic images 170 with stable diffusion can be conditioned by conditional inputs. The conditional inputs may include, for example, Canny edges and / or depth maps and / or semantic label maps. These conditions may be extracted from real images or simulated environments. However, there are cases where it is necessary to add additional objects 125 that were not originally present in the scene. Embodiments of the present invention address this need and provide a simple and general solution for creating images that blend seamlessly into their environment while also taking into account user-specific content. In this case, ControlNet can also be applied to a pre-trained diffusion model and enhanced by the input option of conditional input.
[0034] ControlNet uses scene descriptions such as Canny edge representations, depth maps, semantic label maps, or a combination thereof to control the generation of synthetic images 170. According to an embodiment of the present invention, the conditioning aspects 150 can be modified by integrating objects 125 such as Canny edges into these conditions in a reasonable manner to ultimately create a synthetic but realistic image containing the desired object 125.
[0035] It may also be provided in embodiments of the present invention that depth maps and semantic label maps may be obtained and used to automatically determine the position, orientation and size of objects. As a result, the need for human intervention in such tasks may be eliminated. Thus, this approach paves the way for a more seamless, efficient and scalable image generation process that allows for a higher level of customization and control over the final visual result.
[0036] The use of depth maps, semantic label maps, and Canny edge representations can be understood as a multimodal approach. Although this information comes from the same source (i.e., the image), these scene parameters can represent different aspects of visual information and provide complementary scene information.
[0037] The depth map provides information, for example, about the distance or depth of objects in the scene relative to the camera, while Canny edge detection can emphasize the edges or boundaries of objects in the image. The semantic label map can represent the semantic category of each pixel in the image, which helps to understand the objects 125 in the scene and their relationships. By combining these data types, different representations of the same scene can be effectively fused, which can improve the performance and controllability of the generative model 50 (in this case, such as stable diffusion using ControlNet) by providing more diverse complementary information.
[0038] Embodiments of the present invention provide the ability to insert a desired object 125 into a synthetic scene 175. In this case, the presence of 2D or 3D may be a prerequisite, but is not limited to predetermined categories, as when semantic image synthesis is utilized. In addition, a high degree of control over the appearance of objects (such as position, angle (orientation), pose, etc.) is provided. This results in more realistic object insertion. In addition, when annotations such as depth maps and semantic label maps are available, it can be automatically determined where the desired object 125 should be placed in an objectively reasonable manner without the need for human intervention. Since no human intervention is required, image generation is more efficient and easier to scale.
[0039] Embodiments of the present invention are particularly capable of generating realistic images containing rare objects, i.e., objects that are rarely seen and captured in the real world, to enrich (enhance) the training and validation of autonomous driving systems. However, the proposed method is not limited to this application example and can be used in any scenario where image restoration is required. In addition, the method 100 according to an embodiment of the present invention can be used to generate training data for automatic training. In addition, an embodiment of the present invention is characterized by an end-to-end approach for generating restored objects in a pipeline.
[0040] According to an embodiment of the present invention, the ControlNet architecture can be used to adjust the stable diffusion model for general realistic synthetic image generation and in particular for driving scenes. By creating an adjusted input containing the desired (rare) object 125 when generating realistic images with ControlNet under these conditions, the object 125 can be integrated (image restoration) into the scene in a natural way. From this, the best location and / or the best size for inserting the desired object 125 can be calculated. In this way, the desired object 125 is seamlessly integrated into the environment and the insertion looks realistic.
[0041] An optional fine-tuning process may first be specified during training. Stable Diffusion (the base model) may have been trained using a pan-set of billions of images. Meanwhile, ControlNet (which allows for scaling of conditional image generation) may have been trained on different datasets tailored to specific conditional inputs. When using Canny edge conditioning, ControlNet may have been fine-tuned on a set of, for example, 3 million edge image pairs.
[0042] According to an embodiment of the present invention, ControlNet can be trained based on a data set that is specific to a given application (e.g., autonomous driving). In this way, it can be ensured that the synthetic image generation is better adapted to the target area.
[0043] Regarding reasoning, it can be assumed, for example, that there is a broad dataset including the target scene (driving scene) that is intended to be improved by synthetic data augmentation. Using this augmentation, new objects 125 can be inserted into the scene, thereby changing the scene composition. This dataset can include real-world images, or can also be derived from a simulation environment such as CARLA.
[0044] Regarding object selection, it can be assumed that there is a data set that includes various objects of interest that are intended to be integrated into different scenes. These objects may include, but are not limited to, tires, motorcycles, etc. The data set may include 2D images or 3D models. 3D models may also come from platforms such as Blender software, thus providing the ability to accurately change size, angle, posture, and orientation. This adaptation makes the inserted object 125 look more reasonable in its respective scene.
[0045] With respect to object insertion, after selecting a source image, the selected object 125 may be seamlessly added by superimposition techniques to form a synthetic intermediate image 140 (see FIG. Figure 1 ).
[0046] Physical plausibility should be ensured when responding to a request such as "insert a non-operating motorcycle into the middle of the road 20 meters from the vehicle." This can be achieved using two components: (1) semantic label map annotations and (2) a depth map that can be extracted from the source image via an inference network. The camera parameters in the depth map allow the distance from the camera (or the vehicle in this application example) to be calculated for each pixel in the image. Object 125 can then be positioned at the desired distance, for example 20 meters from the vehicle. Since the distance is known, the size of object 125 can be automatically determined to maintain physical plausibility.
[0047] Since the distance information can be known through the depth map (and the distance from the camera can be queried for each pixel in the image), the ideal position (such as in the middle of the road) of the desired object 125 can be automatically identified by querying the semantic label map that provides a category (such as road) for each pixel in the image. The orientation and position of the object 125 can be predefined or randomly selected, depending on the specific scene requirements.
[0048] Additionally, information about where the object 125 is located, such as 2D and 3D bounding boxes, may be determined and included in the output. This data may be important because these images are optionally used for subsequent object detection tasks that require knowledge (annotation) about the location of the object.
[0049] Regarding the further adjustment extraction step, the intermediate image 140 generated in the previous stage can be used to generate important adjustment data 150 for the final composite image generation. For example, the Canny edges containing edges from the template scene 120 (especially the original source image) and the integrated object 125 can be used as a guide for synthesizing a new image by stable diffusion.
[0050] ControlNet is compatible with a variety of different conditions, including scene parameters such as Canny edges, Hough lines, human key points, segmentation maps, shape normals, depth, etc. These conditions can be extracted from the image pool or can also be provided in the form of labels. By considering these different inputs, ControlNet provides a general and comprehensive solution for refining and enriching driving scenes, paving the way for more accurate and beautiful synthetic image generation. Since stable diffusion operates in text-to-image mode, additional conditioning examples can be provided through text cues such as "snow" or "night" to further refine the desired synthetic image 170.
[0051] In another step for generating the composite image 170, the conditioning information (Canny edge / semantic label map / depth map, etc.) from the previous step may be provided as a diffusion process input along with an edited or predetermined textual hint. The result may then be a composite image 170 including the desired scene and the integrated object 125. Figure 2 The result of the proposal pipeline is illustrated in the case of a person sneaking into the driving scene. This shows how the textual prompt can also affect the image generation of the composite image 170 (thus creating a night scene). The adjustment data 150 that may have been extracted from the intermediate representation 140 is also shown.
[0052] The above explanation of the embodiments only describes the present invention by way of example. As long as it is technically advantageous, the specific features of the embodiments can obviously be combined with each other at will without departing from the scope of the present invention.
Claims
1. A method (100) for automatically generating a synthetic scene (175), the method (100) comprising the following steps: - providing (101) scene data (110) representing a plurality of template scenes (120), - providing (102) object data (115) specifying different objects (125) for insertion into the synthetic scene (175), - providing (103) at least one scene parameter (130) for the template scene (120), the at least one scene parameter (130) describing at least one characteristic of the template scene (120), - generating (106) an intermediate representation (140) for the synthetic scene (175), wherein, with respect to the intermediate representation (140), at least one selected object (125) is respectively inserted into the selected template scene (120), wherein the insertion of the at least one selected object (125) is parameterized based on the provided scene parameters (130) to take into account physical plausibility in the respective intermediate representation (140), - determining (107) adjustment data (150) from said generated intermediate representation (140) in order to also take into account physical plausibility in said synthesized scene (175), - initiating (108) the generation of said synthetic scene (175) based on said determined adjustment data (150).
2. The method (100) according to claim 1, characterized in that: The synthetic scene (175) is generated by a generative model (50), preferably by a diffusion model (50), which generates the synthetic scene (175) based on a scene definition, preferably based on a text prompt, wherein the generation is regulated by the regulation data (150), wherein the regulation data (150) is preferably used for this purpose during the training of the model (50) to influence the output of the model (50).
3. The method (100) according to any one of the preceding claims, characterized in that The provided scene parameters (130) include at least a depth map and / or a semantic label map and / or a Canny edge representation for use in automatic determination of the position and / or orientation and / or size of the object (125) for insertion into the template scene (120).
4. The method (100) according to any one of the preceding claims, characterized in that Physical plausibility of the intermediate representation (140) is taken into account and maintained, at least because the insertion of the at least one selected object (125) is parameterized with respect to the position of the object (125) using the provided scene parameters (130) so as to preferably determine the spatial placement of the object (125) in the template scene (120) as a function of the driving situation and / or the environment of the template scene (120).
5. The method (100) according to any one of the preceding claims, characterized in that The object data (115) comprises at least one data set with different objects (125), wherein the objects (125) comprise at least a tire and a motorcycle, wherein the at least one data set provides each of the objects (125) in two-dimensional form as a 2D image and / or in three-dimensional form as a 3D model, wherein the 3D models differ from one another in terms of parameterization of their size and / or angle and / or posture and / or orientation.
6. The method (100) according to any one of the preceding claims, characterized in that The template scene (120) comprises a plurality of virtual and / or real driving scenes for autonomous driving, wherein the synthetic scene (175) comprises a plurality of virtual driving scenes, which are generated with reference to the determined adjustment data (150), in particular by influencing weighting parameters of a generation model (50), so that information about the inserted objects (125) and preferably about the layout of the respective intermediate representations (140) is taken into account, wherein training data is provided in order to train a machine learning model for autonomous driving based on the generated synthetic scene (175).
7. The method (100) according to any one of the preceding claims, characterized in that Determining (107) adjustment data (150) based on the generated intermediate representation (140) comprises at least the following steps: - extracting at least one scene parameter (130) from the respective intermediate representation (140), preferably from a Canny edge representation comprising edges from the template scene (120) and the inserted object (125).
8. A computer program (20) comprising instructions which, when executed by a computer (10), cause the computer program (20) to perform the method (100) according to any one of the preceding claims.
9. A data processing device (10) configured to perform the method (100) according to any one of claims 1 to 7.
10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer to perform the steps of the method according to any one of claims 1 to 7.