Method for artificially generating multiple images

Abstract

The present invention relates to a method for artificially generating multiple images, and more particularly to a method (100) for artificially generating multiple images of a three-dimensional scene, particularly for providing training and / or validation and / or test data as a machine learning system for application in a technical system (preferably a vehicle), wherein the following steps are performed automatically: providing (101) input data (201, 202), wherein the input data (201, 202) at least partially describes the scene; generating (102) multiple images by an image generator (50) to represent the scene from different perspectives through the generated images, wherein the image generator (50) uses at least one machine learning model for the generation; and providing (103) the generated images.

Description

Technical Field

[0001] This invention relates to a method for artificially generating multiple images. Furthermore, the invention also relates to computer programs, apparatus, and storage media used for this purpose. Background Technology

[0002] Image generation is an active research area in computer vision and generative models. In particular, diffusion models have become increasingly important in recent years because they can generate high-quality images by describing the image generation process as a stepwise transformation from noise into a structured image.

[0003] As described in Ho et al.'s (2020) book, "Denoising Diffusion Probabilistic Models" (see [1], references listed at the end of the document), diffusion models employ a two-stage process: first, the image is converted into noise, and then a model is trained to progressively remove the noise and generate a realistic image. These models have proven to be extremely effective and have achieved impressive results in image synthesis.

[0004] Specific examples of this type of model include Stable Diffusion [2] or Imagen [3]. What these models have in common is that they generate a single image based on the input. The input can be text, an image, an abstract scene, or similar content. There may also be variations, such as ControlNet, which provides more precise control over the composition of the scene.

[0005] However, in many fields, generating a single image is insufficient because complex scenes are captured by multiple cameras. This is the case, for example, in autonomous driving, 3D reconstruction, or filmmaking. This problem is particularly pronounced when an object appears in the field of view of multiple cameras. Existing generative models cannot guarantee that the object will look the same in different generated images. Summary of the Invention

[0006] The subject of this invention is a method, a computer program, an apparatus, and a computer-readable storage medium. Further features and details of the invention are derived from the corresponding dependent claims, description, and drawings. Herein, the features and details described with respect to the method of the invention also apply to the computer program, the apparatus, and the computer-readable storage medium of the invention, and vice versa; therefore, disclosures regarding the content of this invention are always mutually referential.

[0007] The subject of this invention is, in particular, a method for artificially generating (especially synthesizing) multiple images of a three-dimensional scene, especially for providing training and / or validation and / or testing data as a machine learning system, preferably for use in technical systems, preferably in vehicles and / or robots.

[0008] In the method of the present invention, the following steps may be provided, which are preferably performed automatically: - Provide input data, wherein the input data at least partially describes the scene, and for example specifically describes the visual characteristics and / or arrangement of one or more objects in the scene. - Multiple images are generated by an image generator to represent and preferably depict the scene through the generated images—particularly from different perspectives—wherein preferably, the image generator uses at least one machine learning model, such as a diffusion model, for the generation. - Provides the generated image.

[0009] The multiple images can be jointly generated by the image generator, preferably by taking into account the input data and / or information shared by the multiple images during the generation process. This has the advantage that the scene can be represented from different perspectives (particularly viewpoints and / or cameras), and objects visible in multiple perspectives of the scene (particularly simultaneously) can be represented more consistently.

[0010] In this way, the generation of artificial images of a scene can be performed with higher quality and greater efficiency—and preferably also includes the training of machine learning systems. In particular, the described method (in which multiple images are generated together in a single step) ensures a more consistent representation of the scene. This method can also be used to generate images as training data for different application domains, such as object recognition and autonomous driving.

[0011] Preferably, in the method of the present invention, the multiple images can be generated jointly (by an image generator), i.e., particularly in the same (i.e., preferably in a single dimension and / or in parallel) step.

[0012] Furthermore, optionally within the scope of this invention, the generated image includes: - The input data is transformed to an abstract feature space, particularly a BEV representation or a three-dimensional tensor, using an encoder (preferably a BEV encoder), especially for a bird's-eye view transformation. - The abstract feature space is transformed into an output tensor by a decoder (preferably a BEV decoder), which contains the generated image as a single image.

[0013] Therefore, this feature space preferably represents a bird's-eye view of the scene and is beneficial for generating consistent images with different perspectives.

[0014] Within the scope of this invention, BEV stands for Bird's eye view, also known as a top-down view, characterized by at least one of the following features: - The viewpoint is oriented from above or diagonally above. - The horizon is located above the center of the image or outside the image.

[0015] A BEV encoder (Bird's Eye View Encoder) can represent an encoder (known from the prior art) in a neural network used to transform data into a bird's-eye view. Correspondingly, a BEV decoder can be used for the inverse transform.

[0016] Furthermore, it is conceivable that the input data includes at least one of the following: - A textual description of the scene. - Scene layout, - A two-dimensional representation of the scene, preferably in the form of an input image. - A description of the position and / or orientation of the (especially virtual) camera, which serves as the starting point for the different perspectives. - One or more 2D or 3D bounding boxes representing the coordinates and descriptions of predefined objects in the scene.

[0017] Therefore, input data can be designed in various ways to provide the image generator with a comprehensive understanding of the scene. This can improve the accuracy and consistency of the generated images. Different descriptive formats allow for the simultaneous inclusion of global aspects (such as textual descriptions) and local details (such as the position and orientation of objects).

[0018] Furthermore, it can be envisioned that image generation aims to artificially (particularly synthetically) produce different perspectives, thereby expanding the data foundation regarding the scene. Thus, by artificially generating images from different perspectives, a comprehensive dataset can be created for training. This enables machine learning systems to capture more accurate and detailed representations of the scene, thereby improving their performance in applications such as object recognition and scene understanding.

[0019] According to another advantage, the multiple images can be specified as single images generated jointly by the image generator in the same step. This allows the image generator to use information from the other single images when generating one single image, and thus consider them as shared information. In other words, generating all images simultaneously by the image generator enables a consistent representation of the scene. By having the generator jointly process all scene understanding information, all single images can be generated in the same step, resulting in more efficient image generation.

[0020] Furthermore, within the scope of this invention, the image generator may be specified to use at least one diffusion model as the at least one machine learning model for the generation. Therefore, this model can benefit from the advantages of diffusion models, such as high image quality and the ability to generate realistic images. Using a diffusion model as a machine learning model allows for flexible adaptation to different scenes and object types.

[0021] In another possibility, the generated images can be specified as training and / or validation and / or testing data for a machine learning system used in a technological system (preferably a vehicle and / or robot) application. This training aims to train the machine learning system to perform classification (preferably pixel-based object detection) so that the technological system can be controlled, preferably based on the classification results. The scene is preferably a traffic scene and / or a scene in a robotic industrial facility. In this way, the model can help improve object recognition in complex scenes by generating training data for the machine learning system.

[0022] The methods and / or machine learning systems of the present invention can be used in vehicles. The vehicles may be, for example, motor vehicles and / or passenger cars and / or at least partially automated / autonomous driving vehicles. The vehicles may have vehicle devices, for example, for providing autonomous driving functions and / or driver assistance systems. The vehicle devices may be configured to at least partially automatically control, accelerate, brake, and / or steer the vehicle.

[0023] The method of this invention can be used to enrich the data foundation containing the generated images so that the data foundation can subsequently be used as a training dataset. Machine learning systems (particularly in the form of machine learning models) are trained using the generated images, particularly for classification, especially for object detection. Training may include using a training dataset to train the machine learning system or model for classification based on data points (such as pixels in image data), particularly image classification, preferably based on edge or pixel attributes (of the image data). The image data or digital images may, for example, originate from recordings by at least one sensor (preferably at least one camera, more preferably a vehicle camera), particularly recordings by a (vehicle) camera and / or of the vehicle's environment during vehicle movement. Recording may, for example, be performed by at least one camera of the vehicle. Classification can be used to identify objects in the environment depicted by the image data or digital images and / or to capture traffic scenes.

[0024] Classification can be used in a variety of technological applications. One example is its application in vehicles and / or robots. Based on classification, particularly at least one classification result, at least one control action can be initiated and / or executed, which is preferred for vehicles, robots, or other technological systems.

[0025] The classification results may include and / or be specific to at least one of the following: object category, object identification, location of object and / or obstacle (e.g., in or beside the direction of travel), presence of obstacle, description of traffic scene, hazard warning, number of objects, type and / or location of lane markings and / or lane boundaries, location and / or status of traffic lights, lane location, etc.

[0026] Based on the classification results, at least one control action of the vehicle can be initiated and / or executed. The control action may include at least one of the following: braking, steering, acceleration, overtaking maneuver, emergency braking, activation of the alarm system, activation of hazard warning lights, activation of turn signals, and light control.

[0027] By classifying obstacles, such as those directly in the direction of travel or beside it, appropriate control actions, such as deceleration or avoidance, can be initiated based on their location (e.g., depending on the expected vehicle trajectory).

[0028] For example, if the classification results indicate the presence of an obstacle and / or a potential collision in the direction of travel, braking can be initiated. Similarly, it is conceivable to identify lanes and / or lane boundaries based on classification, in order to enable the vehicle to travel at least partially automatically within the lane through control actions.

[0029] "Classification" and "image classification" can also include "object detection" or "object detection in an image." This should be understood in particular as the classification of whether an object exists in certain regions of an image. Furthermore, the terms "classification" and "image classification" can also refer to "semantic segmentation," especially in the form of pixel-by-pixel classification.

[0030] Accordingly, training can produce at least one trained machine learning model that can be used for classification and / or object detection. For example, inference can be performed in a vehicle. The data points of the input data can be, for example, pixels of image data or data based on these pixels for pixel-based classification and / or object detection. The input data can include sensor data and / or image data that are at least partially derived from sensor acquisition (preferably a camera sensor) and / or at least partially synthetic, i.e., particularly analogous to real-world sensor data. Specifically, it is conceivable to represent the sensor and / or vehicle's environment and / or traffic scene through the values ​​of image points (preferably pixels) in the image data. Preferred image classification and / or object detection can be performed based on these values. This allows, for example, the detection of objects in a traffic scene. The image data can be, for example, images from radar sensors, ultrasonic sensors, LiDAR sensors, and / or thermal imaging cameras. Accordingly, the images can also be radar images, ultrasonic images, thermal images, and / or LiDAR images.

[0031] The subject of this invention is also a computer program, and more particularly a computer program product, comprising instructions that, when executed by at least one computer, cause the computer to perform the method of the invention. Therefore, the computer program of the invention provides the same advantages as those described in detail in conjunction with the method of the invention.

[0032] The subject of this invention is also an apparatus for data processing configured to perform the method of this invention. As such an apparatus, for example, at least one computer may be provided, which executes the computer program of this invention. The computer may have at least one processor to execute the computer program. A non-volatile data memory may also be provided, in which the computer program is stored, and from which the processor may read the computer program for execution.

[0033] The subject of this invention can also be a computer-readable storage medium having the computer program of this invention and / or including instructions that, when executed by at least one computer, cause the computer to perform the method of this invention. The storage medium is, for example, configured as a data storage device, such as a hard disk and / or non-volatile memory and / or a memory card. The storage medium can be integrated into a computer.

[0034] Furthermore, the method of the present invention can also be implemented as a computer-based method. Alternatively or additionally, at least one of the disclosed method steps can be computer-based and / or automatically executed. Attached Figure Description

[0035] Other advantages, features, and details of the invention will become apparent from the following description, in which embodiments of the invention are described in detail with reference to the accompanying drawings. Here, the features mentioned in the claims and specification may each contribute to the essence of the invention individually or in any combination. In the drawings: Figure 1 A schematic visualization of a method, apparatus, storage medium, and computer program according to embodiments of the present invention is shown.

[0036] Figure 2 A further illustrative representation of the method according to an embodiment of the present invention is shown. Detailed Implementation

[0037] exist Figure 1 The diagram schematically illustrates a method 100, an apparatus 10, a storage medium 15, and a computer program 20 according to an embodiment of the present invention. Figure 1 The application of method 100 for artificially generating multiple images of a three-dimensional scene is also described according to an embodiment of the present invention.

[0038] According to the first method step 101, input data 201 and 202 can be provided, wherein the input data 201 and 202 at least partially describe the scene. According to the second method step 102, the multiple images can be generated (preferably synthesized) by the image generator 50 to depict the scene from different perspectives through the generated images. The image generator 50 can be used to generate the images using at least one machine learning model. According to the third method step 103, the generated images are provided. It is conceivable that the multiple images are jointly generated by the image generator 50 so that information shared by the input data 201 and 202 and / or the multiple images can be collectively considered in the generation 102 by the image generator 50. It is further conceivable, particularly as indicated by the fourth method step 104, that the images generated in this way are used to train a machine learning system.

[0039] according to Figure 2 Generating a 102-image may also include: - The input data 201, 202 are transformed into an abstract feature space 230 by encoder 210 (preferably a BEV encoder), specifically a BEV representation or three-dimensional tensor, particularly for the transformation to a bird's-eye view BEV. - The abstract feature space is transformed into an output tensor 240 by decoder 230 (preferably BEV decoder), which contains the generated image as a single image.

[0040] There are already conventional solutions for processing multi-view camera images and, for example, detecting objects[7]. However, these typically fail to generate new images.

[0041] To ensure consistent scene generation, a novel network architecture (and several variations thereof) is proposed according to embodiments of the present invention. A fundamental idea of ​​these embodiments is to generate all images of the scene simultaneously to ensure consistency. This can be achieved by generating multiple images into a coherent image stack, such as... Figure 2 As shown, it contains more details.

[0042] In the illustrated implementation, the network architecture receives input describing the scene. This could be, for example, text input 201 or a layout 202 such as multiple 3D boxes or similar content. This input is transformed into an abstract feature space by encoder 210. This could be, for example, a BEV representation 220, or it could be a general 3D tensor. It is then transformed by decoder 230 into a tensor 240 of the form H, W, N. C): Height x Width x Number of Channels x Single Image Consistent images can be generated by processing all images in the same dimension.

[0043] There are two methods that are similar in name and structure but different in function. Therefore, they are described in more detail here.

[0044] A method described in [5] combines a diffusion model and a bird's-eye view (also known as BEV) to improve the prediction of relevant objects in a scene. This can be, for example, based on semantic segmentation of multiple camera images. In contrast, in a variation of the present invention, new camera images are generated, particularly based on scene description.

[0045] In [6], the generation of different viewpoints is achieved through text input describing the camera's location. Therefore, the network has no information about other individual images while generating a single image. In contrast, in a variant of this invention, all images are generated in the same step, preferably in the same dimension.

[0046] By generating all images together, a consistent scene is ensured. In principle, it also saves resources (running time, energy) compared to the traditional method of generating N images sequentially instead of generating all N images in parallel.

[0047] Variants of this invention are used to generate multiple camera images of a scene. These images can be used to train a network that detects objects or similar content in the scene. Models trained or evaluated using the data generated by this invention can achieve better performance on the desired task.

[0048] The diffusion model weights of diffusion model 50 can be further quantified to further improve runtime and energy efficiency.

[0049] Variants of the present invention may include an encoder portion 210. This encoder portion may be implemented as a neural network (or a portion thereof) that creates an abstract representation of a scene based on text input or scene layout.

[0050] Furthermore, variations of this invention may include a decoder section 230. This decoder section may be implemented as a neural network (or a portion thereof) that generates multiple camera images based on an abstract representation.

[0051] A network can be designed such that the abstract representation is a BEV representation. A network can also be designed to generate images during the diffusion process.

[0052] Input data, especially scene layout, can have the following structure: 1. Multiple text inputs with corresponding locations. Each text input describes a part of the scenario.

[0053] 2. One or more 2D or 3D boxes, which are the coordinates and descriptions of a specific object.

[0054] 3. Abstract scenes (e.g., drawing, Canny edges, or similar content).

[0055] 4. Images of similar scenes.

[0056] Similar to common diffusion models, this network can also be used for scene-to-scene transformations (in StableDiffusion: Image-to-Image). For this purpose, for example, in the case of multiple camera images as input, an inversion process (e.g., Direct Inversion) can be applied to the network. This can be used, for example, to modify an existing scene, which can be real or generated.

[0057] Variations of the embodiments of the present invention may be based on and extended from at least one of the following references: [1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusionprobabilistic models." Advances in neural information processing systems 33(2020): 6840-6851.

[0058] [2] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2022.

[0059] [3] Saharia, Chitwan, et al. "Photorealistic text-to-image diffusion models with deep language understanding." Advances in neural informationprocessing systems 35 (2022): 36479-36494.

[0060] [4] Zhang, Lvmin and Rao, Anyi and Agrawala, Maneesh. “AddingConditional Control to Text-to-Image Diffusion Models”, IEEE International Conference on Computer Vision (ICCV) (2023).

[0061] [5] Zou, Jiayu, et al. "Diffbev: Conditional diffusion model forbird's eye view perception." Proceedings of the AAAI Conference on ArtificialIntelligence. Vol. 38. No. 7. 2024.

[0062] [6] Xu, Xiaojie, et al. "From Bird's-Eye to Street View: CraftingDiverse and Condition-Aligned Images with Latent Diffusion Model." 2024 IEEEInternational Conference on Robotics and Automation (ICRA). IEEE, 2024.

[0063] [7] Li, Zhiqi, et al. "Bevformer: Learning bird's-eye-viewrepresentation from multi-camera images via spatiotemporal transformers." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[0064] The foregoing description of the embodiments illustrates the invention by way of example only. Of course, the various features of the embodiments can be freely combined without departing from the scope of the invention, provided it is technically reasonable.

Claims

1. A method (100) for artificially generating multiple images of a 3D scene, particularly for providing training and / or validation and / or testing data for a machine learning system used in a technological system application, preferably a vehicle, wherein, The following steps will be executed automatically: - Provide (101) input data (201, 202), wherein the input data (201, 202) at least partially describes the scenario. - The multiple images are generated (102) by an image generator (50) to represent the scene from different perspectives through the generated images, wherein the image generator (50) uses at least one machine learning model for the generation. - Provide the image generated by (103), Its features are, The multiple images are jointly generated by the image generator (50) so that the input data (201, 202) and / or the information shared by the multiple images are taken into account in the generation (102) by the image generator (50).

2. The method (100) according to claim 1. Its features are, The generation (102) of the image includes: - The input data (201, 202) is transformed to an abstract feature space (230) by an encoder (210), particularly a BEV representation or a three-dimensional tensor, especially for a bird's-eye view (BEV) transformation, wherein the encoder (210) is preferably a BEV encoder. - The abstract feature space is transformed into an output tensor (240) by a decoder (230), the output tensor containing the generated image as a single image, the decoder (230) preferably being a BEV decoder.

3. The method (100) according to claim 1 or 2. Its features are, The input data (201, 202) includes at least one of the following: - A textual description of the scenario. - The scene layout of the described scene, - A two-dimensional representation of the scene, - A description of the position and / or orientation of the camera that serves as the starting point for the different viewpoints, particularly a virtual camera. - One or more 2D or 3D boxes representing the coordinates and description of a predefined object.

4. The method (100) according to any one of the preceding claims. Its features are, The images are generated to artificially create the different perspectives and thereby expand the data base regarding the scene.

5. The method (100) according to any one of the preceding claims. Its features are, The multiple images are generated jointly by the image generator (50) as single images in the same step, such that the image generator (50) uses information from the other single images when generating (102) one of the single images, and thus considers this information as the shared information.

6. The method (100) according to any one of the preceding claims. Its features are, The image generator (50) uses at least one diffusion model as the at least one machine learning model for the generation.

7. The method (100) according to any one of the preceding claims. Its features are, The generated images are used as training and / or validation and / or test data for a machine learning system for use in a technical system application, to train the machine learning system to perform classification and, preferably, pixel-based object detection, so as to control the technical system, preferably a vehicle and / or a robot, based on the results of the classification. The scene is preferably a traffic scene and / or a scene in a robotic industrial facility.

8. A computer program (20) comprising instructions which, when executed by at least one computer (10), cause the computer to perform the method (100) according to any one of the preceding claims.

9. An apparatus (10) for data processing, the apparatus being configured to perform the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) including instructions that, when executed by at least one computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images