Apparatus and computer-implemented method for generating a synthetic digital image of a three dimensional scene

Abstract

A computer-implemented method for generating a synthetic digital image (800) of a three dimensional scene, in particular for a dataset for training and / or testing a machine learning system.SOLUTION: The method comprises providing at least one text prompt, the at least one text prompt comprising a description of a three dimensional layout of the scene, the at least one text prompt comprising a description of the scene, generating the layout in dependence on the description of the layout, assembling the scene in dependence on the layout, determining a three dimensional Gaussian splatting representation of the assembled scene in dependence on the assembled scene, rendering the digital image from the three dimensional Gaussian splatting representation, and determining the synthetic digital image (800) in dependence on the digital image and the description of the style using stable diffusion.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] background The present invention relates to an apparatus and computer-implemented method for generating synthetic digital images of three-dimensional scenes. [Background technology]

[0002] Text-to-3D generative models can be used to generate synthetic digital images of three-dimensional scenes. Summary of the Invention [Means for solving the problem]

[0003] Disclosure of the Invention A computer-implemented method for generating a composite digital image of a three-dimensional scene, particularly for a dataset for training and / or testing a machine learning system, includes: providing at least one text prompt, the at least one text prompt including a description of a three-dimensional layout of the scene; generating a layout dependent on the layout description; assembling the scene dependent on the layout; determining a three-dimensional Gaussian splatting representation of the assembled scene dependent on the assembled scene; rendering a digital image from the three-dimensional Gaussian splatting representation; and determining a composite digital image dependent on the digital image and a style description using stable diffusion. This method enables hallucinating complex scenes containing multiple objects.

[0004] The at least one text prompt may include a description of a position of at least one object in the scene in a two-dimensional perspective, the at least one text prompt includes a description of an orientation of the at least one object in the scene in the two-dimensional perspective, and generating the layout includes creating a three-dimensional bounding box for the at least one object in the scene dependent on the position description and the orientation description. This method allows object-level control during scene generation.

[0005] Creating the bounding box includes determining the box center of the bounding box depending on the position description, and determining the box orientation of the bounding box depending on the orientation description.

[0006] Determining the position description and the orientation description includes providing a canonical coordinate system representing the scene in a two-dimensional perspective, dividing the canonical coordinate system into a grid containing a plurality of rectangular patches, selecting one of the patches, and generating a text description of the position and orientation depending on the position of the patch within the grid, thereby enabling generation of per-object text describing the position of the object in the scene.

[0007] Assembling the scene depending on the layout may include retrieving a 3D model of at least one object from a database containing 3D models of multiple objects, in particular retrieving a 3D model for the at least one object having a minimum Euclidean distance between dimensions of the 3D model and bounding box dimensions of the bounding box, and placing the retrieved 3D model of the at least one object in the scene with a box center and box orientation, thereby enabling the generation of an object in the scene that matches the bounding box dimensions.

[0008] Determining the composite digital image may include determining pixel values ​​of pixels in the composite digital image representing the at least one object dependent on pixel values ​​of pixels in the digital image representing the at least one object using stable diffusion, and setting pixel values ​​of pixels in the composite digital image not representing the at least one object to values ​​of pixels in the digital image not representing the at least one object, thereby generating the at least one object in the composite digital image without altering other portions of the digital image.

[0009] The method may include training a 3D Gaussian splatting representation and / or a stable diffusion representation relying on a loss that depends on values ​​of pixels representing the at least one object, whereby gradients are induced to propagate toward a target of the at least one object.

[0010] The method may include determining a binary mask indicating whether a pixel represents at least one object, and determining pixel values ​​for the pixels representing the at least one object according to the binary mask using stable diffusion.

[0011] The method may include generating different synthetic digital images for the datasets relying on the same 3D Gaussian splatting representation with stable diffusion, thereby generating different synthetic digital images due to the randomness of the stable diffusion.

[0012] The method may include providing at least one other text prompt, determining another three-dimensional Gaussian splatting representation dependent on the description of a three-dimensional layout of the scene in the at least one other text prompt, and determining another composite digital image for the dataset dependent on the another Gaussian splatting representation and the description of style in the at least one other text prompt, whereby different composite digital images are generated due to the different prompts.

[0013] Rendering a digital image from a 3D Gaussian splatting representation may include providing a viewpoint and rendering a view of the scene from that viewpoint.

[0014] The method may include providing three different viewpoints and determining, for the three viewpoints, a composite digital image showing the scene from each of the three viewpoints, using the three viewpoints provided by 3D Gaussian splatting.

[0015] An apparatus for generating synthetic digital images of three-dimensional scenes, particularly for datasets for training and / or testing machine learning systems, comprises at least one processor and at least one memory storing instructions, the at least one processor configured to execute instructions that, when executed by the at least one processor, cause the apparatus to perform the method.

[0016] A computer program for generating synthetic digital images of three-dimensional scenes, particularly for datasets for training and / or testing machine learning systems, comprises computer-executable instructions that, when executed by a computer, cause the computer to perform the method.

[0017] Further embodiments can be derived from the following description and drawings. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 shows a schematic diagram of an apparatus for generating synthetic digital images of a three-dimensional scene, particularly for a dataset for training and / or testing a machine learning system. [Figure 2] FIG. 1 is a diagram illustrating a schematic example of a three-dimensional layout. [Figure 3] FIG. 1 is a diagram illustrating a schematic diagram of an exemplary three-dimensional scene assembled based on a three-dimensional layout. [Figure 4] FIG. 10 is a diagram illustrating an exemplary 3D Gaussian splatting representation of an exemplary assembled scene. [Figure 5] FIG. 1 is a diagram illustrating a first exemplary digital image rendered from an exemplary 3D Gaussian Splatting representation, the image including a view from a first perspective. [Figure 6] FIG. 10 is a schematic diagram illustrating a second exemplary digital image rendered from the exemplary 3D Gaussian Splatting representation and including a view from a second perspective. [Figure 7] FIG. 10 is a schematic diagram illustrating a third exemplary digital image rendered from an exemplary 3D Gaussian Splatting representation and including a view from a third perspective. [Figure 8] FIG. 10 is a schematic diagram illustrating a first exemplary composite digital image including a view from a first viewpoint determined by stable diffusion from the first exemplary digital image and a style description. [Figure 9] FIG. 10 is a schematic diagram illustrating a second exemplary composite digital image including a view from a second perspective determined by stable diffusion from the second exemplary digital image and the style description. [Figure 10] FIG. 10 is a schematic diagram illustrating a third exemplary composite digital image including a view from a third perspective determined by stable diffusion from the third exemplary digital image and the style description. [Figure 11] FIG. 1 is a diagram illustrating a three-dimensional scene. [Figure 12] FIG. 1 shows a flowchart including steps of a method for generating synthetic digital images of a three-dimensional scene, particularly for a dataset for training and / or testing a machine learning system. DETAILED DESCRIPTION OF THE INVENTION

[0019] FIG. 1 shows a schematic diagram of an apparatus 100 for generating a synthetic digital image 102 of a three-dimensional scene 104 in dependence on at least one text prompt 106 .

[0020] The device 100 may be configured to generate synthetic digital images 102 for a dataset 108 for training and / or testing a machine learning system.

[0021] The device 100 includes at least one processor 110 and at least one memory 112. The at least one memory 112 is configured to store the composite digital image 102 and instructions for generating the composite digital image 102.

[0022] The device 100 may include an interface 114 configured to receive at least one text prompt 106. The interface 114 may be configured to output the composite digital image 102 and / or the three-dimensional scene 104.

[0023] At least one text prompt 106 includes a text description Y of the three-dimensional layout of the scene. At least one text prompt includes a description of the style of the scene.

[0024] Description Y may contain one sentence or multiple sentences. The sentences in description Y specify the positions and / or orientations of objects in the scene.

[0025] An exemplary description of an exemplary three-dimensional layout of an exemplary scene follows: [Table 1] "A double bed is placed in the center of the room, slightly against the far left wall, at a right angle. Near the upper left corner, a nightstand is positioned at a right angle. Near the lower left corner, another nightstand can be seen, also at a right angle. In the lower left corner, a wardrobe is placed with no particular orientation. Finally, in the upper right corner, a shelf is placed with no rotation."

[0026] An example for an exemplary style description of a scene is as follows: [Table 2] "Pirates of the Caribbean style."

[0027] The apparatus 100 is configured to generate a three-dimensional layout of a scene depending on a description of the three-dimensional layout of the scene.

[0028] The device 100 is configured to assemble the scene depending on the three-dimensional layout of the scene.

[0029] The apparatus 100 is configured to determine, dependent on the constructed scene, a 3D Gaussian splatting representation of the constructed scene.

[0030] The apparatus 100 is configured to render a digital image from a three-dimensional Gaussian splatting representation.

[0031] The apparatus 100 is configured to determine a composite digital image 102 using stable diffusion and depending on the digital image and the style description.

[0032] FIG. 2 illustrates a schematic diagram of an exemplary three-dimensional layout 200 .

[0033] The exemplary three-dimensional layout 200 includes a bounding box 202 for a double bed located in the center of the layout 200 .

[0034] The exemplary three-dimensional layout 200 includes a bounding box 204 for a nightstand positioned perpendicular to and set at the upper left corner 206 of the layout 200 .

[0035] The exemplary three-dimensional layout 200 includes a bounding box 208 for another nightstand located near the bottom left corner 210 of the layout 200, also set at a right angle.

[0036] The exemplary three-dimensional layout 200 includes a bounding box 212 for the wardrobe in the bottom left corner 210 with no particular orientation.

[0037] The exemplary three-dimensional layout 200 includes a bounding box 216 for the shelf without any particular rotation in the upper right corner 214 of the layout 200 .

[0038] FIG. 3 shows a schematic diagram of an exemplary three-dimensional scene 300 assembled according to the three-dimensional layout 200 .

[0039] The exemplary three-dimensional scene 300 includes a three-dimensional model 302 for a double bed located in the center of the scene 300 .

[0040] The exemplary three-dimensional scene 300 includes a three-dimensional model 304 for a nightstand positioned orthogonally in the upper left corner 306 of the scene 300 .

[0041] The exemplary three-dimensional scene 300 includes a three-dimensional model 308 for another nightstand located near the lower left corner 310 of the scene 300, also orthogonally oriented.

[0042] The exemplary three-dimensional scene 300 includes a three-dimensional model 312 for a wardrobe in the lower left corner 310 with no particular orientation.

[0043] The exemplary three-dimensional scene 300 includes a three-dimensional model 316 for a shelf without any particular rotation in the upper right corner 314 of the scene 300 .

[0044] FIG. 4 schematically illustrates an exemplary three-dimensional (3D) Gaussian splatting representation 400 of the exemplary constructed scene 300.

[0045] 3D Gaussian splatting represents the underlying scene as a collection of anisotropic 3D Gaussian distributions 402, each with its own center position μ∈R 3 and, Σ=RSS T R T It is defined by a 3D covariance matrix Σ parameterized as follows: where R refers to the rotation matrix and S is the scale matrix.

[0046] Each 3D Gaussian in 3D Gaussian splatting is assigned a color c represented by spherical harmonics (SH) coefficients to capture view-dependent appearance. To enable α-blending of the splats, the Gaussian is associated with an opacity value α∈R.

[0047] 3D Gaussian splatting allows for faster training and rendering with differentiable rasterization.

[0048] The 3D Gaussian distribution is rendered by projecting it onto the camera's image plane as a 2D Gaussian distribution and assigned to each image tile. The color of each pixel p in the image plane is then calculated as

number

[0049] For optimization, we can employ the L1 loss, i.e., the sum of all absolute differences between the true and predicted values, and the structural similarity index (SSIM).

[0050] FIG. 5 schematically illustrates a first exemplary digital image 500 rendered from the exemplary 3D Gaussian Splatting representation 400, which includes a view of a synthetic 3D scene from a first perspective.

[0051] FIG. 6 schematically illustrates a second exemplary digital image 600 that includes a view of a synthetic three-dimensional scene from a second perspective, rendered from an exemplary three-dimensional Gaussian splatting representation.

[0052] FIG. 7 schematically illustrates a third exemplary digital image 700 that includes a view of a synthetic three-dimensional scene from a third perspective, rendered from an exemplary three-dimensional Gaussian splatting representation.

[0053] The exemplary digital images show a double bed 502 positioned in the center of each digital image.

[0054] The exemplary digital images show a nightstand 504 positioned perpendicular to the upper left corner 506 of each digital image.

[0055] The exemplary digital images show another nightstand 508 also positioned at a right angle near the bottom left corner 510 of each exemplary digital image.

[0056] The exemplary digital image shows a wardrobe 512 in the bottom left corner 510 with no particular orientation.

[0057] The exemplary digital image shows a shelf 516 in the upper right corner 514 of the first exemplary digital image 500 without any particular rotation.

[0058] FIG. 8 schematically illustrates a first exemplary composite digital image 800 including a view from a first perspective determined by stable diffusion from the first exemplary digital image 500 and an exemplary style description.

[0059] FIG. 9 schematically illustrates a second exemplary composite digital image 900 including a view from a second perspective determined by stable diffusion from the second exemplary digital image 600 and the exemplary style description.

[0060] FIG. 10 schematically illustrates a third exemplary composite digital image 1000 including a view from a third perspective determined by stable diffusion from the third exemplary digital image 700 and the exemplary style description.

[0061] The exemplary composite digital images show a double bed 502 positioned in the center of each exemplary digital image.

[0062] The exemplary composite digital images show a nightstand 504 positioned perpendicular to the upper left corner 506 of each exemplary digital image.

[0063] The exemplary composite digital images show another nightstand 508 placed near the lower left corner 510 of each exemplary digital image, also oriented at a right angle.

[0064] The exemplary composite digital images show a wardrobe 512 in the lower left corner 510 of each exemplary digital image without any particular orientation.

[0065] The exemplary composite digital image shows a shelf 516 in the upper right corner 514 of the first exemplary digital image 500 without any particular rotation.

[0066] FIG. 11 shows a schematic diagram of an exemplary composite three-dimensional scene 1100, with multiple exemplary composite digital images each showing a different viewpoint.

[0067] The exemplary composite three-dimensional scene 1100 includes a double bed 502 located in the center of the exemplary composite three-dimensional scene 1100 .

[0068] The exemplary composite three-dimensional scene 1100 includes a nightstand 504 positioned perpendicular to the upper left corner 506 of the exemplary composite three-dimensional scene 1100 .

[0069] The exemplary composite three-dimensional scene 1100 includes another nightstand 508 positioned near the lower left corner 510 of the exemplary composite three-dimensional scene 1100, also oriented at a right angle.

[0070] The exemplary composite three-dimensional scene 1100 includes a wardrobe 512 in the bottom left corner 510 of the exemplary composite three-dimensional scene 1100 with no particular orientation.

[0071] The exemplary composite three-dimensional scene 1100 includes a shelf 516 in the upper right corner 514 of the exemplary composite three-dimensional scene 1100 without any particular rotation.

[0072] FIG. 12 shows a flow chart including steps of a method for generating a synthetic digital image of a three-dimensional scene.

[0073] The composite digital image is, for example, one of the exemplary composite digital images.

[0074] The method includes step 1202 .

[0075] Step 1202 includes providing at least one text prompt.

[0076] At least one text prompt includes a description of the layout of the three-dimensional scene.

[0077] The description of the layout includes, for example, a description of the position of at least one object in the scene in a two-dimensional perspective.

[0078] The description of the layout includes, for example, a description of the orientation of at least one object in the scene in a two-dimensional perspective.

[0079] Step 1202 includes providing at least one text prompt 106 that includes, for example, an example description of an example three-dimensional layout 200 and an example description of an example style.

[0080] An example description of the layout includes a description of the positions of the objects 502, 504, 508, 512, 514 in the scene 300, 1100 in a two-dimensional perspective.

[0081] The exemplary description includes a description of the orientation of the objects 502, 504, 508, 512, 514 in the exemplary scene 300, 1100 in a two-dimensional perspective.

[0082] A position description and an orientation description can be determined.

[0083] Determining the position description and the orientation description may include providing a canonical coordinate system that represents the scene in a two-dimensional perspective.

[0084] Determining the position and orientation descriptions may include dividing the canonical coordinate system into a grid containing a number of rectangular patches.

[0085] Determining the position description and the orientation description may include selecting one of the patches and generating a textual description of the position and orientation Y depending on the position of the patch within the grid.

[0086] Category name c i An exemplary description of the position and orientation of object i, identified by: [Table 3] "c i is installed in the upper left corner of the room, oriented vertically.

[0087] The position description and the orientation description may be determined based on rules. The position description and the orientation description may be determined using a large-scale language model, for example, using LayoutGPT (arXiv:2305.15393). A new position and / or orientation description may be determined from a given position and orientation description by prompting the large-scale language model to paraphrase the given description.

[0088] The method includes step 1204 .

[0089] Step 1204 includes generating a layout dependent on the description of the layout. Step 1204 includes, for example, generating the example layout 200 dependent on the example description of the example layout 200.

[0090] A layout contains at least one object at a position and orientation consistent with the layout description.

[0091] Generating the layout includes, for example, creating a three-dimensional bounding box for at least one object in the scene depending on the position description and the orientation description.

[0092] Bounding box b iCreating a bounding box b depends on a text description Y, for example. i Box center

number

number

number

[0093] Bounding boxes for multiple N objects

number

[0094] The method is not limited to box center, box dimension, and box orientation as bounding box values, and may use other representations of bounding box values.

[0095] The bounding box values ​​may be mapped to attributes in standard CSS format, with each bounding box category name c i may be taken as the selector for each bounding box.

[0096] Bounding box b i is the bounding box b i The language model may be generated by prompting the large language model to generate a

[0097] Building the prompt: The large-scale language model can be provided with a prompt that includes a given description of position and orientation, a given bounding box value, and an explanation that the large-scale language model should provide a predetermined bounding box value for the given description of position and orientation.

[0098] The large language model can be provided with a prompt that includes a further description of the position and orientation and a task to output further bounding box values ​​for the further description.

[0099] An exemplary prompt to a large-scale language model includes three parts: a task specification, an in-context example, and a query condition.

[0100] Task specification: A task description is embedded at the beginning of each prompt. The task description explains the goal of the task, establishes standards for 3D layout formatting in CSS styles, and provides unit information for bounding box values.

[0101] The task description may include constraints to guide the large-scale language model and minimize errors during task performance. Exemplary constraints include constraints on bounding box values ​​that prevent overlapping boxes from being predicted, or boundaries on bounding box values ​​that prevent bounding boxes from being placed outside the boundaries. The boundaries may be the boundaries of the scene.

[0102] In-context learning: Support examples for in-context learning are selected by adopting the search-based approach used in LayoutGPT.

number

number

[0103] Query criteria: Inference condition C q , the inference condition C q A large-scale language model is assumed to predict the layout for

[0104] For example, the example three-dimensional layout 200 is generated based on the description of the example layout 200 .

[0105] For example, three-dimensional bounding boxes 202, 204, 208, 212, 214 are generated for objects 502, 504, 508, 512, 514 in the scene 300, 1100 depending on an example position description and an example orientation description.

[0106] Creating the bounding boxes includes, for example, determining box centers, box dimensions, and box orientations for the bounding boxes 202, 204, 208, 212, and 214 depending on a description of the example scene.

[0107] The method includes step 1206 .

[0108] Step 1206 involves assembling the scene depending on the layout.

[0109] For example, the exemplary scene 300 is assembled based on the exemplary layout 200 .

[0110] Assembling a scene depending on the layout may involve, for example, retrieving a three-dimensional model of at least one object from a database containing three-dimensional models of multiple objects.

[0111] For example, a three-dimensional model having a minimum Euclidean distance between the dimensions of the three-dimensional model and the bounding box dimensions of the bounding box for at least one object is searched for.

[0112] Assembling a scene involves, for example, assembling the retrieved 3D model of at least one object i in the scene into a box center c i To box orientation i This includes installing it in

[0113] For example, three-dimensional models of objects 502 , 504 , 508 , 512 , and 514 are retrieved from a database to assemble exemplary scene 300 .

[0114] For example, the three-dimensional models of objects 502, 504, 508, 512, and 514 that have the smallest Euclidean distance between the dimensions of the respective three-dimensional models for objects 502, 504, 508, 512, and 514 and the bounding box dimensions of the respective bounding boxes 202, 204, 208, 212, and 214 are searched for.

[0115] Assembling the scene 300 includes, for example, placing the retrieved 3D models of the objects 502, 504, 508, 512, and 514 in the scene 300 at their respective box centers and with their corresponding box orientations.

[0116] The method includes step 1208 .

[0117] Step 1208 includes determining, dependent on the assembled scene, a 3D Gaussian splatting representation of the assembled scene.

[0118] For example, the exemplary 3D Gaussian splatting representation 400 of the exemplary constructed scene 300 is determined depending on the exemplary constructed scene 300 .

[0119] The method includes step 1210 .

[0120] Step 1210 involves rendering the digital image from the 3D Gaussian splatting representation.

[0121] For example, the exemplary first digital image 500 is rendered from the exemplary 3D Gaussian splitter representation 400 .

[0122] Rendering a digital image from a 3D Gaussian splatting representation may include providing a viewpoint and rendering a view of the scene from that viewpoint.

[0123] For example, rendering the first exemplary digital image 500 from the exemplary 3D Gaussian Splatting representation 400 includes providing a first viewpoint and rendering a first view of the scene 300 from the first viewpoint.

[0124] The first exemplary digital image 500 includes a first view of the scene 300 .

[0125] The method includes step 1212 .

[0126] Step 1212 includes determining a composite digital image using stable diffusion dependent on the digital image and the style description.

[0127] Determining the composite digital image may include, for example, determining pixel values ​​of pixels in the composite digital image representing the at least one object in dependence on pixel values ​​of pixels in the digital image representing the at least one object using stable diffusion.

[0128] Determining the composite digital image may include, for example, setting pixel values ​​of pixels of the composite digital image that do not represent the at least one object to values ​​of pixels of the digital image that do not represent the at least one object.

[0129] For example, the exemplary first composite digital image 800 is determined using stable diffusion based on the exemplary first digital image 500 and the exemplary style description.

[0130] Exemplary first digital image 500 is an unedited conditioning image for stable diffusion to determine exemplary first composite image 800. That is, first view of scene 300 is an initial first view of scene 1100.

[0131] Determining the exemplary first composite digital image 800 may include, for example, determining pixel values ​​of pixels in the exemplary first composite digital image 800 representing the objects 502, 504, 508, 512, and 514 depending on pixel values ​​of pixels in the exemplary first digital image 500 representing the objects 502, 504, 508, 512, and 514 using stable diffusion.

[0132] Determining the exemplary first composite digital image 800 includes, for example, setting pixel values ​​of pixels of the exemplary first composite digital image 800 that do not represent one of the objects 502, 504, 508, 512, and 514 to the values ​​of corresponding pixels of the exemplary first digital image 500 that do not represent one of the objects 502, 504, 508, 512, and 514.

[0133] The method may include training a 3D Gaussian splatting representation and / or a stable diffusion representation relying on a loss that depends on values ​​of pixels representing at least one object in the digital image and in the composite digital image, respectively.

[0134] The method may include training the 3D Gaussian splatting representation 400 and / or stable diffusion depending on losses that depend on the values ​​of pixels representing the objects 502, 504, 508, 512, and 514 in the exemplary digital image and the exemplary composite digital image, respectively.

[0135] An example of stable diffusion is the unedited conditioning image.

number

number

number

number

number

[0136] Stable diffusion is trained by editing training images from the dataset to determine new images for the dataset in dataset updates, which are performed, for example, every 2500 training iterations.

[0137] The method may include determining a segmentation mask M that indicates whether a pixel represents an object. The segmentation mask M may, for example, be a binary mask.

[0138] The method may include determining pixel values ​​for pixels representing the object according to the segmentation mask M using stable diffusion.

[0139] For example, a binary mask

number

number

number

number

number

number

number

[0140] That is, the method only maintains edits in pixels of the target object set.

number

number

[0141] In this way,

number

[0142] In training, the edited image,mask

number

[0143] The method is described using a first viewpoint as an example. The method may include providing three different viewpoints: a first viewpoint, a second viewpoint, and a third viewpoint. For the three viewpoints, the method may include determining a digital image showing the scene 300 from each viewpoint and determining a composite digital image 800, 900, 1000 showing the scene 1100 from each viewpoint.

[0144] Composite digital images 800, 900, and 1000 are examples of composite digital image 102. Scene 1100 is an example of a three-dimensional scene 104.

[0145] The steps of the method may be performed iteratively to determine different synthetic digital images for a dataset of synthetic digital images, which may be used to train and / or test a machine learning system.

[0146] By repeating step 1210 relying on the same 3D Gaussian splatting representation with stable diffusion, a variety of different composite digital images can be generated.

[0147] By repeating the steps of the method with different at least one first prompt, different composite digital images can be generated based on different three-dimensional layout descriptions and / or style descriptions.

Claims

1. 1. A computer-implemented method for generating synthetic digital images (800, 900, 1000) of a three-dimensional scene (1100), particularly for a dataset for training and / or testing a machine learning system, comprising: The method comprises: providing (1202) at least one text prompt (106), the at least one text prompt (106) including a description of a three-dimensional layout (200) of a scene (300, 1100), the at least one text prompt (106) including a description of the scene (1100); generating (1204) a layout (200) dependent on the layout description; Assembling (1206) the scene (300) depending on the layout (200); determining (1208) a 3D Gaussian splatting representation (400) of the assembled scene (300) depending on the assembled scene (300); Rendering (1210) a digital image (500, 600, 700) from said 3D Gaussian splatting representation (400); determining (1212) a composite digital image (800, 900, 1000) dependent on said digital image (500, 600, 700) and a style description using stable diffusion; A method comprising:

2. the at least one text prompt (106) includes a description of the location of at least one object (502, 504, 508, 512, 514) within the scene (300, 1100) in a two-dimensional perspective; the at least one text prompt (106) includes a description of an orientation of the at least one object (502, 504, 508, 512, 514) within the scene (300, 1100) in a two-dimensional perspective; generating (1204) the layout (200) includes creating a three-dimensional bounding box (202, 204, 208, 212, 214) for the at least one object (502, 504, 508, 512, 514) in the scene (300, 1100) depending on the position description and the orientation description; The method of claim 1.

3. Creating the bounding boxes (202, 204, 208, 212, 214) includes: determining a box center of said bounding box (202, 204, 208, 212, 214) depending on said position description; determining a box orientation of said bounding boxes (202, 204, 208, 212, 214) depending on said orientation description; The method of claim 2 , comprising:

4. Determining the position description and the orientation description comprises: providing a canonical coordinate system representing the scene in a two-dimensional perspective; Dividing the canonical coordinate system into a grid containing a plurality of rectangular patches; selecting one of the patches; generating a textual description of the position and orientation depending on the position of the patch within the grid; The method of claim 3, comprising:

5. Assembling (1206) the scene (300) depending on the layout (200) includes: retrieving a three-dimensional model of said at least one object (502, 504, 508, 512, 514) from a database containing three-dimensional models of a plurality of objects, in particular retrieving a three-dimensional model having a minimum Euclidean distance between dimensions of the three-dimensional model for said at least one object (502, 504, 508, 512, 514) and dimensions of the bounding box of said bounding box (202, 204, 208, 212, 214); placing the retrieved three-dimensional model of the at least one object (502, 504, 508, 512, 514) in the scene (300) at the box center and with the box orientation; 5. The method of claim 2, comprising:

6. Determining the composite digital image (800, 900, 1100) comprises: determining pixel values ​​of pixels in the composite digital image (800, 900, 1100) representing the at least one object (502, 504, 508, 512, 514) dependent on pixel values ​​of pixels in the digital image representing the at least one object (502, 504, 508, 512, 514) using the stable diffusion; setting pixel values ​​of pixels of the composite digital image that are not representative of the at least one object (502, 504, 508, 512, 514) to values ​​of pixels of the digital image that are not representative of the at least one object (502, 504, 508, 512, 514); 6. The method of claim 2, comprising:

7. The method comprises: training the 3D Gaussian splatting representation (400) and / or the stable diffusion in dependence on a loss that depends on the values ​​of pixels representing the at least one object (502, 504, 508, 512, 514); 7. The method according to any one of claims 2 to 6.

8. The method comprises: determining a binary mask indicating whether a pixel represents said at least one object (502, 504, 508, 512, 514); determining pixel values ​​for pixels representing the at least one object according to the binary mask using the stable diffusion; 8. The method of claim 6 or 7, comprising:

9. The method comprises: generating another synthetic digital image (800, 900, 1000) for said data set using said stable diffusion and relying on the same three-dimensional Gaussian splatting representation (400); 9. The method according to any one of claims 1 to 8.

10. The method comprises: providing (1202) at least one other text prompt (106); determining (1204, 1206, 1208) another three-dimensional Gaussian splatting representation (400) dependent on a description of the three-dimensional layout of the scene (300, 1100) in the at least one other text prompt (106); determining (1210) another composite digital image (800, 900, 1000) for the data set depending on the another Gaussian splatting representation (400) and the style description in the another at least one text prompt (106); 10. The method of claim 1, comprising:

11. Rendering (1210) the digital image (800, 900, 1000) from the 3D Gaussian splatting representation (400) comprises: Providing perspective and Rendering a view of the scene (300, 1100) from that viewpoint; 11. The method of claim 1, comprising:

12. The method comprises: Providing three different perspectives, determining, for the three viewpoints, the composite digital images (800, 900, 1000) showing the scene (300, 1100) from each viewpoint; The method of claim 11 , comprising:

13. 1. An apparatus (100) for generating synthetic digital images (800, 900, 1000) of a three-dimensional scene (1100), particularly for a dataset for training and / or testing a machine learning system, comprising: The device (100) comprises: at least one processor (110); at least one memory (112) for storing instructions; Equipped with The at least one processor (110) is configured to execute instructions that, when executed by the at least one processor (110), cause the device (100) to perform the method of any one of claims 1 to 12.

1. An apparatus (100) comprising:

14. 1. A computer program for generating synthetic digital images of three-dimensional scenes, particularly for datasets for training and / or testing machine learning systems, comprising: The computer program comprises computer-executable instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 12. A computer program characterized by:

Images