Generating 2d image of 3D scene

Abstract

To provide a machine-learning method and system for machine-learning a function configured for generating a 2D image of a 3D scene.SOLUTION: A function comprises a scene encoder and a generative image model. The scene encoder receives as input a layout of a 3D scene and a viewpoint and outputs a scene encoding tensor. The generative image model receives as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. A machine-learning method comprises obtaining a dataset comprising 2D images and corresponding layouts and viewpoints of 3D scenes. The machine-learning method comprises training the function based on the obtained dataset. The machine-learning method forms an improved solution for generating a 2D image of a 3D scene.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for machine learning functions configured to generate 2D images of 3D scenes. [Background technology]

[0002] Numerous systems and programs are available on the market for designing, engineering, and manufacturing objects. CAD is an acronym for Computer-Aided Design, e.g., software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, e.g., software solutions for simulating the physical behavior of future products. CAM is an acronym for Computer-Aided Manufacturing, e.g., software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays a key role in the efficiency of the technology. These technologies can be embedded within product lifecycle management (PLM) systems. PLM, in conjunction with the concept of the extended enterprise, refers to a business strategy that helps companies share product data, apply common processes, and leverage corporate knowledge in product development from conception to the end of its life. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA, 3DVIA, and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to both the Engineering Hub and the Manufacturing Hub. All together, the system provides an open object model that links products, processes, and resources, enabling dynamic knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this context, applications for 3D scene creation are being developed. These applications generally propose to create, manipulate, and present 3D scenes, especially (but not exclusively) for touch-sensitive devices (e.g., smartphones or tablets). One task of these applications is to generate realistic 2D images of the 3D scene.

[0004] Solutions for generating 2D images of 3D scenes have been developed in recent years, for example, using deep learning generative models. However, these solutions do not fully consider the entire 3D environment of the scene being imaged. In particular, these solutions cannot utilize knowledge about the 3D structure and relationships of objects within the scene or environment. In particular, they do not consider, for example, perspective, occlusion, or lighting factors, and therefore cannot create accurate and natural immersive content. Summary of the Invention [Problem to be solved by the invention]

[0005] In this context, there is still a need for improved solutions for generating 2D images of 3D scenes. [Means for solving the problem]

[0006]

[0009] Accordingly, there is provided a computer-implemented method (hereinafter referred to as the machine learning method) for machine learning a function configured to generate 2D images of a 3D scene. The function comprises a scene encoder and an image generation model. The scene encoder receives as input a layout and a viewpoint of the 3D scene and outputs a scene encoding tensor. The image generation model receives as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. The machine learning method comprises obtaining a dataset comprising the 2D images and the corresponding layout and viewpoint of the 3D scene. The machine learning method comprises training the function based on the obtained dataset.

[0007] The machine learning method may comprise one or more of the following: Each 3D scene layout includes a set of bounding boxes that represent objects in the 3D scene, as well as the bounds of the 3D scene. The scene encoder includes a layout encoder configured to encode a set of bounding boxes, the layout encoder receiving as input, for each bounding box in the set, parameters representing the position, size, and orientation within the 3D scene of the object represented by the bounding box, and optionally the class of the object represented by the bounding box. The scene encoder further includes a floor encoder configured to encode a boundary of the 3D scene. The scene encoder further includes a camera encoder configured to encode the viewpoint. For each given 2D image of a given 3D scene in the dataset, the size and position of the object represented by the bounding box within the layout of the given 3D scene are defined in a coordinate system based on the position and orientation of the camera from which the given 2D image was taken, each viewpoint comprising the field of view and pitch of the camera. The scene encoder further includes a transformer encoder that receives as input the set of bounding boxes encoded by the layout encoder, the viewpoints encoded by the camera encoder, and the concatenation of the 3D scene boundary encoded by the floor encoder. The transformer encoder outputs a scene encoding tensor. -The image generation model is a diffusion model. The diffusion model has an architecture that includes a denoiser with blocks, at least one of which is enhanced by cross-attention using a scene encoding tensor; and / or -The diffusion model is configured to operate in the latent space: the diffusion model is trained to denoise the compressed latent representation of the 2D images in the dataset.

[0008] Also provided is a method of using the machine-learned function according to the machine learning method (hereinafter referred to as the "method of use"), which comprises obtaining a layout of a 3D scene, and applying the function to the layout of the 3D scene to thereby generate a 2D image of the 3D scene.

[0009] The method of use may comprise one or more of the following: The image generation model is a diffusion model; and / or - Applying a function applying a scene encoder to the obtained layout, thereby outputting a scene encoding tensor; and It involves using a diffusion model conditioned on the output scene encoding tensor to generate a 2D image of the 3D scene.

[0010] Additionally, a computer program comprising instructions for carrying out the machine learning method and / or the method of use is provided.

[0011] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0012] Additionally, a system is provided that includes a processor coupled to a memory having a computer program recorded thereon, and the system may further include a graphical user interface coupled to the processor.

[0013] Furthermore, a device is provided comprising a data storage medium having a computer program recorded thereon.

[0014] The device may form or function as a non-transitory computer-readable medium, such as in a software-as-a-service (SaaS) or other server or cloud-based platform. Alternatively, the device may comprise a processor coupled to a data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a graphical user interface coupled to the processor. [Brief explanation of the drawings]

[0015] Non-limiting examples will now be described with reference to the accompanying drawings.

[0016] [Figure 1] 1 shows a flowchart of an example machine learning method and method of use. [Figure 2] 1 shows an example of a scene encoder. [Figure 3] An example of camera coordinates is shown below. [Figure 4] Illustrates examples of 3D scene layouts and viewpoints. [Figure 5] 1 shows the resulting 2D image produced by the trained function. [Figure 6] Illustrates examples of 3D scene layouts and viewpoints. [Figure 7] 1 shows the resulting 2D image produced by the trained function. [Figure 8] Illustrates examples of 3D scene layouts and viewpoints. [Figure 9] 1 shows the resulting 2D image produced by the trained function. [Figure 10] 1 shows the resulting 2D image produced by the trained function. [Figure 11] Illustrates examples of 3D scene layouts and viewpoints. [Figure 12] 1 shows the resulting 2D image produced by the trained function. [Figure 13] 10 shows the results of a quantitative evaluation performed to evaluate the trained functions. [Figure 14] An example of a system is shown. DETAILED DESCRIPTION OF THE INVENTION

[0017] A computer-implemented method (hereinafter referred to as the machine learning method) for machine learning a function configured to generate 2D images of a 3D scene is proposed. The function comprises a scene encoder and an image generation model. The scene encoder receives as input a layout and a viewpoint of the 3D scene and outputs a scene encoding tensor. The image generation model receives as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. The machine learning method comprises obtaining a dataset comprising the 2D images and the corresponding layout and viewpoint of the 3D scene. The machine learning method comprises training the function based on the obtained dataset.

[0018] Such machine learning methods form an improved solution for generating 2D images of 3D scenes.

[0019] In particular, machine learning techniques can be used to train a function to automatically and efficiently generate 2D images of a 3D scene. Specifically, the function is trained to generate diverse, realistic 2D images from a high-level, abstract proxy representation of the 3D scene (which is therefore easy to define). Indeed, once trained, the function can generate 2D images of a 3D scene from only the 3D scene's layout and viewpoint. From these two inputs, the trained function can generate 2D images that are particularly useful and interesting for describing objects within a 3D scene. Notably, providing these two inputs to the trained function is much easier for a user than providing accurate object models for each object in the 3D scene and then using traditional rendering methods. Thus, using the trained function, a user can easily and quickly generate 2D images of a 3D scene they are building by simply defining its layout and providing viewpoints for these images.

[0020] In particular, the function is trained to generate particularly realistic and relevant 2D images of a 3D scene. In fact, the function comprises a scene encoder that allows it to take into account the layout in the generated 2D image. Thus, the function can generate 2D images that take into account the perspective of the 3D scene and its lighting, as well as occlusions between objects (the scene encoder ensures that this information is taken into account during the generation of the 2D image). In other words, the 2D images generated by the trained function are 3D-enabled, that is, the 3D environment of the 3D scene is taken into account. In particular, the method can take into account off-screen objects in the 3D scene when generating the 2D image, that is, objects that are not within the camera's field of view will affect the generated image (e.g., light from a window).

[0021] Furthermore, the function is trained to generate a variety of 2D images. Indeed, for a given 3D layout, the function can generate a variety of 2D images in terms of object style, color, etc., while still taking the layout into account. Thus, users can get multiple inspirations from a single input abstract layout.

[0022] Furthermore, the proposed machine learning method is trained end-to-end for the task in a single training phase: it does not rely on large-scale pre-trained image generation models, pre-trained depth estimators, or other external modules.

[0023] The machine learning method and / or method of use is computer-implemented. This means that the steps (or substantially all steps) of the machine learning method and / or method of use are performed by at least one computer or any similar system. Thus, the steps of the machine learning method and / or method of use are performed by a computer, possibly fully automatically or semi-automatically. In examples, triggering of at least some of the steps of the machine learning method and / or method of use may be performed through user-computer interaction. The level of user-computer interaction required may vary depending on the expected level of automation, balanced against the need to realize the user's wishes. In examples, this level may be user-defined and / or pre-defined.

[0024] For example, the method of use may comprise a step of determining the layout of the 3D scene, e.g., by user interaction (e.g., performed by a user who is currently designing the 3D scene, for example), before applying the function to the layout of the 3D scene. Determining the layout may comprise determining the boundaries of the 3D scene and a set of bounding boxes representing objects within the 3D scene.

[0025] Next, determining the set of bounding boxes will be described. Determining the set of bounding boxes may include, for each bounding box in the set, determining a size of the bounding box and positioning the sized bounding box within the determined boundaries of the 3D scene. The sizing and positioning steps may be performed manually by a user. For example, the sizing step may include the user inputting a width, depth, and height (e.g., through user interaction using a keyboard). The positioning step may include the user inputting coordinates of a point on the bounding box (e.g., a corner or its center) and its orientation, or the user moving the bounding box to its position in the 3D scene (e.g., the bounding box may be displayed on a screen and moved by the user using a mouse). For one or more bounding boxes (e.g., all bounding boxes), the sizing step may be performed semi-automatically. For example, the sizing step may include the user selecting a category of object represented by the bounding box and automatically suggesting a size (e.g., width, depth, and height) of the bounding box representing the object (e.g., from a database storing default sizes for various categories of objects). The suggested size may be accepted by the user or may be adjusted by the user (e.g., manually as described above). For example, if a user wants to add a couch to their 3D layout, the method may suggest a default bounding box that matches the object category entered by the user, while allowing the user to change these dimensional values.

[0026] Next, determining the boundary of the 3D scene will be described. Determining the boundary of the 3D scene may comprise determining a respective set of points that represent the boundary. For example, determining the boundary may comprise determining some points of the set (e.g., representing corners of the 3D scene) and then sampling other points on the boundary between these points that represent the corners of the 3D scene (i.e., along the walls of the 3D scene).

[0027] The method of use may comprise determining a viewpoint before applying the function to the layout of the 3D scene. For example, the viewpoint determination may be performed by a user, e.g., by inputting viewpoint coordinates and / or direction, or by selecting this information on a screen displaying the 3D scene. Alternatively, the viewpoint determination may be performed automatically, e.g., by another function that predicts one or more relevant viewpoints for the 3D scene given its layout.

[0028] A typical example of a computer implementation of the machine learning method and / or method of use is to execute the machine learning method and / or method of use using a system adapted for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), where the memory has recorded thereon a computer program comprising instructions for executing the machine learning method and / or method of use. The memory may also store a database. The memory is any hardware adapted for such storage, possibly comprising several physically distinct parts (e.g., one for the program and possibly one for the database).

[0029] The data set considered by the machine learning method to train the function may be stored in a database. A "database" refers to any collection of data (i.e., information) organized for search and retrieval (e.g., a relational database based on a predefined structured language, e.g., SQL). When stored in memory, a database allows for rapid search and retrieval by a computer. In practice, databases are structured to facilitate the storage, retrieval, modification, and deletion of data in conjunction with various data processing operations. A database may consist of a file or a set of files that can be divided into records, each consisting of one or more fields. A field is the basic unit of data storage. Users may retrieve data primarily through queries. Keywords and sorting commands allow users to quickly search, sort, group, and select fields within many records to retrieve or create reports on a particular collection of data according to the rules of the database management system being used.

[0030] Methods generally manipulate modeled (3D) objects. A modeled object is any object defined by data stored, for example, in a database. By extension, the term "modeled object" refers to the data itself. Depending on the type of system, a modeled object may be defined by various types of data. Indeed, a system may be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these different systems, a modeled object is defined by corresponding data. Thus, one may speak of a CAD object, a PLM object, a PDM object, a CAE object, a CAM object, CAD data, PLM data, PDM data, CAM data, and CAE data. However, these systems are not exclusive of one another, as a modeled object may be defined by corresponding data in any combination of these systems. Thus, a system may be both a CAD system and a PLM system.

[0031] A CAD system further refers to any system at least adapted for designing a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, data defining the modeled object comprises data enabling the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, or in certain cases faces or surfaces. Lines, edges, or surfaces may be represented in various ways, for example, as non-uniform rational B-splines (NURBS). Specifically, a CAD file contains specifications from which geometry may be generated, thereby enabling the generation of a representation. The specifications of a modeled object may be stored in a single CAD file or in multiple CAD files.

[0032] In an example, each 3D scene may represent a real room, e.g., an indoor real room. For example, the rooms represented by the 3D scenes may be rooms in a residence (e.g., a house or apartment), such as a kitchen, bathroom, bedroom, living room, garage, laundry room, attic, office (e.g., private or shared), conference room, children's room, nursery, hallway, dining room, and / or study (this list may include other types of rooms). Alternatively, the rooms represented by the 3D scenes may be other indoor rooms, such as in a factory, museum, and / or theater. Alternatively, the rooms represented by the 3D scenes may be outdoor scenes, such as a garden, terrace, amusement park, etc.

[0033] Each object in each 3D scene may represent the geometry of a real object placed in the real room that the 3D scene represents. This real object may be manufactured in the real world after completion of its virtual design (e.g., using a CAD software solution or system). The 3D scene may include one or more furniture objects, such as one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more sideboards, one or more nightstands, one or more desks, and / or one or more wardrobes. Alternatively or additionally, the 3D scene may include one or more decorative objects, such as one or more accessories, one or more plants, one or more books, one or more frames, one or more kitchen accessories, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors, and / or one or more electronic appliances (e.g., refrigerators, freezers, and / or washing machines).

[0034] The method of use may be included in a real-life room design (i.e., effective placement) process, which may comprise using a 2D image generated after performing the method of use to illustrate the room to be placed. For example, the illustration may be intended for a user, such as the owner of the house in which the room is located. The generated 2D image may be used by the user to decide whether to acquire one or more objects in a 3D scene, and may assist the user in their selection by displaying one or more objects in the room. During the real-life room design process, the method of use may be repeated to determine several 2D images of the room. Iterations of the method of use may be used to illustrate a complete virtual interior of the room (i.e., including several 2D images of the room) and / or to acquire 2D images of the 3D scene with different styles and / or object appearances.

[0035] Alternatively or additionally, real-life room design may comprise using the generated 2D image(s) to perform a similarity-based search for 3D objects from a catalog to be placed at the locations of the bounding boxes. The trained function enables this due to the realism of the generated 2D images. For example, real-life room design may comprise a user defining the layout of a given 3D scene by placing 3D bounding boxes. Then, real-life room design may comprise using the trained function (e.g., by repeating the aforementioned method of use) to generate several 2D images of the given 3D scene. Real-life room design may then comprise the user selecting one of the generated 2D images. For example, a user may particularly appreciate the appearance and / or style of one of the generated 2D images and wish to provide the most similar 3D object from the catalog for the given 3D scene (i.e., replace the bounding box with the actual 3D object). In that case, designing a real-life room may comprise, for each object in the generated 2D image, deriving a position of each object in the generated 2D image from the defined layout, cropping the object in the image, computing an image embedding of the object (e.g., using a pre-trained language-image model such as described in the paper by Radford et al., "Learning Transferable Vision Models from Natural Language Supervision," International Conference on Machine Learning, PMLR2021, (hereinafter referred to as CLIP)), and comparing this image embedding of the object with images from the catalog to obtain the most similar object. Designing a real-life room may comprise replacing a bounding box in the 3D scene with the most similar object from the catalog obtained for each object.

[0036] Alternatively or additionally (e.g., prior to illustration), the real-life room design process may comprise adding one or more new objects to a 3D scene representing a room (which may initially be, for example, partially empty) by modifying the layout of the 3D scene. Adding may comprise repeating, for each new object, the steps of sizing and positioning a bounding box representing the new object as described above. Thus, the generated 2D image may include the new objects added to the 3D scene by modifying the layout. The real-life room design process may create richer and more enjoyable environments (for the generation of virtual environments such as animations, advertisements, and / or simulations). The real-life room design process may be used for the generation of virtual environments. The real-life room design process may be included in a general process that may comprise repeating the real-life room design process for several 3D scenes, thereby showing several 3D scenes with objects.

[0037] Alternatively or additionally, the real-life room design process may comprise physically arranging a (i.e., real) room to match the 3D scene shown in the 2D image(s) from which the design will be generated after the method is performed. For example, a room (without the objects represented by the input 3D scene) may already exist in the real world, and the real-life room design process may comprise positioning a real object (i.e., an object represented by one of the bounding boxes of the layout) represented by one of the objects of the 3D scene within the existing room (i.e., in the real world). The bounding box of this object may have been added to the layout of the 3D scene by a user. The real object may be positioned according to the location of its bounding box within the 3D scene. The real-life room design process may repeat this process to place various real objects within the existing room. Alternatively, the room may not yet exist when the method is performed. In that case, the real-life room design process may comprise building a room (i.e., adding real objects to the room) according to the generated 2D image of the 3D scene (i.e., by placing the real objects in the positions of bounding boxes that represent them in the layout of the 3D scene), and since the method improves the positioning of the 3D objects in the 3D scene, the method also improves the building of the room corresponding to the 3D scene, thus increasing the productivity of the real-life room design process.

[0038] Next, acquisition of the data set will be described.

[0039] The dataset comprises multiple 2D images of a 3D scene (e.g., more than 50,000 2D images of the same type of room). For each 2D image, the dataset also includes the 3D scene imaged (e.g., partially) in the 2D image (e.g., only the layout of the 3D scene) and the viewpoint from which the 2D image was taken (e.g., coordinates of the viewpoint within the 3D scene). Each 2D image of the dataset may be taken for a respective (i.e., different) 3D scene. Alternatively, the dataset may comprise 2D images of the same 3D scene taken, for example, from different viewpoints and / or with different lighting. The dataset may also comprise, for each 2D image, information indicating the layout of the 3D scene it images and its viewpoint (e.g., a table with multiple rows, each row including a 2D image reference, a reference to the layout of the corresponding 3D scene, and coordinates of the viewpoint of the 2D image).

[0040] The 2D images of the dataset may be realistic 2D images of the 3D scene generated (e.g., by a designer) prior to execution of the method. These 2D images may include, for example, perspective, occlusion, and / or lighting factors. To achieve such rendering, the 2D images of the 3D scene of the dataset may have been manually reworked by the designer(s) (at least in part, for example, in places where rendering is difficult due to perspective, occlusion, and / or lighting factors).

[0041] The dataset may be stored in a database. Obtaining the dataset may comprise obtaining the dataset from the database. Obtaining the dataset may then comprise storing the obtained dataset in a memory. After recording, the machine learning method may perform training of the function based on the recorded dataset. Alternatively, obtaining the dataset may comprise providing access to the dataset in a database. In that case, the machine learning method may use this access to perform training of the function.

[0042] The rooms represented by the 3D scenes of the dataset may or may potentially exist in the real world (already at the time of acquisition of the dataset or in the future). For example, the rooms may be actual, real rooms (in terms of layout) in the real world, and objects may be placed within these real rooms as specified in the layout of the 3D scenes provided by the dataset. The 3D scenes may represent rooms designed (e.g., by an interior designer) and implemented in the real world (i.e., the multiple 3D scenes correspond to virtually designed rooms that have been or may be recreated in people's homes). In an example, each room represented in the dataset is of the same type. For example, all rooms represented in the dataset and the 3D scenes may be kitchens, bathrooms, bedrooms, living rooms, garages, laundry rooms, attics, offices (e.g., private or shared), conference rooms, children's rooms, nurseries, hallways, dining rooms, or studies (this list may include other types of rooms). In that case, the layout obtained during execution of the method of use may be a layout of a 3D scene that is also of the same type as that of the dataset. This may result in more realistic 2D images and improve the stability of the method of use. Alternatively, the dataset may contain different types of rooms, in which case the output domain of the image generation model may be larger, the number of rooms represented in the dataset may be greater, and training the image generation model may also take more time.

[0043] In an example, the layout of each 3D scene may include a set of bounding boxes representing objects in the 3D scene. Each bounding box may be rectangular in space and may encapsulate the outer envelope of the object it represents. The layout may comprise, for each bounding box, parameters representing the position, size, and orientation in the 3D scene of the object represented by the bounding box. For example, the layout may comprise, for each bounding box, parameters representing the position of the bounding box (e.g., coordinates of the corners or center of the bounding box), parameters representing the size of the bounding box (e.g., width, depth, and height of the bounding box), and parameters representing the orientation of the bounding box (e.g., rotation relative to each axis of a global reference frame). Optionally, the layout may comprise, for each bounding box, parameters representing the class of the object represented by the bounding box. The object classes may be predetermined and may each correspond to the type of object they represent. The object classes may be the decorative object and functional object types described above.

[0044] Each 3D scene layout may also include a boundary of the 3D scene. For example, the boundary of the 3D scene may be represented by a respective set of points, e.g., corresponding to corners of the 3D scene or sampled along a wall of the 3D scene. The 3D scene layout may comprise coordinates of the respective sets of points.

[0045] Training the function may comprise training a scene encoder and an image generation model to generate 2D images of the dataset when they receive as input corresponding layouts and viewpoints included in the dataset (e.g., in a supervised manner). For example, the scene encoder and the image generation model may each have respective parameters (e.g., weights), and the supervised training may consist of determining values for these respective parameters so as to optimally reproduce the 2D images of the dataset when receiving as input corresponding layouts and viewpoints included in the dataset. Supervised training of the function may comprise training the scene encoder and the image generation model together (i.e., it may determine their respective parameters simultaneously or during the same process).

[0046] In an example, the scene encoder may include a layout encoder configured to encode a set of bounding boxes. The layout encoder may receive as input, for each bounding box in the set (e.g., whether it is visible from the viewpoint), parameters representing the position, size, and orientation in the 3D scene of the object represented by the bounding box. Optionally, the layout encoder may further receive as input, for each bounding box, parameters representing the class of the object represented by the bounding box. These parameters may be included in the layout of the 3D scene as described above. The layout encoder may estimate these parameters from the layout received as input by the function. The layout encoder may be configured to output a vector (hereinafter referred to as a "layout vector") embedding the parameters.

[0047] For example, the layout encoder may include a positional encoding module that receives bounding box parameters as input and outputs a layout vector. The positional encoding module may be configured to deterministically increase the dimensionality of the scalar values of the parameters received as input. For example, the positional encoding module may be configured to output, for each bounding box, a position vector representing the position and size of the bounding box and a direction vector representing the orientation of the bounding box. Optionally, the layout encoder may further include a first multilayer perceptron configured to increase the dimensionality of the direction vector output by the positional encoding module. If the layout encoder also receives object classes as input, the layout encoder may further include a second multilayer perceptron that receives, for each bounding box, parameters representing the class (or category) of the object represented by the bounding box as input and outputs a category vector. The layout encoder may also include a concatenation layer configured to concatenate the vector output by the positional encoding module, the first multilayer perceptron layout, and / or the second multilayer perceptron layout and output the layout vector.

[0048] In an example, the scene encoder may further include a floor encoder configured to encode a boundary of the 3D scene. As described above, the boundary of the 3D scene may be represented by a respective set of points, and the floor encoder may receive the respective set of points as input and output a floor vector. For example, the floor encoder may include a PointNet model (e.g., as described in the paper "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" by Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, CVPR 2017) configured to encode the respective set of points, and optionally a multilayer perceptron configured to receive the output of the PointNet model as input.

[0049] In an example, the scene encoder may further include a camera encoder configured to encode a viewpoint. For example, the viewpoint may comprise parameters of a camera from which the 2D image is generated (e.g., camera position, field of view, and pitch). The scene encoder may be configured to receive these camera parameters as input and output a camera vector. For example, the scene encoder may comprise a position encoder configured to receive the camera parameters as input, and optionally a multi-layer perceptron.

[0050] In the example, for each given 2D image of a given 3D scene in the dataset, the size and position of an object represented by a bounding box in the layout of the given 3D scene are defined in a coordinate system based on the position and orientation of the camera from which the given 2D image was taken. Therefore, the camera position is already encoded in the layout vector output by the layout encoder. Therefore, the viewpoint may comprise only two scalar values representing the field of view and pitch of the camera, respectively. This reduces the number of training parameters, improving robustness and thus facilitating convergence of the function.

[0051] In an example, the scene encoder may further include a transformer encoder. The transformer encoder receives as input a concatenation of a set of bounding boxes encoded by the layout encoder (i.e., a layout vector), a viewpoint encoded by the camera encoder (i.e., a camera vector), and a boundary of the 3D scene encoded by the floor encoder (i.e., a floor vector). The transformer encoder outputs a scene encoding tensor. For example, the transformer encoder may include a transformer model configured to receive a vector as input and form a sequence of tokens represented as a tensor (the scene encoding tensor). The vector received as input by the transformer model may be a concatenation of all the aforementioned vectors (i.e., the layout vector, the camera vector, and the floor vector), optionally padded (or supplemented) with one or more “zero” tokens, thereby constituting a vector of a fixed (e.g., predetermined) size.

[0052] The image generation model may be any image generation model capable of generating 2D images conditioned on the output scene encoding tensor. The image generation model may be a type of deep neural network trained on a large image dataset to learn the underlying distribution of training images. By sampling from the learned distribution, such a model may be configured to generate new images with features from images in the training dataset. Examples of image generation models for generating 2D images conditioned on the output scene encoding tensor include generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models.

[0053] In an example, the image generation model may be a diffusion model. The diffusion model may be configured to generate an output 2D image by iteratively removing noise from an initial noisy image based on a scene encoding tensor output by a scene encoder. Examples of diffusion models include a cascade model or a latent diffusion model. A cascade model is a model that includes several diffusion modules (e.g., a module that outputs an image conditioned on the scene encoding tensor) and a super-resolution model that upscales the image to a higher resolution. During inference, the diffusion model may generate an output 2D image by iteratively removing noise from the initial noisy image. Each iteration of denoising may comprise determining a new version of the initial noisy image that is less noisy than a previous version of the initial noisy image determined during a previous iteration. The determination of the new version may be based on a prediction of the noise of the previous version.

[0054] Next, the training of the diffusion model will be described. The training of the diffusion model may be based on generated noisy versions of the 2D images included in the dataset. For example, the machine learning method may comprise generating noisy versions of the 2D images of the dataset (by adding noise to these 2D images), and the training of the diffusion model may be based on the generated noisy versions of the 2D images. The diffusion model may be trained to remove the noise added to the 2D images of the dataset, taking into account the scene encoding tensor output by the scene encoder. In that case, the training of the diffusion model and the scene encoder may take into account a training loss that penalizes the distance between the noise predicted in the generated noisy version and the actual noise of the generated noisy version.

[0055] In an example, the diffusion model may have an architecture that includes a denoiser. The denoiser may comprise several blocks configured to create the resulting 2D image. At least one of these blocks may be augmented with cross-attention using a scene encoding tensor. The cross-attention mechanism may be an attention mechanism applied between elements from different sequences. The cross-attention may be applied between the representations returned by the transformer encoder for various tokens and the visual features computed within the denoiser. This improves the function's ability to learn the visual and spatial dependencies / relationships that exist between the scene features (encoded by the transformer encoder) and their visual representations in the image (generated by the denoiser). Therefore, this mechanism is particularly suitable for generating 2D images of 3D scenes, where the location / spatiality of objects in the 2D image is paramount.

[0056] In an example, the diffusion model may be configured to operate in a latent space (i.e., it may be a latent diffusion model). In that case, during training, the diffusion model may be trained to denoise compressed latent representations of 2D images of the dataset. In that case, training the function may comprise compressing the 2D images of the dataset in a latent space (e.g., of smaller dimensionality), thereby obtaining compressed latent representations. Training may be performed based on these compressed latent representations (rather than the 2D images directly). During inference, the diffusion model may receive as input, instead of an initial noisy tensor, a compressed initial noisy tensor of the same dimension as the compressed latent representation. The diffusion model may iteratively remove noise from this compressed initial noisy tensor given a scene encoding tensor output by a scene encoder. Decompression may then be applied to the result to obtain the resulting 2D image. An example implementation of such compression / decompression includes a variational autoencoder (VAE).

[0057] Next, application of the function is described. Application of the function may comprise first forming an input for the diffusion model. If the diffusion model does not operate in latent space, this step may comprise, for example, sampling an initial noisy tensor having the shape of the 2D image to be generated. This initial noisy tensor may be received as input by the diffusion model. If the diffusion model operates in latent space, this step may comprise sampling a compressed initial noisy tensor (i.e., having the same dimensions as the compressed latent representation). This compressed initial noisy tensor may be received as input by the diffusion model.

[0058] Next, applying the function may comprise applying a scene encoder to the obtained layout, thereby outputting a scene encoding tensor. Thereafter, applying the function may comprise using a diffusion model conditioned on the output scene encoding tensor to generate a 2D image of the 3D scene. The diffusion model may iteratively remove noise from the sampled initial noisy tensor when not operating in latent space or otherwise to its compressed representation. When operating in latent space, applying the function may further comprise decompressing the clean latent (obtained by iteratively applying the denoiser) to obtain a 2D image that is generated back in image space.

[0059] With reference to Figures 1-14, example implementations of machine learning methods and methods of use are described.

[0060] The trained function is conditioned (i.e., 3D-aware) based on the output scene encoding tensor. This allows it to leverage knowledge about the 3D structure and relationships of objects in a scene or environment, thereby improving consideration of, for example, perspective, occlusion, and lighting factors, resulting in more accurate and natural-looking immersive content. In particular, the trained function only comprises a scene encoder and a single conditional diffusion model specifically trained end-to-end for this task. It does not utilize large-scale, general-purpose pre-trained image synthesis pre-training, does not require multiple training phases to train modules individually, and does not require training a neural volume renderer or NeRF for each generated scene. Because 3D awareness is built in through the layout of the training samples, it does not rely on separate depth estimators, which are typically flawed and prone to error propagation, and does not require multi-view datasets for training. The method also enhances the level of user interaction and control.

[0061] The machine learning methods and usages solve the problem of generating high-quality user-specified 2D views of a 3D environment with a high-level / abstract description of the environment, i.e., annotated 3D bounding boxes representing elements (or objects) in the scene, that are not made up of off-the-shelf 3D models and textures.

[0062] To do so, the function is trained to generate an image of a scene given as input the camera viewpoint (position and rotation), a set of annotated bounding boxes representing objects in the scene (position, dimensions, orientation, and tags representing the class of the object), and the corners of the room the 3D scene represents (i.e., the boundary, shape, or floor plan).

[0063] Machine learning methods and usages solve this technical problem using a deep learning-based approach, the pipeline of which can be divided into two main stages.

[0064] In the first stage (offline stage), the machine learning method performs supervised training of the function. Given a dataset of pairs of 2D images and corresponding underlying scene annotations (camera viewpoint and field of view, annotated bounding boxes of visible and invisible objects present in the scene, and floor corners), the machine learning method comprises training the function, including a deep learning pipeline consisting of: Scene encoder (hereafter also called scene layout encoder). The scene encoder A layout encoder that outputs a vector embedding for each object present in the scene. a camera encoder that outputs a vector embedding that captures information about the camera; A floor encoder that outputs a vector embedding that captures information about the floor's shape. It comprises a transformer encoder that takes as input the sequence resulting from the concatenation of the aforementioned embeddings and outputs a sequence of new representations / embeddings. A diffusion model (hereafter also referred to as denoising diffusion model) that takes as input a noisy version of the provided 2D image and the scene embedding output by the scene encoder for conditioning. The denoising diffusion model may operate directly in image space or in latent space, in which case it may include a variational autoencoder (VAE).

[0065] The goal of the supervised training phase is to give the function the ability to reproduce (or generate) 2D reference images given as input the corresponding scene annotations (i.e., layout) of the 3D scene.

[0066] In the second stage (the inference or online stage), the trained function can be used to generate 2D images. On the one hand, a scene embedding tensor is computed using a layout encoder, a camera encoder, a floor encoder, and a transformer encoder by taking user-defined scene annotations as input. On the other hand, random Gaussian noise images are sampled (either in image space or latent space, depending on the nature of the diffusion model). The scene embedding and the image to be denoised are iteratively fed into the trained denoising diffusion model, which outputs a 2D image corresponding to the desired scene at the desired viewpoint.

[0067] Key benefits of machine learning methods and usage include: Time-efficient Once trained, the function (or model) is able to create high-quality 2D images faster than existing rendering solutions from traditional computer graphics. No expensive ray tracing or post-processing effects are used for generation. Generating a 2D view takes less than a few seconds on a GPU and less than a minute on a CPU. Creativity and user-driven The trained function can output scenes containing objects that were not present in the original dataset, without the need to create new 3D models and textures. The trained functions enable new creation workflows by allowing users to easily and quickly provide abstract 3D layouts without the typically tedious and time-consuming task of specifying and manually selecting the exact 3D items to be represented. Space efficiency Once trained, deep learning models can weigh on the order of several gigabytes on disk, but can generate infinitely many variations of objects. In comparison, storing thousands of 3D meshes and their textures is not very space-efficient. Because they can be specialized for relatively small datasets, 3D-aware diffusion models can be smaller (i.e., have fewer training parameters) than models employed by other methods. Image fidelity The trained function can output an image in which the influence of unseen objects is taken into account (e.g., for an interior scene, a window outside the camera's viewing frustum will affect the output image by affecting the lighting).

[0068] Definitions of certain terms are explained.

[0069] Deep neural networks (DNNs) are a powerful set of techniques for learning neural networks, a biologically inspired programming paradigm that enables computers to learn from observed data. In object recognition, the success of DNNs stems from their ability to learn rich mid-level media representations, as opposed to the hand-engineered low-level features (e.g., Zernike moments, HOG, bag-of-words, SIFT) used in other techniques (e.g., min-cut, SVM, boosting, random forests). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move away from feature engineering as much as possible by achieving end-to-end optimization starting from raw features and ending with labels.

[0070] Image generation models are a type of deep neural network that are trained on large image datasets to learn the underlying distribution of training images. By sampling from the learned distribution, such models can generate new images with features from images in the training dataset. GANs (generative adversarial networks), VAEs (variational autoencoders), and diffusion models are widely recognized as the most common image generation models, with diffusion models currently considered the state-of-the-art approach in the field.

[0071] Diffusion models are a type of deep learning model that can be used for image generation. They aim to learn the structure of a dataset by modeling how data points spread through a latent space. Diffusion models consist of three components: a forward process, a backward process, and a sampling phase. In the forward process, Gaussian noise is added to the training data through a Markov chain. The goal of training a diffusion model is to incrementally teach it how to undo the noise addition. This is done in the backward process, where the diffusion model reverses the noise addition performed in the forward process, thus restoring the data. In the sampling phase, the image-generating diffusion model starts with random Gaussian noise images. After being trained to reverse the diffusion process for images from the training dataset, the model can generate new images that resemble images from the dataset. This is achieved by starting with pure Gaussian noise and reversing the diffusion process until a clear image is obtained.

[0072] An autoencoder is a neural network architecture used for dimensionality reduction and data compression. It consists of an encoder that maps input data to a lower-dimensional representation and a decoder that reconstructs the original data from the encoded representation. Through compression and data reconstruction, autoencoders extract meaningful features, enabling tasks such as data compression. A variational autoencoder (VAE) is a special type of autoencoder that incorporates probabilistic modeling. Instead of learning a deterministic mapping, a VAE learns the parameters of a probability distribution over a latent space.

[0073] Transformers are a type of deep neural network architecture that has a remarkable ability to recognize relationships between elements in an input sequence. A mechanism called self-attention allows the Transformer to enable the model to learn the associations between each element and other elements and appropriately weight the contextual information. The Transformer module takes a sequence as input and outputs a new vector representation of the input data that emphasizes the relationships in the input sequence.

[0074] Cross-attention extends the self-attention mechanism by allowing correlations or contextual information to be selected between different sequences. The inputs used for cross-attention are two different sequences of the same or different modalities (e.g., text or images). The model learns to pay attention to relevant information from one of the sequences to the other. Cross-attention is appropriate when dealing with tasks that involve incorporating information from other sources to enhance the model's capabilities.

[0075] In the context of generative AI models for image synthesis, conditioning refers to the process of injecting additional information into the image generation process to obtain results that match user-driven constraints. Conditioning can take a variety of forms, including text (e.g., DALL-E2, Midjourney, or Stable Diffusion) or image (e.g., Control Net or semantic segmentation).

[0076] The (3D) bounding box of a three-dimensional (3D) object is the smallest rectangular parallelepiped that encloses the object. Its location, its dimensions, and its orientation characterize the 3D bounding box. Two opposite vertices are sufficient to completely describe a 3D bounding box.

[0077] A "viewpoint" represents the perspective or "camera" from which the render is captured. It may have four components: position, orientation, field of view, and pitch. The position and orientation of the viewpoint, as well as the position and orientation of the bounding box, may be defined within a single reference frame.

[0078] The term "3D abstract scene" refers to a list of labeled bounding boxes (a layout of a 3D scene) representing objects in the scene (where the labels correspond to classes of objects), a viewpoint, and, optionally, other elements that may enhance the description of the environment (e.g., information about the shape of a room). The adjective "abstract" emphasizes that the objects in the scene have no visual representation and are not defined beyond the characteristics of their bounding boxes and their labels.

[0079] One-hot encoding is a technique for representing categorical variables as binary vectors. It consists of creating a binary vector for each category with only one coordinate set to 1 and the rest set to 0. This representation allows converting categorical data into a numerical form that can be processed by deep neural networks.

[0080] A scene encoder is a specialized deep neural network that learns to extract a comprehensive representation from a 3D scene, which may include spatially arranged objects, layout, or viewpoints. A scene encoder accepts a variety of inputs depending on its specific architecture and the user's needs, and produces a high-dimensional vector output. This encoded representation should capture important features of the scene and serve as valuable input to subsequent stages of a deep learning model.

[0081] FIG. 1 shows a flow chart of an example machine learning method and method of use.

[0082] The pipeline comprises a diffusion model 100 conditioned by a novel 3D scene encoder 200. Similar to other deep learning models, it is characterized by an offline stage S100 (training a function by executing machine learning methods) and an online stage S200 (generating 2D images by executing machine learning methods, also called the inference stage).

[0083] Next, we will describe the offline training stage S100 in more detail. The purpose of this stage is to simultaneously (i) train the scene encoder 200 to generate a comprehensive mathematical representation that can be used for conditioning, and (ii) train the diffusion model 100 to generate images from noise. The scene encoder 200 receives as input a set of elements characterizing a 3D scene (the layout of the 3D scene) and outputs a scene encoding tensor. The diffusion model 100 receives as input a noisy version of the image to be generated and the scene encoding tensor, and outputs a denoised version of the input image. This training is end-to-end: a single loss value is computed and backpropagated to adjust the weights of both the diffusion model and the scene encoder. The training stage setup may comprise the following subtasks: Data preprocessing steps The data samples of the dataset, in particular the scene annotation (i.e., layout), can be processed so that they can be passed to a scene encoder. ·Architecture definition step The scene encoder 200 may return a single fixed-size tensor embedding for the entire scene being rendered. The diffusion model 100 may receive as input a noisy version of an image and a scene encoding vector. It may return an estimate of the noise added to the image and use it to suggest a less noisy version of the input image. · Training loss definition step The training loss function may measure the distance between the predicted noise in the input image and the true noise in the image added through the forward process. Training steps Training can be performed by iterating over the dataset (image and scene annotation pairs) several times.

[0084] Next, we will describe in more detail the image generation / inference stage S200, which aims to output a rendering that matches the viewpoint of an abstract 3D scene. At this stage, the method of use may comprise the following subtasks: Scene embedding vector determination step The scene embedding vector is computed using the scene encoder 200 and the input abstract 3D scene. The abstract 3D scene does not necessarily have to be part of the database (e.g., it can be user-created or generated using other techniques). Random Gaussian noise image generation step The Gaussian noise image 301 may have the size of the desired final image if training in pixel space, or the size of the latent space of the VAE if training a latent diffusion model. Iteratively denoising the generated images Using the diffusion model 100, we first denoise a random Gaussian noise image 301, and then iteratively denoise the output of the diffusion model. The U-Net denoiser (the DNN backbone of the diffusion model) can be conditioned on the scene embedding vectors using cross-attention between the layers of the U-Net and the scene embedding vectors. After a fixed number of denoising steps, a final sharp image 302 is produced.

[0085] We now describe an example implementation of the general framework described above, focusing on interior scene generation.

[0086] Details regarding the acquisition and the contents of the dataset used to train the function are provided below. The data used can be extracted from HomeByMe renderings (i.e., 2D images created by real users) taken by users. HomeByMe is a free interior design app that allows users to model their home in 3D by selecting and arranging furniture from an extensive object catalog and generate photorealistic 2D renderings of their rooms. Each time a high-quality rendering is produced by the application, a rich annotation file is saved along with the image. The raw data from this annotation file contains information about the rendering (semantic segmentation maps and / or 2D bounding boxes of visible objects) and information about the 3D scene where the rendering occurred (3D bounding boxes of objects, room geometry and / or viewpoint). Three elements can be extracted from this raw data: 3D bounding box For each object in the scene (not necessarily visible in the user-taken rendering), the annotation file contains a list of various features that describe the object, in particular the attributes of the object's class and its 3D bounding box. The raw data obtained from this file defines a 3D bounding box by two 3D points corresponding to two opposite vertices of the bounding box. There are a total of 174 possible classes in the HomeByMe dataset. ·perspective The user's point of view from which the rendering was taken may be saved in an annotation file. In particular, the camera position, its orientation and its field of view are captured and used later in the pipeline. Room shape The room geometry is stored in the annotation file as a list of 2D points representing the corners of the room.

[0087] The example function is trained on approximately 60,000 (3D annotated, HQ rendered) bedroom project pairs, although it can be extended to much larger datasets featuring other types of rooms.

[0088] FIG. 2 illustrates an example of the scene encoder 200 of FIG. 1. The scene encoder 200 includes a layout encoder 210 configured to encode a set of bounding boxes 201. For each bounding box in the set 201, the layout encoder 210 receives as input parameters representing the position, size, and orientation in the 3D scene of the object represented by the bounding box, and optionally the class of the object represented by the bounding box. The scene encoder 200 further includes a floor encoder 230 configured to encode a boundary 203 of the 3D scene. The scene encoder 200 further includes a camera encoder 220 configured to encode a viewpoint 202. As shown in FIG. 1, the scene encoder 200 further includes a transformer encoder 240. The transformer encoder receives as input the concatenation of the set of bounding boxes 201 encoded by the layout encoder 210, the viewpoint 202 encoded by the camera encoder 220, and the boundary 203 of the 3D scene encoded by the floor encoder 230. The transformer encoder 240 outputs a scene encoding tensor.

[0089] The offline training stage S100 will now be described in more detail.

[0090] Machine learning may comprise a data processing step prior to the training step to process the layout and perspective of each 2D image in the dataset.

[0091] The data processing step may include a first step for processing 3D bounding boxes in each 3D scene. The first step may include converting raw 3D bounding boxes from a representation based on two opposing vertices to a representation based on their positions (x, y, z), their dimensions (width w, height h, depth d), and their orientation. Objects present in a scene may have only one rotational degree of freedom, namely, their rotation around a vertical axis. As a result, machine learning methods may use only a single angle θ to define the orientation of the bounding box. In practice, machine learning methods may use a different representation, encoding the orientation of the 3D bounding box by the corresponding pair (cos(θ), sin(θ)). While such parameterization is mathematically equivalent to a single-valued parameterization, it enforces the continuity of the deep learning model between θ = 0 and θ = 2π. This is beneficial for model convergence. Thus, the processed 3D bounding box is defined by a list of eight parameters: x, y, z, w, h, d, cos(θ), sin(θ).

[0092] The data processing step may comprise a second step for processing classes of objects. Each object from the HomeByMe dataset may be described by a class that provides a broad description (chair, table, or door). There may be a total of 174 classes in the HomeByMe dataset. To be fed into the deep learning model, the classes of the objects are set to {0,1} 174 This step can be performed in other ways (e.g., using vector representations / embeddings of text classes of objects using common techniques such as word2vec, as described in Mikolov, T., Chen, K., Corrado, G., Dean, J., 2013, "Efficient Estimation of Word Representations in Vector Space," arXiv preprint arXiv:1301.3781).

[0093] The data processing step may comprise a third step for processing the boundaries of the layout. The third step may comprise increasing the dimensions of the floor points. The raw points of the data annotation are 2D points (x,y) because their Z coordinate is implicitly 0. The 2D points are converted to 3D points by using 0 as the Z coordinate. This step is necessary so that the 3D points can be affected by the transformations described below.

[0094] The data processing step may include a fourth step to process the bounding box coordinates, specifically from world coordinates to camera coordinates (shown in Figure 3). The raw positions and orientations found in the annotation file use the world coordinates defined in HomeByMe. To reduce the number of training parameters, increase robustness, and thus speed convergence, the data processing step may perform a change of basis, moving from the original world coordinates to a viewpoint-based coordinate system. In the new coordinate system, the world origin is set to the camera's position, and the basis vectors are selected as follows: the "Z" basis vector remains unchanged, the "Y" basis vector is the projection of the viewpoint's forward vector onto a plane perpendicular to the "Z" vector, and the "X" vector is perpendicular to the two mentioned above. With this change of basis, the viewpoint can be described purely by two scalar values: the field of view (FOV) and pitch (the angle its forward vector makes with the "Y" basis vector). This change of basis affects the position and rotation of all objects and points in the scene. While this change of basis is optional, it aids in the convergence of the function.

[0095] We now describe the scene encoder architecture in more detail.

[0096] The scene encoder consists of four components: a layout encoder, a camera encoder, a floor encoder, and a transformer module (or transformer encoder) (see Figure 2).

[0097] Next, we describe the layout encoder. The scalar values (x, y, z, w, h, d, cos(θ), sin(θ)) describing each bounding box in the scene can be passed to a position encoding module (PE) that deterministically increases the dimensionality of the scalar values. In this example, the scalar values are

number

[0098] After the position encoding module, the position and dimensions of the bounding box, originally described by three scalar values each, are described by a 192-dimensional vector (3 × 64 = 192). Meanwhile, the rotation, originally described by a pair of scalar values, is described by a 128-dimensional vector after position encoding. To ensure that the position, dimensions, and rotation are weighted similarly by the model, a higher-dimensional version of the rotation is passed to a multi-layer perceptron, which then

number

[0099] The one-hot encoded categories are {0,1} 174 To ensure that the categories are weighted similarly to the position, dimension, and rotation of the bounding box, the category vector is passed to a multi-layer perceptron, which then calculates

number

[0100] All previously calculated vectors are

number

[0101] Next, we describe the camera encoder. A camera or viewpoint is completely described by two scalar values: field of view and pitch. Both of these values can be encoded using position encoding into a higher dimensional

number

[0102] Next, we will describe the floor encoder. The floor is represented solely by a set of unordered 3D points corresponding to its corners. Such a representation is ambiguous and cannot be easily interpreted by a deep neural network. Alternatively, the data processing step may comprise densely sampling points along the walls of the room so that the room boundary is represented by a 3D point cloud, thereby generating a set of sampled points along the boundary. This 3D point cloud is then

number

number

[0103] Next, we describe the Transformer module. The 3D bounding box tokens, camera tokens, and floor tokens are all concatenated to form a sequence of tokens. These tokens are independent of each other. To capture the relationships between different elements of this sequence, the Transformer module is used. Due to its inherent architecture, the operation of the Transformer module can be improved for a fixed input size. However, the sequence constructed by concatenating the outputs of the layout encoder, camera encoder, and floor encoder can be of variable length because the number of 3D bounding boxes in a scene can vary from scene to scene. To be compatible with the Transformer architecture, we add "zero" tokens to the concatenation of the vector sequence so that the sequence is of fixed length.

number

number

[0104] Next, we describe the architecture of the diffusion model. The function can comprise one of two versions of the diffusion model: one that operates directly in image space, or one that operates in the latent space of a pre-trained VAE to increase the final image dimensionality. In the first case, diffusion occurs directly on pixels from the image, while in the second case, diffusion occurs on a latent version of the image, which is decoded using a VAE decoder. The two approaches are not fundamentally different and do not require significant changes beyond the introduction of the VAE.

[0105] The diffusion model has a conditional generative architecture and features a U-Net backbone with four down blocks and four up blocks. In particular, the last two down blocks and the first two up blocks can be augmented with cross-attention using scene embedding vectors. The number of up / down blocks and the number of blocks augmented with cross-attention can vary depending on the user's needs and means. This configuration provides an optimal compromise between image quality and training time.

[0106] Next, we will explain the training loss used in training. The diffusion model can be trained using various losses / parameterizations. It is known that the noise added to the input image during the forward process is added deterministically. During training, the diffusion model tries to predict the noise added to the image. The loss used in this case is the ratio of the true noise ε to the predicted noise ε. θ where θ indicates that the prediction is based on the parameters of the model.

[0107]

number

[0108] Alternatively, other commonly used diffusion training parameterizations / losses may be interchangeably adopted. For example, the v-prediction parameterization with a min-SNR weighting value of 5.0 leads to a good image quality / resolution / computation tradeoff (e.g., as described in Tiankai Hang et al.'s paper "Efficient Diffusion Training with Min-SNR Weighting Strategy" ICCV2023).

[0109] Next, we will describe the generation / inference stage S200 in more detail. Diffusion models can be constructed during inference using various techniques. For example, two different sampling processes can be used: the denoised diffusion probabilistic model (DDPM) and the denoised diffusion implicit model (DDIM). DDIM, for example, offers the best balance between inference speed and image quality. Inferring an image of a given 3D abstract scene can take approximately 1 second to generate on an NVIDIA RTX A6000 GPU.

[0110] Next, examples of the results will be described with reference to FIGS.

[0111] Figure 4 shows a first example of a 3D scene layout and viewpoint. The figure shows bounding boxes encapsulating the objects in the room represented by the 3D scene. The bounding boxes of the objects can be colored according to their category. These layouts and viewpoints are received as input by a trained function to generate the 2D image shown in Figure 5. The generated 2D image shows well-represented conditioning elements. Off-screen objects also influence the generated content (e.g., the effect of lighting on a bed directly in front of a room window). The 3D bedroom layouts used for qualitative results and quantitative evaluation were extracted from an independent set not used in the training distribution. This helps ensure the robustness and generalization capabilities of the model.

[0112] Similarly, Figure 6 shows a second example of a 3D scene layout and viewpoint, and Figure 7 shows the resulting 2D image produced by the trained function.

[0113] Figure 8 shows a third example of a 3D scene layout and viewpoint, and Figure 9 shows the resulting 2D image produced by the trained function. If the user manipulates the input 3D layout, for example by removing the lamp on the left nightstand, the function may again produce a different image, shown in Figure 10.

[0114] FIG. 11 shows a fourth example of a 3D scene layout and viewpoint, and FIG. 12 shows the resulting 2D image produced by the trained function.

[0115] The trained function tackles two challenges: generating 2D images from a 3D scene. First, the diffusion model adheres to its 3D conditioning to ensure objects appear in their expected positions. That is, given a viewpoint and a set of objects in 3D space, the model outputs the objects in positions similar to how a traditional 3D renderer would. Second, the trained function not only accurately positions objects, but also generates recognizable or correct objects. As a result, the trained function enables high local image fidelity and therefore high-quality output images.

[0116] Next, we present the results of a quantitative evaluation performed to evaluate the trained function. The 2D images generated by the trained function are evaluated using metrics that assess both 3D conditioning and local image quality. The goal of these metrics is to determine whether an object is correctly positioned in the 2D image and whether it is recognizable. This requires a ground truth image and a generated image corresponding to the same viewpoint. This leverages the CLIP model (as described in the paper "Learning Transferable Visual Models from Natural Language Supervision" by Radford, Alec, et al., International Conference on Machine Learning, PMLR, 2021). CLIP (Contrastive Language-Image Pretraining) is a multimodal foundational model that learns a joint latent space between text and images. It is trained on hundreds of millions of (text, image) pairs, allowing it to connect complex visual concepts with their descriptions in natural language. Embeddings computed by CLIP's text encoder from a text prompt will have high cosine similarity to embeddings computed by its image encoder from images semantically close to the prompt. CLIP has been widely adopted in recent research work in computer vision due to its powerful zero-shot capabilities, and can perform several tasks such as image classification or open-vocabulary semantic segmentation. Its shared latent space allows text and image modalities to be used interchangeably. Metrics are computed by performing the following steps for each object in the ground truth image: 1. Crop the area corresponding to the object. 2. Compute the CLIP embedding of the crop. 3. Use CLIP as a zero-shot classifier of the top-k acquisitions to check whether the true category of the cropped object is contained in the top category of the acquisition. The classifier key is calculated using the CLIP embedding of the parsed text object category. Cosine similarity is used as a measure to compare the crop CLIP with the category CLIP. 4. If a crop is detected in the ground truth image, crop the generated image to the same area. 5. Compute CLIP embeddings for the generated crops. 6. Use CLIP as a zero-shot classifier for top-k retrieval and check whether the generated object categories are correct or not. 7. The accuracy value is the ratio of correctly identified generated crops to correctly identified ground truth crops.

[0117] The results obtained using this metric on an evaluation set of approximately 100 scenes are shown in Figure 13. A quantitative evaluation of the function shows that it correctly identifies 60% of the objects in the top 10 acquisition scenarios (out of 174 classes). This accuracy improves to over 75% for the top 25 acquisitions. These results demonstrate that the trained diffusion model of the function produces objects that are not only accurately positioned, but also realistic enough to be recognized by the CLIP zero-shot classifier in most cases.

[0118] FIG. 14 shows an example of a system, where the system is a client computer system, for example a user's workstation.

[0119] The example client computer includes a central processing unit (CPU) 1010 connected to an internal communication BUS 1000 and a random access memory (RAM) 1070 also connected to the BUS. The client computer is further provided with a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass memory devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the foregoing may be supplemented by, or incorporated in, specially designed ASICs (application-specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a keyboard, or other haptic device 1090. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. Furthermore, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, with mouse buttons used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0120] A computer program may comprise computer-executable instructions, which comprise means for causing the system to perform a method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as an article of manufacture tangibly embodied in a machine-readable storage device, executed by a programmable processor. The steps of the method may be performed by a programmable processor executing a program of instructions that performs the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language, as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In either case, applying the program to a system generates instructions for performing the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, the server communicating with one or more clients via a network, in which case the processing unit executes the instructions contained in the program, thereby performing the method in the cloud computing environment.

Claims

1. 1. A computer-implemented method for machine learning a function configured to generate a 2D image of a 3D scene, the function comprising a scene encoder and an image generation model, the scene encoder receiving as input a layout and a viewpoint of the 3D scene and outputting a scene encoding tensor, the image generation model receiving as input the scene encoding tensor output by the scene encoder and outputting the generated 2D image; acquiring a dataset comprising 2D images and corresponding layouts and viewpoints of a 3D scene; training the function based on the acquired data set; The machine learning method comprises:

2. The layout of each 3D scene is: a set of bounding boxes representing objects in the 3D scene; and the boundary of the 3D scene; The machine learning method of claim 1 , comprising:

3. 3. The machine learning method of claim 2, wherein the scene encoder includes a layout encoder configured to encode the set of bounding boxes, the layout encoder receiving as input, for each bounding box in the set, parameters representing a position, a size, and an orientation within the 3D scene of the object represented by the bounding box and, optionally, a class of the object represented by the bounding box.

4. The machine learning method of claim 3 , wherein the scene encoder further comprises a floor encoder configured to encode the boundary of the 3D scene.

5. The machine learning method of claim 4 , wherein the scene encoder further comprises a camera encoder configured to encode the viewpoint.

6. 6. The machine learning method of claim 5, wherein, for each given 2D image of a given 3D scene in the dataset, the size and the position of the object represented by the bounding box in the layout of the given 3D scene are defined in a coordinate system based on a position and orientation of a camera from which the given 2D image was taken, each viewpoint comprising a field of view and pitch of the camera.

7. 7. The machine learning method of claim 5, wherein the scene encoder further comprises a transformer encoder that receives as input a concatenation of the set of bounding boxes encoded by the layout encoder, the viewpoint encoded by the camera encoder, and the boundary of the 3D scene encoded by the floor encoder, and outputs the scene encoding tensor.

8. The machine learning method according to claim 1 , wherein the image generation model is a diffusion model.

9. 9. The machine learning method of claim 8, wherein the diffusion model has an architecture including a denoiser with blocks, at least one of the blocks being augmented by cross-attention using the scene encoding tensor.

10. 10. The machine learning method of claim 8 or 9, wherein the diffusion model is configured to operate in a latent space, and the diffusion model is trained to denoise compressed latent representations of the 2D images of the dataset.

11. 11. A method for using a machine-learned function according to any of the methods of claims 1 to 10, comprising: Obtaining a layout of a 3D scene; applying said function to said layout of a 3D scene, thereby generating a 2D image of said 3D scene; A method of use comprising:

12. The image generation model is a diffusion model, and applying the function applying the scene encoder to the obtained layout, thereby outputting a scene encoding tensor; using the diffusion model conditioned on the output scene encoding tensor to generate the 2D image of the 3D scene; The method of claim 11, comprising:

13. A computer program comprising instructions for carrying out the machine learning method according to any of claims 1 to 10 and / or the method of use according to claim 11 or 12.

14. A computer-readable storage medium having the computer program according to claim 13 recorded thereon.

15. 14. A system comprising a processor coupled to a memory having the computer program of claim 13 recorded thereon.

Images