Generating a 3D image of a 2d scene using a conditioning signal

Abstract

To provide a computer-implemented method for generating a 3D image of a 2D scene.SOLUTION: The method includes obtaining a layout of a 3D scene and at least one conditioning signal. Each conditioning signal has a type from a predetermined set of at least two types. The method also includes applying a machine learning function to the obtained location data and viewpoint. The function comprises a scene encoder and a generative image model. The scene encoder takes the obtained placement data and the viewpoint as input and outputs a scene encoding tensor. The generated image model takes as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. Such a generation method forms an improved solution for controllably generating a 3D image of a 2D scene.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems and programs for generating 2D images of 3D scenes. [Background technology]

[0002] Numerous systems and programs are available on the market for designing, engineering, and manufacturing objects. CAD is an acronym for Computer-Aided Design, which relates to software solutions for designing objects, for example. CAE is an acronym for Computer-Aided Engineering, which relates to software solutions for simulating the physical behavior of future products, for example. CAM is an acronym for Computer-Aided Manufacturing, which relates to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, the graphical user interface plays a key role in the efficiency of the technology. These technologies can be incorporated into product lifecycle management (PLM) systems. PLM refers to a business strategy that supports sharing enterprise product data, applying common processes, and leveraging enterprise knowledge to develop products from conception to the end of their lifespan across the extended enterprise. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA, 3DVIA, and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to both the Engineering and Manufacturing Hubs. Overall, the systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this context, applications for 3D scene creation have been developed. These applications generally propose to create, manipulate, and present 3D scenes, especially (though not exclusively) for touch-sensitive devices (e.g., smartphones or tablets). One task of these applications is the generation of realistic 2D images of the 3D scene.

[0004] Solutions for generating 2D images of 3D scenes have been developed in recent years, for example, using generative deep learning models. However, these solutions do not fully consider the entire 3D environment of the imaged scene. In particular, these solutions cannot leverage knowledge of the 3D structure and relationships of objects in the scene or environment. In particular, they do not account for, for example, perspective, occlusion, or lighting factors, and therefore cannot create accurate and natural immersive content. Furthermore, current solutions do not allow for control over the overall visual appearance of the generated image, nor the objects contained therein.

[0005] Within this context, there remains a need for improved solutions for controllably generating 2D images of 3D scenes. Summary of the Invention

[0006] Accordingly, a computer-implemented method (hereinafter referred to as the generating method) for generating a 2D image of a 3D scene is provided. The method includes acquiring geometry data including a layout of the 3D scene and at least one conditioning signal. Each conditioning signal has one type from a predetermined set of at least two types. The method includes applying a machine learning function to the acquired geometry data and a viewpoint. The function comprises a scene encoder and a generative image model. The scene encoder takes as input the acquired geometry data and the viewpoint and outputs a scene encoding tensor. The generative image model takes as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image.

[0007] The method of production may include one or more of the following: Obtaining the configuration data includes selecting, upon user interaction, a type of each conditioning signal from a predetermined set of at least two types. The predetermined set of at least two types includes an image type and a text type. The layout of the 3D scene includes bounding boxes each representing a respective object in the 3D scene. The at least one conditioning signal includes: one or more conditioning signals for the 3D scene, and / or o Includes one or more conditioning signals for each object represented by one of the at least portions of the bounding boxes. Obtaining the positioning data includes selecting, upon user interaction, one or more conditioning signals for the 3D scene. and / or - obtaining the positioning data includes, for each given bounding box of at least a portion of the bounding boxes, selecting, upon user interaction, one or more respective conditioning signals for the object representing the given bounding box; The scene encoder comprises a multimodal encoder configured to project each conditioning signal into a single latent space.

[0008] Also provided is a computer-implemented method (hereinafter referred to as the machine learning method) for machine learning a function used in the generating method. The machine learning method includes acquiring a dataset including training samples, each of which includes a 2D image, configuration data, and a viewpoint. The configuration data of at least some of the training samples include conditioning signals. The conditioning signals of at least some of the training samples include at least one first conditioning signal having a first type from a predetermined set of at least two types and / or at least one second conditioning signal having a second type from the predetermined set of at least two types. The machine learning method includes training a function based on the acquired dataset.

[0009] The machine learning methods may include one or more of the following: The machine learning method further comprises replacing a predetermined portion of the conditioning signals of the dataset by conditioning signals having predetermined values ​​before or during training. Obtaining the configuration data includes determining at least one conditioning signal for the object from 2D images of training samples. The first type is an image type, and the obtaining comprises determining at least one first conditioning signal by applying an image generator. the first type is an image type, and the obtaining comprises modifying at least a portion of at least one first conditioning signal, and the function is trained taking into account each modified first conditioning signal; and / or The second type is a text type. Obtaining includes determining the at least one second conditioning signal by applying a text generator.

[0010] There is also provided a computer program comprising instructions, which when executed by a computer cause the computer to perform the generation method and / or the machine learning method.

[0011] There is also provided a computer readable storage medium having a computer program recorded thereon.

[0012] There is also provided a system comprising a processor coupled to a memory, the memory having a computer program stored thereon, and optionally a graphical user interface coupled to the processor.

[0013] There is further provided a device comprising a data storage medium having a computer program recorded thereon.

[0014] The device may form or function as a non-transitory computer-readable medium, for example, in a Software as a Service (SaaS) or other server or cloud-based platform. Alternatively, the device may comprise a processor coupled to a data storage medium. The device may thus form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a graphical user interface coupled to the processor. [Brief explanation of the drawings]

[0015] Non-limiting examples will now be described with reference to the accompanying drawings. [Figure 1] FIG. 1 is a flowchart illustrating an example of a generation method and a machine learning method. [Figure 2] FIG. 1 illustrates an example of a scene encoder. [Figure 3] FIG. 10 is a diagram illustrating an example of camera coordinates. [Figure 4]FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 5] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 6] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 7] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 8] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 9] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 10] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 11] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 12] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 13] FIG. 1 shows an example of the layout and viewpoint of a 3D scene and the resulting 2D image generated by the trained function. [Figure 14] FIG. 10 shows the results of a quantitative evaluation performed to evaluate the trained functions. [Figure 15] FIG. 1 illustrates an example of a system. DETAILED DESCRIPTION OF THE INVENTION

[0016] A computer-implemented method (hereinafter referred to as the generative method) for generating a 2D image of a 3D scene is proposed. The method includes acquiring geometry data including a layout of the 3D scene and at least one conditioning signal. Each conditioning signal has one type from a predetermined set of at least two types. The method includes applying a machine learning function to the acquired geometry data and a viewpoint. The function comprises a scene encoder and a generative image model. The scene encoder takes as input the acquired geometry data and the viewpoint and outputs a scene encoding tensor. The generative image model takes as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image.

[0017] Such a generation method forms an improved solution for controllably generating 2D images of a 3D scene.

[0018] In particular, the generation method enables the automatic and efficient generation of 2D images of a 3D scene. In particular, applying the function enables the generation of (various and realistic) 2D images from high-level, abstract, and proxy representations of the 3D scene (and is therefore easy to define). Indeed, upon training, the function enables the generation of 2D images of a 3D scene from only the layout and viewpoint of the 3D scene. From the geometry data and viewpoint, the trained function enables the generation of 2D images, which are particularly useful and interesting for presenting objects within the 3D scene. In particular, providing these two inputs to the trained function is much easier for a user than providing accurate object models for each object in the 3D scene and then using traditional rendering methods. Thus, the trained function enables the easy and rapid generation of 2D images of the 3D scene the user is building by simply defining their layout and providing viewpoints for these images.

[0019] In particular, the function generates particularly realistic and relevant 2D images of a 3D scene. In fact, the function comprises a scene encoder that can take into account the layout in the generated 2D image. Thus, the function can generate 2D images that take into account the perspective of the 3D scene and its lighting, as well as occlusions between objects (the scene encoder can take this information into account during the generation of the 2D image). In other words, the 2D images generated by the trained function are 3D-aware, i.e., take into account the 3D environment of the 3D scene. In particular, the method allows for the consideration of off-screen objects in the 3D scene when generating the 2D image, i.e., objects that are not within the camera's field of view affect the generated image (e.g., light from a window).

[0020] Furthermore, the method allows for control of the generated 2D image. Indeed, the configuration data taken as input by the function also includes at least one conditioning signal used to condition the rendering of the image. The at least one conditioning signal allows for control of the visual aspects of the generated 2D image. In particular, the at least one conditioning signal may be of several types (i.e., at least two types of a predetermined set), giving the user a large degree of freedom in conditioning. The scene encoder allows for the use of at least two types of conditioning signals to control the visual aspects of the generated 2D image.

[0021] Furthermore, the function is trained to generate a variety of 2D images conditioned by at least one conditioning signal. Indeed, for a given 3D layout and viewpoint, the function can generate a variety of 2D images in terms of object style, color, etc., while still taking into account the layout, viewpoint, and at least one conditioning signal obtained and captured by the function as input. Thus, the user can obtain multiple inspirations conditioned by at least one conditioning signal for a given layout and viewpoint.

[0022] Furthermore, the proposed machine learning method is trained end-to-end for the task in a single training phase: it does not rely on large-scale pre-trained image generation models or pre-trained depth estimators.

[0023] The generation method and / or machine learning method are computer-implemented. This means that the steps (or substantially all steps) of the generation method and / or machine learning method are performed by at least one computer, or any similar system. Thus, the steps of the generation method and / or machine learning method are performed by a computer, possibly fully automatically or semi-automatically. In examples, triggering of at least some of the steps of the generation method and / or machine learning method may be performed through user-computer interaction. The level of user-computer interaction required depends on the expected level of automation and may be balanced against the need to implement user preferences. In examples, this level may be user-defined and / or predefined.

[0024] For example, in the generation method, the step of obtaining the positioning data and viewpoint may include executing a user-generated process for generating, upon user interaction (e.g., performed by a user, e.g., currently designing a 3D scene), the positioning data and viewpoint of the 3D scene, which are taken as inputs by the function. Generating the positioning data and viewpoint may be performed in any manner. For example, the user-generated process may include determining a layout of the 3D scene and / or at least one conditioning signal, which, upon user interaction, are taken as inputs by the function. Determining these inputs will now be described.

[0025] With respect to the layout of the 3D scene, the user-generated process may include determining one or more (e.g., all) of the bounding boxes that the 3D scene will contain. Determining the given bounding box may be performed in any manner. For example, determining the given bounding box may include setting a size of the given bounding box and placing the sized bounding box inside the 3D scene. The sizing and placing steps may be performed manually by a user. For example, the sizing step may include a user inputting a width, depth, and height (e.g., through user interaction using a keyboard). The placing step may include a user inputting coordinates of a point on the bounding box (e.g., a corner or its center) and its direction, or may include a user moving the bounding box to its position within the 3D scene (e.g., the bounding box may be displayed on a screen and moved by a user using a mouse). In an example, the sizing step may be performed semi-automatically. For example, the step of setting the size may include a user selecting a category of object to be represented by a given bounding box and automatically suggesting a size (e.g., width, depth, and height) of the given bounding box according to the selected category (e.g., from a database storing default sizes for different categories of objects). The suggested size may be accepted by the user or may be adjusted by the user (e.g., manually as described above). For example, if a user wants to add a couch to their 3D layout, the user-generated process may suggest a default bounding box that matches the object category entered by the user, with the user being able to modify these dimensional values.

[0026] Alternatively or additionally, the user-generated process may include determining a boundary of the 3D scene. Determining the boundary of the 3D scene may be performed in any manner. For example, determining the boundary of the 3D scene may include determining a set of respective points that represent the boundary of the 3D scene. For example, determining the boundary may include determining some of the points in the set (e.g., representing corners of the 3D scene) and then sampling other points on the boundary between these points (i.e., along the walls of the 3D scene) that represent the corners of the 3D scene.

[0027] Alternatively or additionally, the user-generated process may include, upon user interaction, determining a viewpoint of the 3D scene, which is taken as input by the function. Determining the viewpoint may include setting parameters of a camera from which the 2D image is generated. These parameters may include camera position, field of view, and / or pitch. Determining these parameters may include the user manually entering them and, optionally, suggesting at least some of them to the user (e.g., with default values). For example, determining the viewpoint may be performed by the user, e.g., by entering viewpoint coordinates and / or direction or by selecting this information on a screen displaying the 3D scene. Alternatively, determining the viewpoint may be performed automatically, e.g., by another function predicting one or more relevant viewpoints of the 3D scene given its layout.

[0028] In other examples, obtaining the configuration data may include retrieving, for example, from a database, the configuration data and viewpoints to be taken as inputs by the function. In that case, the configuration data and / or viewpoints may have been generated prior to execution of the generation method, for example, using the user-generation process described above, or may have been recorded in the database after execution of the user-generation process. In yet other examples, obtaining may include retrieving some of the inputs (e.g., a layout of the 3D scene) and executing the user-generation process described above to generate the remainder (e.g., at least one conditioning signal and / or viewpoint). In that case, some of the inputs may also have been generated prior to execution of the generation method (e.g., using the user-generation process).

[0029] For at least one conditioning signal, obtaining the configuration data may include selecting one or more (e.g., each) conditioning signal types from a predetermined set of at least two types upon user interaction. For example, selecting each conditioning signal type may include a user inputting the selected conditioning signal, where the selected type is the type of the input selected conditioning signal. The selected conditioning signal may be stored, for example, in a database, and inputting may include a user selection of the stored conditioning signal. Each conditioning signal type may be among a predetermined set of at least two types, which may include an image type and a text type. An image-type conditioning signal may be a 2D image of a 3D scene or an object. A text-type conditioning signal may be free text (e.g., a dialog box) describing, for example, the atmosphere of a room or an object.

[0030] The at least one conditioning signal may include one or more conditioning signals for the 3D scene. These one or more conditioning signals condition the entire 3D scene, for example, the atmosphere or feel of the entire 3D scene. The conditioning signal for the 3D scene may be of an image type or a text type. Alternatively or additionally, the at least one conditioning signal may include one or more conditioning signals for each object represented by one of at least a portion of the bounding boxes. For example, the placement data may include one or more conditioning signals for at least some (e.g., all) of the objects present in the 3D scene (each represented by a respective bounding box in the layout).

[0031] One or more conditioning signals for the 3D scene or for the objects represented by the bounding boxes may have been selected as described above. For example, obtaining the positioning data may include selecting one or more conditioning signals for the 3D scene upon user interaction. Selecting the conditioning signals for the 3D scene may include a user inputting the selected conditioning signals for the 3D scene. The selected conditioning signals may be stored, for example, in a database, and the inputting may include a user selection of the stored conditioning signals. Alternatively or additionally, obtaining the positioning data may include, for each given bounding box of at least a portion of the bounding boxes, selecting, upon user interaction, one or more respective conditioning signals for the objects representing the given bounding box. Selecting one or more respective conditioning signals for the objects may be performed sequentially or simultaneously, for example, by user selection from a database storing candidate conditioning signals (e.g., including 2D or image images of the objects) and / or by inputting text describing the objects.

[0032] A typical example of a computer implementation of the generation method and / or machine learning method is to perform the generation method and / or machine learning method using a system adapted for this purpose. The system may comprise a processor coupled to a memory and a graphical user interface (GUI), and the memory stores a computer program including instructions for performing the generation method and / or machine learning method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally comprise several physically separate parts (e.g., one for the program, and possibly one for the database).

[0033] The dataset considered by the machine learning method for training the function may be stored in a database. By "database" is meant any collection of data (i.e., information) organized for search and retrieval (e.g., a relational database based on a predefined structured language, e.g., SQL). When stored in memory, a database allows for rapid search and retrieval by a computer. In practice, databases are structured to facilitate the storage, retrieval, modification, and deletion of data, along with various data processing operations. A database can consist of a file or a set of files that can be divided into records, each of which consists of one or more fields. A field is the basic unit of data storage. Users can retrieve data primarily through queries. Using keywords and sorting commands, users can quickly explore, rearrange, group, and select fields within many records to search or create reports on specific aggregations of data according to the rules of the database management system they are using.

[0034] Generative and / or machine learning methods generally manipulate modeled (3D) objects. A modeled object is any object defined by data stored, for example, in a database. By extension, the term "modeled object" refers to the data itself. Depending on the type of system, a modeled object may be defined by various types of data. In fact, a system may be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these various systems, a modeled object is defined by corresponding data. Thus, it can be referred to as a CAD object, a PLM object, a PDM object, a CAE object, a CAM object, CAD data, PLM data, PDM data, CAM data, or CAE data. However, since a modeled object may be defined by data corresponding to any combination of these systems, these systems are not exclusive of one another. Thus, a system may be both a CAD system and a PLM system.

[0035] A CAD system also refers to any system at least adapted for designing a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, data defining the modeled object includes data enabling the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, and in certain cases, faces or surfaces. Lines, edges, or surfaces may be represented in various ways, for example, with non-uniform rational B-splines (NURBS). In particular, a CAD file contains specifications from which geometry can be generated, thereby enabling the generation of a representation. The specifications of a modeled object may be stored in a single CAD file or in multiple CAD files.

[0036] In an example, each 3D scene may represent a real room, e.g., an indoor real room. For example, the rooms represented by the 3D scenes may be rooms in a residential (e.g., house or apartment) such as a kitchen, bathroom, bedroom, living room, garage, laundry room, attic, office (e.g., private or shared), conference room, children's room, nursery, hallway, dining room, and / or library (this list may include other types of rooms). Alternatively, the rooms represented by the 3D scenes may be other indoor rooms, such as in a factory, museum, and / or theater. Alternatively, the rooms represented by the 3D scenes may be outdoor scenes, such as a garden, terrace, or amusement park.

[0037] Each object in each 3D scene may represent the geometry of a real object located in the real room that the 3D scene represents. This real object may be manufactured in the real world after completion of its virtual design (e.g., using a CAD software solution or system). The 3D scene may comprise one or more furniture objects, such as one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more sideboards, one or more nightstands, one or more desks, and / or one or more wardrobes. Alternatively or additionally, the 3D scene may comprise one or more decorative objects, such as, for example, one or more accessories, one or more plants, one or more books, one or more frames, one or more kitchen accessories, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors, and / or one or more electronic objects (e.g., a refrigerator, a freezer and / or a washing machine).

[0038] The generative method may be included in a real-life room design (i.e., effective placement) process, which may include using the generated 2D image to present the room to be placed after performing the generative method. For example, the presentation may be to a user, such as the owner of the home in which the room is located. The generated 2D image may be used to help the user decide whether to acquire one or more objects in the 3D scene, or may assist the user in their selection by displaying one or more objects in the room. During the real-life room design process, the generative method may be iterated to determine several 2D images of the room. Iterations of the generative method may be used to present a complete virtual interior of the room (i.e., including several 2D images of the room) and / or to obtain 2D images of the 3D scene with different styles and / or object appearances.

[0039] Alternatively or additionally, real-life room design may include using the generated 2D images to perform a similarity-based search for 3D objects from a catalog to be placed at the locations of the bounding boxes. The trained function enables this due to the realism of the generated 2D images. For example, real-life room design may include a user defining the layout of a given 3D scene by placing 3D bounding boxes. Next, real-life room design may include using the trained function to generate several 2D images of the given 3D scene (e.g., by repeating the aforementioned generation method). Then, real-life room design may include the user selecting one of the generated 2D images. For example, a user may particularly like the appearance and / or style of one of the generated 2D images and want to populate the given 3D scene with the most similar 3D object from the catalog (i.e., replace the bounding box with the actual 3D object). In that case, real-life room design may include: for each object in the generated 2D image, deriving a position of each object in the generated 2D image from the defined layout; cropping the object in the image; calculating an image embedding of the object (e.g., using a pre-trained language image model (hereinafter referred to as CLIP) such as that described in the paper "Learning transferable visual models from natural language supervision" by Radford et al., International Conference on Machine Learning, PMLR2021); and comparing this image embedding of the object with one from the catalog to obtain the most similar object. The real-life room design may also include replacing a bounding box in the 3D scene with the most similar object from the obtained catalog for each object.

[0040] Alternatively or additionally (e.g., prior to presentation), the real-life room design process may include adding one or more new objects to a 3D scene (which may initially be, e.g., partially empty) representing a room by modifying the layout of the 3D scene. Adding may include repeating, for each new object, the steps of sizing and positioning a bounding box representing the new object as discussed above. Thus, the generated 2D image may include the new object added to the 3D scene by modifying the layout. The real-life room design process enables the creation of rich and comfortable environments (for animation, advertising, and / or for generating virtual environments, e.g., for simulation). The real-life room design process may also be used to generate virtual environments. The real-life room design process may be included in a general process that may include repeating the real-life room design process for several 3D scenes, thereby presenting several 3D scenes with objects.

[0041] Alternatively or additionally, the real-life room design process may include physically arranging a (i.e., actual) room after the method is performed so that its design matches the 3D scene presented using the generated 2D image. For example, a room (without the objects represented by the input 3D scene) may already exist in the real world, and the real-life room design process may include placing an actual object (i.e., an object represented by one of the bounding boxes of the layout) represented by one of the objects of the 3D scene within the already-existing (i.e., real-world) room. The bounding box of this object may be added by the user to the layout of the 3D scene. The actual object may be placed according to the location of its bounding box within the 3D scene. The real-life room design process may repeat this process to place various actual objects within the already-existing room. Alternatively, the room may not yet exist at the time the method is performed. In that case, the real-life room design process may include building a room (i.e., adding real objects to the room) according to the generated 2D image of the 3D scene (i.e., by placing real objects in the positions of bounding boxes representing the real objects in the layout of the 3D scene). Because the method improves the placement of 3D objects in the 3D scene, the method also improves the construction of a room corresponding to the 3D scene, and therefore improves the productivity of the real-life room design process.

[0042] We now describe the machine learning method, and in particular the steps for obtaining a dataset.

[0043] The dataset includes multiple training samples (e.g., over 50,000 training samples including 2D images of the same type of room), each including a 2D image. The dataset also includes, for each training sample, configuration data (e.g., the layout of the 3D scene and / or conditioning signals) of the 3D scene captured (e.g., partially) within the 2D image included in the training sample, and the viewpoint from which the 2D image is captured (e.g., coordinates of the viewpoint within the 3D scene). In the dataset, some (e.g., all) of the conditioning signals for objects may be 2D images of these objects stored in a database (e.g., first retrieved using a reference to the object). Each 2D image included in the training sample may be captured for a respective (i.e., different) 3D scene. Alternatively, the dataset may include training samples including 2D images of the same 3D scene captured, for example, from different viewpoints and / or with different lighting. The dataset may also include, for each training sample, information indicating the layout of the 3D scene that the 2D image presents and its viewpoint (e.g., a table with rows containing the 2D image reference, a reference to the corresponding 3D scene layout, and the coordinates of the viewpoint of the 2D image, respectively).

[0044] At least some of the training samples of the dataset (e.g., more than 50 or 75% of the training samples) include conditioning signals for 3D scenes and / or for objects within the 3D scenes. The conditioning signals of at least some of the training samples include at least one first conditioning signal having a first type from a predetermined set of at least two types and / or at least one second conditioning signal having a second type from the predetermined set of at least two types. For example, the dataset may include at least one conditioning signal of a text type and at least one conditioning signal of an image type for most of the training samples (e.g., more than 75% or more than 80% of the training samples), which allows the function to be trained to take both the first type and the second type into account. Alternatively, the dataset may include conditioning signals of only the same type (text or image). In that case, to train the function to take into account both the first and second types, the machine learning method may include generating the other type of conditioning signal before training (e.g., using an image or text generator as described below), or alternatively, the scene encoder may comprise a multi-modal encoder (such as CLIP) that can take another type of conditioning signal as input.

[0045] The 2D images included in the training samples of the dataset may be realistic 2D images of the 3D scene created prior to execution (e.g., by a designer) of the method. These 2D images may include, for example, perspective, occlusion, and / or lighting factors. To achieve such rendering, the 2D images of the 3D scene in the dataset may have been manually reworked by the designer (e.g., at least partially in places where rendering is difficult due to perspective, occlusion, and / or lighting factors).

[0046] The dataset may be stored in a database. Obtaining the dataset may include retrieving the dataset from the database. Obtaining the dataset may then include storing the retrieved dataset in a memory. After recording, the machine learning method may perform training of the function based on the recorded dataset. Alternatively, obtaining the dataset may include providing access to the dataset in a database. In that case, the machine learning method may use this access to perform training of the function.

[0047] The rooms represented by the 3D scenes in the dataset may or may exist in the real world (either already at the time of acquisition of the dataset or in the future). For example, the rooms may be actual, real rooms in the real world (in terms of layout), and objects may be placed inside these real rooms as specified in the layout of the 3D scenes contained in the dataset. The 3D scenes may represent rooms designed (e.g., by an interior designer) and then implemented in the real world (i.e., the multiple 3D scenes correspond to virtually designed rooms that have been or could be recreated in people's homes). In an example, each room represented in the dataset is of the same type. For example, all rooms represented in the dataset and the 3D scenes may be kitchens, bathrooms, bedrooms, living rooms, garages, laundry rooms, attics, offices (e.g., private or shared), conference rooms, children's rooms, nurseries, hallways, dining rooms, or libraries (this list may include other types of rooms). In that case, the layout obtained during execution of the generation method may be a layout of a 3D scene of the same type as that in the dataset, which allows for the generation of realistic 2D images and improves the stability of the generation method. Alternatively, the dataset may include a variety of room types, in which case the output domain of the generative image model may be large, the number of rooms represented in the dataset may be large, and the training time of the generative image model may also be long.

[0048] In an example, the layout for each 3D scene may include a set of bounding boxes representing objects in the 3D scene. Each bounding box may be rectangular in space and may encapsulate the outer envelope of the object it represents. The layout may include, for each bounding box, parameters representing the position, size, and orientation within the 3D scene of the object represented by the bounding box. For example, the layout may include, for each bounding box, parameters representing the position of the bounding box (e.g., coordinates of the corners or center of the bounding box), parameters representing the size of the bounding box (e.g., width, depth, and height of the bounding box), and parameters representing the orientation of the bounding box (e.g., rotation about each axis of the global reference frame). Optionally, the layout may include, for each bounding box, parameters representing the class of the object represented by the bounding box. The object classes may be predetermined or may correspond to the types of objects they represent. The object classes may be the decorative object and functional object types described above.

[0049] Each 3D scene layout may also include a boundary of the 3D scene. For example, the boundary of the 3D scene may be represented by a respective set of points, e.g., corresponding to corners of the 3D scene or sampled along a wall of the 3D scene. The 3D scene layout may include coordinates of the respective set of points.

[0050] Training the function may include training (e.g., in a supervised manner) a scene encoder and a generative image model to generate 2D images of the dataset when taking as input corresponding layouts and viewpoints included in the dataset. For example, the scene encoder and the generative image model may each include respective parameters (e.g., weights), and supervised training may consist of determining values ​​for these respective parameters so as to favorably reproduce the 2D images of the dataset when taking as input corresponding layouts and viewpoints included in the dataset. Supervised training of the function may include training the scene encoder and the generative image model together (i.e., it may determine their respective parameters simultaneously or in the same process).

[0051] In an example, the machine learning method may further include replacing a predetermined portion of the conditioning signal of the dataset with a conditioning signal having a predetermined value before or during training. This predetermined value may represent a null value (i.e., a value used when there is no conditioning signal). For example, at each iteration of training, the replacing may include determining (e.g., with a given probability) whether to retain or drop the conditioning signal of the training sample. Alternatively, the decision to retain or drop the conditioning signal may be performed before training. Performing the replacement before training would simply result in always the same conditioning settings for each sample in the dataset, so performing the replacement at each iteration improves variability (and therefore the accuracy of the trained function).

[0052] In an example, a machine learning method may first include enhancing an initial dataset by adding conditioning signals for some of the objects represented in the 3D scene of the dataset (i.e., the added conditioning signals are included in the dataset before training). The machine learning function may determine conditioning signals for certain parts (e.g., all) of the objects that are missing conditioning signals in the initial dataset. This allows for enhancing the acquired dataset and thus improving the training of the function based on this enhanced dataset.

[0053] For example, the determined conditioning signal may be of an image type or estimated from a 2D image of the 3D scene. Obtaining the position data of the dataset may include determining at least one conditioning signal for the object from the 2D image of the training sample. The machine learning method may determine a respective conditioning signal for one or more objects included in the 3D scene represented in the dataset. Determining the conditioning signal for the object may be performed by determining a portion of the 2D image of the training sample that includes the object (i.e., by cropping the representation of the object in the 2D image). Determining the portion of the 2D image that includes the object may be performed in any manner, for example, using a detection algorithm (e.g., based on the position of the object in the imaged 3D scene), by reprojecting a 3D bounding box from the annotation to crop the image, or by using an object segmentation mask, for example, from the annotation. Determining the conditioning signal may then include adding this determined portion of the 2D image as the conditioning signal for this object. This operation may be repeated for each object for which a conditioning signal has been determined.

[0054] Alternatively or additionally, the determined conditioning signal may be of an image type. In that case, obtaining the alignment data of the dataset may include determining at least one conditioning signal of an image type by applying an image generator. The image generator may be any type of model configured to take text as input and generate an image for the input text. The determined at least one conditioning signal of an image type may be for one or more objects and / or for one or more 3D scenes. To determine a conditioning signal for an object, the image generator may take as input a text description of the object (e.g., which is also a conditioning signal for this object or another image of this object stored in a database). For determining a conditioning signal for a 3D scene, the image generator may take as input a text description included in a corresponding training sample. The determined at least one conditioning signal may be an image output by the image generator.

[0055] Alternatively or additionally, the determined conditioning signal may be of a text type. In that case, obtaining the alignment data of the dataset may include determining at least one conditioning signal of a text type by applying a text generator. The text generator may be any type of model configured to take an image as input and generate a text description for the input image. The determined at least one conditioning signal of a text type may be for one or more objects and / or for one or more 3D scenes. For determining a conditioning signal for an object, the text generator may take an image of the object as input (e.g., this is also a conditioning signal for this object or another image of this object stored in a database). For determining a conditioning signal for a 3D scene, the text generator may take a 2D image included in the corresponding training sample as input. The determined at least one conditioning signal may be a text description output by the text generator. The text description of an object may also be derived from its associated category (i.e., class), as described above.

[0056] In an example, the machine learning may also include enhancing the dataset by modifying an image-type conditioning signal. For example, when the first type is an image type, acquiring may include modifying at least a portion of at least one first conditioning signal (the at least one first conditioning signal being of an image type). Modifying each of at least a portion of the at least one first conditioning signal may be performed in any manner, for example, by applying one or more random data-enhancing transformations to the image (such as a slight rotation and / or a horizontal flip). Each modified conditioning signal may be determined from a 2D image of the represented 3D scene, as described above. The function may then be trained taking into account each modified first conditioning signal (i.e., using the modified first conditioning signal instead of the original signal). Modifying the image-type conditioning signal allows for enhancing the acquired dataset, thus improving the training of the function.

[0057] In an example, the scene encoder may include a layout encoder configured to encode a set of bounding boxes. For each of the set of bounding boxes (e.g., visible or invisible from a viewpoint), the layout encoder may take as input parameters that describe the position, size, and orientation within the 3D scene of the object represented by the bounding box. Optionally, for each bounding box, the layout encoder may further take as input parameters that describe the class of the object represented by the bounding box. These parameters may be included in the layout of the 3D scene, as described above. The layout encoder may estimate these parameters from the layout taken as input by a function. The layout encoder may be configured to output a vector (hereinafter referred to as a "layout vector") that embeds the parameters.

[0058] For example, the layout encoder may include a positional encoding module that takes bounding box parameters as input and outputs a layout vector. The positional encoding module may be configured to deterministically increase the dimensionality of the scalar values ​​of the parameters taken as input. For example, the positional encoding module may be configured to output, for each bounding box, a position vector representing the position and size of the bounding box and a direction vector representing the orientation of the bounding box. Optionally, the layout encoder may further include a first multi-layer perceptron encoder configured to increase the dimensionality of the direction vectors output by the positional encoding module. The layout encoder may also include a concatenation layer configured to concatenate the vectors output by the positional encoding module and / or the first multi-layer perceptron encoder and output the layout vector.

[0059] In an example, the scene encoder may include a multimodal encoder. The multimodal encoder may be configured to project each conditioning signal into a single latent space. The multimodal encoder may be any model, such as a contrastive language-image pre-training (CLIP) model, configured to project several types of conditioning signals into a single latent space. When the layout data includes one or more conditioning signals for each object represented by at least one of the bounding boxes, the layout encoder may further take as input, for each bounding box of at least the portion of the bounding boxes, a projection of each conditioning signal for the object represented by the bounding box. In other words, the projection of one or more conditioning signals for the objects of the 3D scene may be included in the parameters taken as input by the layout encoder. In that case, the layout encoder may further include a second multilayer perceptron that takes as input the projection of the conditioning signal for the object and outputs a conditioning signal vector. The concatenation layer of the layout encoder may be configured to concatenate the output conditioning signal vector with other vectors output by the position encoding module and / or the first multi-layer perceptron encoder.

[0060] In an example, the scene encoder may further include a floor encoder configured to encode a boundary of the 3D scene. As described above, the boundary of the 3D scene may be represented by a respective set of points, and the floor encoder may take the respective sets of points as input and output a floor vector. For example, the floor encoder may include a PointNet model configured to encode the respective sets of points (e.g., as described in the paper by Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, "Pointnet: Deep learning on point sets for 3D classification and segmentation," CVPR 2017), and optionally a multi-layer perceptron configured to take the output of the PointNet model as input.

[0061] In an example, the scene encoder may further include a camera encoder configured to encode a viewpoint. For example, the viewpoint may include parameters of a camera from which the 2D image is generated (e.g., camera position, field of view, and pitch). The scene encoder may be configured to take these camera parameters as inputs and output a camera vector. For example, the scene encoder may comprise a position encoder configured to take the camera parameters as inputs and, optionally, a multi-layer perceptron.

[0062] In the example, for each given 2D image of a given 3D scene in the dataset, the size and position of an object represented by a bounding box in the layout of the given 3D scene are defined in a coordinate system based on the position and orientation of the camera from which the given 2D image is captured. Therefore, the camera position is already encoded in the layout vector output by the layout encoder. Therefore, the viewpoint can include only two scalar values ​​representing the field of view and pitch of the camera, respectively. This allows for a reduction in the number of learned parameters, improving robustness and thus facilitating convergence of the function.

[0063] In an example, the scene encoder may further include a transformer encoder. The transformer encoder takes as input a concatenation of the set of bounding boxes encoded by the layout encoder (i.e., layout vector), the viewpoints encoded by the camera encoder (i.e., camera vector), and the boundary of the 3D scene encoded by the floor encoder (i.e., floor vector). When the configuration data includes one or more conditioning signals for the 3D scene, projections of the one or more conditioning signals for the 3D scene by the multimodal encoder may be included in the concatenation taken as input by the transformer encoder. The transformer encoder outputs a scene encoding tensor. For example, the transformer encoder may comprise a transformer model configured to take as input a vector to form a sequence of tokens represented as a tensor (the scene encoding tensor). The vector taken as input by the transformer model is the concatenation of all the vectors mentioned above (i.e., layout vector, camera vector, floor vector, and optionally projection of one or more conditioning signals for the 3D scene), optionally padded (or supplemented) with one or more "zero" tokens to form a vector of a fixed (e.g., predetermined) size.

[0064] The generative image model may be any generative image model capable of generating 2D images conditioned with an output scene encoding tensor. The generative image model may be a type of deep neural network trained on a large image dataset to learn the underlying distribution of training images. By sampling from the learned distribution, such a model may be configured to create novel images with properties from those in the training dataset. Examples of generative image models for generating 2D images conditioned with an output scene encoding tensor include generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models.

[0065] In an example, the generative image model may be a diffusion model. The diffusion model may be configured to generate an output 2D image by iteratively removing noise from an initial noisy image based on a scene encoding tensor output by a scene encoder. Examples of diffusion models include a cascade model or a latent diffusion model. A cascade model is a model that includes several diffusion modules, such as a diffusion module for outputting an image conditioned with a scene encoding tensor, and a super-resolution model that then upscales the image to a higher resolution. During inference, the diffusion model may generate an output 2D image by iteratively removing noise from the initial noisy image. Each iteration of noise removal may include determining a new version of the initial noisy image that is less noisy than a previous version of the initial noisy image determined during a previous iteration. Determining the new version may be based on a prediction of the noise of the previous version.

[0066] We now describe training of a diffusion model. The training of the diffusion model may be based on generated noisy versions of the 2D images included in the dataset. For example, a machine learning method may include generating noisy versions of the 2D images of the dataset (by adding noise to these 2D images), and the training of the diffusion model may be based on the generated noisy versions of the 2D images. The diffusion model may be trained to remove noise added to the 2D images of the dataset by taking into account the scene encoding tensor output by the scene encoder. In that case, the training of the diffusion model and the scene encoder may take into account a training loss that penalizes the distance between the predicted noise in the generated noisy version and the actual noise in the generated noisy version.

[0067] In an example, the diffusion model may have an architecture that includes a denoiser. The denoiser may comprise several blocks configured to create a generated 2D image. At least one of these blocks may be enhanced by cross-attention using a scene encoding tensor. The cross-attention mechanism may be an attention mechanism applied between elements from various sequences. The cross-attention may be applied between the representations returned by the transformer encoder for various tokens and the visual features calculated within the denoiser. This improves the function's ability to learn the visual and spatial dependencies / relationships that exist between the scene features (encoded by the transformer encoder) and their visual representations in the image (created by the denoiser). Therefore, this mechanism is particularly suitable for generating 2D images of 3D scenes, where the spatial location / concept of objects within the 2D image is of utmost importance.

[0068] In an example, the diffusion model may be configured to operate in a latent space (i.e., it may be a latent diffusion model). In that case, during training, the diffusion model may be trained to denoise compressed latent representations of 2D images of the dataset. In that case, training the function may include compressing 2D images of the dataset in a latent space (e.g., one with small dimensions), thereby obtaining compressed latent representations. Training may be performed based on these compressed latent representations (rather than the 2D images directly). During inference, the diffusion model may take as input, instead of the initial noisy tensor, a compressed initial noisy tensor of the same dimension as the compressed latent representation. The diffusion model may iteratively remove noise from this compressed initial noisy tensor, given a scene encoding tensor output by a scene encoder. Decompression may then be applied to the result to obtain the generated 2D image. An example of such a compression / decompression implementation includes a variational autoencoder (VAE).

[0069] Applying the function will now be described. Applying the function may first include forming an input for the diffusion model. When the diffusion model does not operate in latent space, this step may include, for example, sampling an initial noisy tensor having the shape of the 2D image to be generated. This initial noisy tensor may be taken as an input by the diffusion model. When the diffusion model operates in latent space, this step may include sampling a compressed initial noisy tensor (i.e., having the same dimensions as the compressed latent representation). This compressed initial noisy tensor may be taken as an input by the diffusion model.

[0070] Applying the function may then include applying a scene encoder to the obtained layout, thereby outputting a scene encoding tensor. Applying the function may then include using a diffusion model conditioned on the output scene encoding tensor to generate a 2D image of the 3D scene. The diffusion model may iteratively remove noise from the sampled initial noisy tensor when not operating in latent space or on an otherwise compressed representation thereof. When operating in latent space, applying the function may further include decompressing the clean latent (obtained by iteratively applying the denoiser) to return to image space and obtain the generated 2D image.

[0071] Here, implementation forms of the generation method and machine learning method will be described with reference to FIGS.

[0072] The trained function is conditioned (i.e., 3D-aware) on the output scene encoding tensor. This allows for the creation of accurate and natural-looking immersive content, thanks to the exploitation of knowledge about the 3D structure and relationships of objects in a scene or environment and, therefore, improved consideration of, for example, perspective, occlusion, or lighting factors. In particular, the trained function includes only a scene encoder and a single conditional diffusion model specifically trained end-to-end for this task. It does not utilize large-scale, general-purpose pre-trained image synthesis priors, does not involve several training stages to train modules separately, and does not require training a neural volume renderer, or NeRF, for each generated scene. Because 3D-awareness is built in through the layout of the training samples, it does not rely on a separate depth estimator, which is typically flawed and propagates errors, and does not require a multi-view dataset for training. The generation method also enhances the level of user interaction and controllability.

[0073] The machine learning and generation methods solve the problem of generating high-quality, user-specified 2D views of a 3D environment, where the 3D environment is composed not of pre-created 3D models and textures, but rather of a high-level / abstract description of the environment, i.e., annotated 3D bounding boxes representing elements (or objects) in the scene. The machine learning and generation methods also provide a way to control the 2D view of the 3D environment, specifically the visual aspects of the objects therein. Control may include user text prompts for scene or object descriptions, as well as visual input (e.g., images) to guide the generation.

[0074] To do so, the function is trained to generate an image of the scene when given as input a camera viewpoint (position and rotation), a set of annotated bounding boxes (position, dimensions, and orientation) representing objects in the scene, an arbitrary set of vector representations of several objects to be generated (e.g., obtained from visual or textual representations of the corresponding objects to be generated), the corners of the room that the 3D scene represents (i.e., boundary, shape, or floor plan), and an arbitrary textual or visual input of the view to be generated.

[0075] Machine learning and generative methods solve this technical problem using techniques based on deep learning. The pipeline can be divided into two main stages:

[0076] In the first stage (offline stage), the machine learning method performs supervised training of a function with conditioning dropout. Given a dataset of pairs of 2D images and the corresponding underlying scene annotations (camera viewpoint and field of view, annotated bounding boxes of objects, visible or invisible, presence in the scene and floor corners), the machine learning method involves training a function that includes a deep learning pipeline consisting of: The scene encoder (hereafter referred to as the scene layout encoder) includes: A layout encoder that outputs a vector embedding for each object present in the scene. This object embedding is calculated from the object's position, dimensions, and orientation in 3D. The machine learning method may also include performing custom conditioning dropout on input object semantic embeddings obtained from one or more modalities representing the object. This means that this object semantic embedding has fixed probabilities of being replaced by null tokens during training, and other probabilities calculated, for example, from a textual description or visual representation of the object. A camera encoder that outputs a vector embedding that captures information about the camera. A floor encoder that outputs a vector embedding that captures information about the floor's shape. A multimodal encoder (hereafter also called a global style encoder) that outputs vector embeddings that capture the overall style and semantics of a scene. Again, conditioning dropout is implemented so that the input can be a null token or an embedding calculated from the target image to be generated. A transformer encoder that takes as input the sequence resulting from the concatenation of the aforementioned embeddings and outputs a sequence of novel representations / embeddings. A diffusion model (hereafter also referred to as denoising diffusion model) that takes as input the noisy version of the provided 2D image as well as the scene embedding output by the scene encoder for conditioning. The denoising diffusion model may operate either directly in image space or in latent space, in which case it may include a variational autoencoder (VAE), or it may take a noisy version of the provided latent as input.

[0077] The goal of the supervised training phase is to give the function the ability to reproduce (or generate) 2D reference images given as input the corresponding scene annotations (i.e., layout) of the 3D scene. Conditioning dropout may be performed at several scales of the scene: at the local / object level (using conditioning signals for objects) and at the global / scene level (using conditioning signals for the 3D scene).

[0078] In the second stage (inference stage or online stage, with classifier-free guidance), the trained function may be used to generate 2D images. Meanwhile, by taking user-defined scene annotations as input, a scene embedding tensor is calculated using a layout encoder, a camera encoder, a floor encoder, and a transformer encoder. During training, conditioning dropout is optional for scene annotations. This means that users can choose to provide optional textual or visual guidance for each object and for the entire scene, which can then be used to calculate semantic embeddings. Meanwhile, random Gaussian noise images are sampled (either in image space or latent space, depending on the nature of the diffusion model). The scene embeddings and the images to be denoised are iteratively fed to the trained denoising diffusion model, which outputs 2D images corresponding to the desired scene at the desired viewpoint. Classifier-free guidance may also be implemented during the iterative denoising to better represent the optional user guidance in the generated images.

[0079] The main advantages of machine learning and generative methods include: Scene-level guidance Once trained, the function (or model) can take textual or visual input and guide the generated scene, allowing a lot of control over the generation. Object-level guidance Once trained, the function (or model) can take textual or visual input for each object in the scene to be generated and guide their visual aspects, allowing a lot of control over the generation. Multi-modality and flexibility By leveraging multimodal underlying models to compute object-level and scene-level semantic embeddings, users can interchangeably use textual or visual modalities to condition generation at inference time, even if some of these modalities are not used during training. The use of conditioning dropout during training also gives users the flexibility to either guide the generation process with a specific inspiration or leave it unconditional, maximizing the diversity of generated images.

[0080] We now provide definitions of certain terms.

[0081] Deep neural networks (DNNs) are a powerful set of techniques for learning in neural networks, a biologically inspired programming paradigm that enables computers to learn from observed data. In object recognition, the success of DNNs is attributed to their ability to learn rich mid-level media representations, as opposed to the manually engineered low-level features (e.g., Zernike moments, HOG, bag-of-words, SIFT) used in other methods (e.g., min-cut, SVM, boosting, random forests). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move away from feature engineering as much as possible by achieving end-to-end optimization starting from raw features and ending with labels.

[0082] A generative image model is a type of deep neural network that is trained on a large image dataset to learn the underlying distribution of training images. By sampling from the learned distribution, such a model can create novel images with properties from those in the training dataset. GANs (generative adversarial networks), VAEs (variational autoencoders), and diffusion models are widely recognized as the most common generative image models, with diffusion models currently considered the state-of-the-art in the field.

[0083] Diffusion models are a type of deep learning model that can be used for image generation. They aim to learn the structure of a dataset by modeling how data points diffuse through a latent space. Diffusion models consist of three components: a forward process, a backward process, and a sampling phase. In the forward process, Gaussian noise is added to the training data through a Markov chain. The goal of training a diffusion model is to teach the model how to undo the noise addition step by step. This is done in the backward process, where the diffusion model reverses the noise addition performed in the forward process and thus recovers the data. During the sampling phase, the image-generating diffusion model starts with random Gaussian noise images. After being trained to reverse the diffusion process on images from the training dataset, the model can generate novel images similar to those in the dataset. This is achieved by starting with pure Gaussian noise and reversing the diffusion process until a clear image is obtained.

[0084] An autoencoder is a neural network architecture used for dimensionality reduction and data compression. It consists of an encoder that maps input data to a low-dimensional representation and a decoder that reconstructs the original data from the encoded representation. By compressing and reconstructing data, autoencoders extract meaningful features, enabling tasks such as data compression. A variational autoencoder (VAE) is a special type of autoencoder that incorporates probabilistic modeling. Instead of learning a deterministic mapping, a VAE learns the parameters of a probability distribution over a latent space.

[0085] A Transformer is a type of deep neural network architecture that has a remarkable ability to perceive relationships among elements in an input sequence. Thanks to a mechanism called self-attention, a Transformer allows the model to learn the associations of each element with other elements and appropriately weight the contextual information. The Transformer module takes a sequence as input and outputs a novel vector representation of the input data, where relationships within the input sequence are emphasized.

[0086] Cross-attention extends the self-attention mechanism by allowing correlation or contextual information to be selected between different sequences. The inputs used for cross-attention are two different sequences of the same or different modalities (e.g., text or images). The model learns to attend to relevant information from one of the sequences to the other. Cross-attention is appropriate when dealing with tasks that involve incorporating information from other sources to enhance the model's capabilities.

[0087] In the context of generative AI models for image synthesis, conditioning refers to the process of injecting additional information into the image generation process to obtain results that match user-driven constraints. Conditioning can take a variety of forms, including, for example, text (e.g., DALL-E 2, Midjourney, or Stable Diffusion) or images (e.g., ControlNet or semantic segmentation).

[0088] The bounding box of a three-dimensional (3D) object is the smallest rectangular cuboid that encloses the object. Its position, its dimensions, and its orientation characterize the 3D bounding box.

[0089] A "viewpoint" represents the perspective or "camera" from which a rendering is captured. It may include four components: position, orientation, and field of view and pitch. The position and orientation of the viewpoint and the position and orientation of the bounding box may be defined within a single reference frame.

[0090] The term "3D abstract scene" refers to a list of labeled bounding boxes (the layout of the 3D scene) representing objects in the scene (labels correspond to classes of objects), a viewpoint, and optionally other elements that may enhance the description of the environment (e.g., information about the shape of the room). The adjective "abstract" emphasizes that the objects in the scene have no visual representation and are not defined beyond the properties of their bounding boxes and their labels.

[0091] A scene encoder is a specialized deep neural network that learns to extract a comprehensive representation from a 3D scene, which may include spatially arranged objects, layout, or viewpoints. A scene encoder takes in a variety of inputs depending on its specific architecture and the needs of the user, and produces a high-dimensional vector output. This encoded representation should capture important features of the scene and serve as valuable input for subsequent stages of a deep learning model.

[0092] When training a conditional generative model, such as a diffusion model, conditioning dropout may consist of randomly removing the conditioning signal passed to the model with a null token, indicating the absence of conditioning, according to a dropout rate that is often 10%-20% of the time in practice. Thus, a generative model trained with conditioning dropout should be able to function both conditioned (when conditioning information is passed) and unconditioned (when a null token is passed).

[0093] Diffusion models trained with conditioning dropout may generate new samples for inference using a method called classifier-free guidance (CFG). The CFG method is a method for training conditional diffusion models. It introduces the concept of conditioning dropout: during training, the conditioning signal y (e.g., a text description) is removed for a given percentage of the time (often 10%-20%), and a null vector representing the absence of conditioning information is generated.

number

number

number

number

[0094] Multimodal foundational models (deep multimodal models) learn a shared latent space between several modalities (e.g., text and images) and thus facilitate semantic alignment between various representations. One example is the CLIP model (e.g., as disclosed in the paper "Learning Transferable Visual Models From Natural Language Supervision" by Alec Radford et al., arXiv 2021, incorporated herein by reference), whose encoder projects text and images into a common vector space. The CLIP model is a foundational model, i.e., it has been trained on millions of (text, image) pairs and therefore has a broad latent space for represented concepts, which can be leveraged for various downstream tasks. Thus, the CLIP model can link complex visual concepts with their descriptions in natural language. Embeddings computed by CLIP's text encoder from text prompts have high cosine similarity to those computed by its image encoder from images semantically close to the prompts. CLIP has strong zero-shot capabilities, enabling it to perform several tasks, such as image classification or open-vocabulary semantic segmentation. The shared latent space allows for the interchangeable use of text and image modalities.

[0095] FIG. 1 is a flow chart illustrating an example of a machine learning method and a generation method.

[0096] The pipeline consists of a (latent) diffusion model 100 conditioned by a novel 3D scene encoder 200. Similar to other deep learning models, it features an offline stage S100 (training a function by implementing machine learning methods) and an online stage S200 (generating 2D images by implementing generative methods, also called the inference stage). The training stage is here performed using conditioning dropout on semantic embeddings of objects, computed from different modalities representing the objects, and on global style embeddings of the scene, computed from the target renderings to be generated.

[0097] We now describe the offline training stage S100 in detail. The purpose of this stage is to simultaneously (i) train the scene encoder 200 to create a comprehensive mathematical representation that can be used for conditioning, and (ii) train the diffusion model 100 to generate images from noise. The scene encoder 200 takes as input a set of elements characterizing a 3D scene (the layout of the 3D scene) and outputs a scene encoding tensor. The diffusion model 100 takes as input a noisy version of the image to be generated and the scene encoding tensor, and outputs a denoised version of the input image. This training is end-to-end, i.e., a single loss value is calculated and back-propagated to adjust the weights of both the diffusion model and the scene encoder. The setup of the training stage may include the following subtasks: Data preprocessing steps The data samples of the dataset, particularly the scene annotations (i.e., layout), may be processed so that they can be passed to a scene encoder. The preprocessing stage may optionally include computing object semantic embeddings of the training dataset offline (prior to training experiments), as well as global style embeddings, using, for example, a pre-trained encoder from multimodal base model 250. Computing and storing these embeddings offline helps reduce memory usage and computation during training. ·Architecture definition step The scene encoder 200 may return a single fixed-size tensor embedding for the entire scene being rendered. The diffusion model 100 may take as input a noisy version of the image and a scene encoding vector. It may return an estimate of the noise added to the image and use it to suggest a less noisy version of the input image. Definition of conditioning dropout method Depending on the available modalities representing each object in the dataset, a multinomial probability distribution may be designed to define the probability of selecting a modality for which the object semantic embedding is calculated or a null token, respectively. The probability of selecting a null token is the dropout rate. In each training iteration, the machine learning method may include sampling from this probability distribution independently for each object. Similarly, a separate dropout rate may be set to perform conditioning dropout for the scene-global style embedding. Training loss definition step The training loss function may measure the distance between the predicted noise in the input image and the true noise in the image added through the forward process. Training Steps Training can be performed by iterating over the dataset (image and scene annotation pairs) several times.

[0098] We now describe in detail the image generation / inference stage S200, which aims to output a rendering that matches the viewpoint of an abstract 3D scene. At this stage, the generation method may include the following subtasks: Step to determine scene embedding vectors The scene encoder 200 and the input abstract 3D scene are used to compute a scene embedding vector. The abstract 3D scene does not necessarily have to be part of a database (e.g., it can be created by the user or generated using other techniques). Optionally, object semantic embeddings 204 (conditioning signals for objects), which can be computed from user-provided prompts such as images or text descriptions, can be associated with any object in the scene to guide the generation. Similarly, scene style embeddings 205 (conditioning signals for the 3D scene) can also be optionally passed. Generate a random Gaussian noise image The Gaussian noise image 301 may have the size of the desired final image when training in image space, or the size of the latent space of the VAE when training a latent diffusion model. Iteratively denoising the generated images Using the diffusion model 100, a random Gaussian-noised image 301 is first denoised, and the output of the diffusion model is then iteratively denoised. A U-Net denoiser (the DNN backbone of the diffusion model) can be conditioned with the scene embedding vectors using cross-attention between the layers of the U-Net and the scene embedding vectors. After a fixed number of denoising steps, a final sharp image 302 is produced. Because the diffusion model is trained with conditioning dropout, classifier-free guidance is implemented during denoising to push predictions toward the conditioned model output and away from the unconditioned model output, and thus may better represent the conditioning signal, e.g., the scene style embedding computed from the text description.

[0099] We now describe an example implementation of the general framework described above, focusing on interior scene generation.

[0100] We now present details regarding the acquisition and contents of the dataset used to train the function. The data used may be extracted from HomeByMe® renderings made by users (i.e., 2D images created by actual users). Whenever a high-quality rendering is made in the application, a rich annotation file is saved along with the image. The raw data from this annotation includes information about the rendering (semantic segmentation maps and / or 2D bounding boxes of visible objects) and information about the scene from which the rendering was made (3D bounding boxes of objects, room geometry and / or viewpoint). The dataset also includes object-level annotations, i.e., 3D meshes, object categories, image thumbnails of objects, material data, text annotations describing the objects, etc. From this raw data, three elements can be extracted: Annotated Objects For each object in the scene (not necessarily visible in the rendering captured by the user), the annotation file contains a list of various features that describe the object. In particular, it defines the object's 3D bounding box by two 3D points corresponding to two opposite vertices of the bounding box. This also indicates the object category from a total of 174 possible classes in the HomeByMe dataset. Each object also has a unique reference (also called an id) associated with an image thumbnail (a rendering of the object on a neutral background). Finally, objects visible in the rendering are located by their 2D bounding boxes. Therefore, machine learning methods augment the image representation of 3D objects in HQ renderings by cropping their 2D bounding boxes. ·perspective The user's rendering perspective can be saved in an annotation file. In particular, the camera position, its orientation, and its field of view are captured and used later in the pipeline. Room shape The room geometry is stored in the annotation file as a list of 2D points representing the corners of the room.

[0101] FIG. 2 illustrates an example of the scene encoder 200 of FIG. 1. The scene encoder 200 includes a layout encoder 210 configured to encode a set of bounding boxes 201 and a conditioning signal 204 for the objects represented by the set of bounding boxes 201. For each set of bounding boxes 201, the layout encoder 210 takes as input parameters representing the position, size, and orientation of the object represented by the bounding box within the 3D scene, and, when a conditioning signal is input for the object represented by the bounding box, a projection of this conditioning signal by a multimodal encoder 250 (also referred to as a global style encoder). The scene encoder 200 further includes a floor encoder 230 configured to encode a boundary 203 of the 3D scene. The scene encoder 200 further includes a camera encoder 220 configured to encode a viewpoint 202. As shown in FIG. 1, the scene encoder 200 further includes a transformer encoder 240. The transformer encoder takes as input a concatenation of a set of bounding boxes 201 encoded by the layout encoder 210, a viewpoint 202 encoded by the camera encoder 220, a 3D scene boundary 203 encoded by the floor encoder 230, and a projection of the conditioning signal of the 3D scene 205 by the multimodal encoder 250. The transformer encoder 240 outputs a scene encoding tensor.

[0102] The offline training stage S100 will now be described in detail.

[0103] The machine learning method may include a data processing step prior to the training step to process the layout and viewpoint of each 2D image in the dataset.

[0104] The data processing step may include a first step for processing 3D bounding boxes for each 3D scene. The first step may include converting raw 3D bounding boxes from a representation based on two opposing vertices to a representation with their positions (x, y, z), their dimensions (width w, height h, depth d), and their orientations. Objects present in the scene may have only one rotational degree of freedom, i.e., rotation around a vertical axis. As a result, machine learning methods may use only a single angle θ to define the orientation of the bounding box. In practice, machine learning methods may use a different representation, encoding the orientation of the 3D bounding box by the corresponding pair (cos(θ), sin(θ)). Such a parameterization is mathematically equivalent to the single-value parameterization, but strengthens the continuity of the deep learning model between θ=0 and θ=2π. This is beneficial for model convergence. As a result, the processed 3D bounding box is defined by a list of eight parameters (x, y, z, w, h, d, cos(θ), sin(θ)).

[0105] The data processing step may include a second step for processing the conditioning signals for objects of text type (class or description). Each object from the HomeByMe dataset may be described by a class that provides a broad description (e.g., chair, table, or door). There may be a total of 174 classes in the HomeByMe dataset. The machine learning and / or generation method may include calculating and storing latent embeddings for each of them by applying a text encoder of the trained CLIP model to their string representations (e.g., "nightstand," "chair," etc.). Instead of capturing a string description of one of the 174 categories to which the object belongs, the machine learning and / or generation method may include building a detailed string description by leveraging other available object-level annotations such as material, color, etc., resulting in a rich text description unique to each object (e.g., "wooden single bed"), on which CLIP text embeddings may be calculated.

[0106] The data processing step may include a third step for processing conditioning signals of image-type (visual representation) objects. Similar to text-type conditioning signals, latent embeddings of visual representations of objects may be calculated and stored by applying a trained CLIP model image encoder to thumbnails or in-context crops of the objects. The machine learning and / or generation method may include applying random data augmentation transformations (slight rotations, horizontal flips) to the in-context crops. Other visual representations may also be considered; for example, the machine learning and / or generation method may include storing latent signatures calculated from the point cloud representation of each object by applying a trained point cloud encoder.

[0107] The data processing step may include a third step to process the layout boundaries. The third step may include increasing the dimensions of the floor points. The raw points from the data annotations are 2D points (x,y) because their Z coordinate is implicitly 0. The 2D points are changed to 3D points by using 0 as the Z coordinate. This step is necessary because the 3D points are affected by the transformations described below.

[0108] The data processing step may include a fourth step for processing bounding box coordinates, specifically camera coordinates, from world coordinates (as shown in Figure 3). The raw position and orientation found in the annotation file use the world coordinates defined in HomeByMe. To reduce the number of learned parameters, promote robustness, and thus facilitate convergence, the data processing step may implement a coordinate system change from the original world coordinates to a viewpoint-based coordinate system. In the new coordinate system, the world origin is set to the camera position, and the basis vectors are selected as follows: the "Z" basis vector is unchanged, the "Y" basis vector is the projection of the viewpoint's forward vector onto a plane perpendicular to the "Z" vector, and the "X" vector is perpendicular to the two mentioned above. With this basis change, the viewpoint can be described purely by two scalar values: field of view (FOV) and pitch (the angle its forward vector makes with the "Y" basis vector). This basis change affects the position and rotation of all objects and points in the scene. While optional, this basis change aids in the convergence of the function.

[0109] The data processing step may include a fourth step for processing semantic embeddings of the scenes. The semantic latent embeddings associated with each scene may be computed by applying a trained CLIP image encoder to an HQ rendering of each scene. Machine learning and / or generative methods may also be considered to compute latent embeddings to textual descriptions of entire scenes, but this may be less straightforward to implement.

[0110] In summary, each training scene consists of the following elements: 3D objects. Each object is represented by: ○ A 3D bounding box defined in the camera coordinate system (x, y, z, w, h, d, cos(θ), sin(θ)). Multimodal semantic embeddings computed by: Applying the CLIP text encoder to a string representation of a class of objects. Applying the CLIP image encoder to object thumbnails. Applying CLIP image encoder to the extended crop of objects in HQ rendering of the scene (crop in context). Camera viewpoint parameterized by two scalars (FOV, pitch). 3D floor points of the scene defined in the camera coordinate system. HQ rendered image (target rendering) and its semantic embedding computed by applying the CLIP image encoder.

[0111] We now describe the scene encoder architecture in detail.

[0112] The scene encoder consists of four components: a layout encoder, a camera encoder, a floor encoder, and a transformer module (or transformer encoder) (see Figure 2).

[0113] Next, we describe the layout encoder. The scalar values ​​(x, y, z, w, h, d, cos(θ), sin(θ)) describing each bounding box in the scene may be passed through a position encoding module (PE) that deterministically increases the dimensionality of the scalar values. In this example, the scalar values ​​are

number

[0114] After the position encoding module, the position and dimensions of a bounding box, each originally described by three scalar values, are described by a 192-dimensional vector (3 x 64 = 192). On the other hand, the rotation, originally described by a pair of scalar values, is described by a 128-dimensional vector after position encoding. To ensure that the position, dimensions, and rotation are weighted equally by the model, the high-dimensional version of the rotation is

number

[0115] The object semantic embeddings computed by CLIP's pre-trained text and image encoders are

number

number

[0116] All of the previously calculated vectors are

number

[0117] We now describe the camera encoder. A camera or viewpoint is completely described by two scalar values: field of view and pitch. Both of these values ​​are encoded in a high-dimensional space using position encoding.

number

number

[0118] We now describe a floor encoder. The floor is represented solely by an unordered set of 3D points corresponding to its corners. Such a representation is ambiguous and cannot be easily interpreted by a deep neural network. Alternatively, the data processing step may involve densely sampling points along the walls of the room, thereby generating a set of sampled points along the boundary, so that the room boundary is represented by a 3D point cloud. This 3D point cloud is then

number

number

[0119] We now describe a multimodal encoder (or global style encoder). Similar to object semantic embeddings, a global style embedding of a scene obtained by applying a trained CLIP image encoder to a target HQ rendering is

number

number

[0120] We now describe the Transformer module. 3D object tokens, camera tokens, floor tokens, and scene semantic tokens are all concatenated to form a sequence of tokens. These tokens are independent of each other. To capture the relationships between the various elements of this sequence, the Transformer module is used. Due to its inherent architecture, the operation of the Transformer module can be improved using a fixed input size. However, the sequence constructed through the concatenation of the outputs of the layout encoder, camera encoder, floor encoder, and multimodal encoder can have variable length since the number of 3D bounding boxes in a scene can vary from scene to scene. To be compatible with the Transformer architecture, the concatenation of vector sequences is performed using a "zero" token so that the sequence is of fixed length.

number

number

[0121] We now describe the architecture of the diffusion model. The function may include one of two versions of the diffusion model: one that operates directly in image space, or one that operates in the latent space of a pre-trained VAE to increase the final image dimensions. In the first case, diffusion occurs directly on pixels from the image, while in the second case, diffusion occurs on a latent version of the image, which is then decoded using a VAE decoder. The two approaches are not fundamentally different and do not require significant changes beyond the introduction of the VAE.

[0122] The diffusion model has a conditional generation architecture featuring a U-Net backbone with four down blocks and four up blocks. In particular, the last two down blocks and the first two up blocks can be augmented with cross-attention using scene embedding vectors. The number of up / down blocks and the number of blocks augmented with cross-attention can be varied depending on the needs and user resources. This configuration provides a good compromise between image quality and training time.

[0123] We now describe the definition of our conditioning dropout scheme. In particular, we first describe conditioning dropout for object semantic embeddings. For each object and at each training iteration, the machine learning method involves drawing an object semantic embedding that is passed to the layout encoder (see Figure 2) according to the following probability law: CLIP encoding calculated from the string representation of the object's class. P=0.1. CLIP encoding calculated from object thumbnails. P=0.3 CLIP encoding calculated from the crop during extended rendering of the object. Null token. Machine learning methods use the "zero" token to indicate no object semantic conditioning.

number

[0124] Other probabilities may be set, however, this redistribution allows the architecture to correctly interpret the CLIP latent space while also providing good results, especially when no object semantic conditioning is provided.

[0125] Next, we describe conditioning dropout in the global style embedding of a scene. At each training iteration, the global style embedding computed by applying the CLIP encoder to the target rendering is conditioned to a dropout rate of 0.2, i.e., "zero" tokens.

number

[0126] We now describe the training loss used in training. Diffusion models can be trained using a variety of losses / parameterizations. As introduced in the paper "Denoising Diffusion Probabilistic Models" by Jonathan Ho et al., NeurIPS 2020, which is incorporated herein by reference, at each training iteration, and for each training image, the time step t is uniformly sampled t~U({1,...,T}) and a distribution schedule β is applied. t Gaussian noise is added to the image according to the following: The diffusion model tries to predict the noise added to the image. The loss used is the true noise ε and the predicted noise ε θ where θ indicates that the prediction is based on the parameters of the model.

[0127]

number

[0128] Alternatively, other commonly used diffusion training parameterizations / losses may be utilized interchangeably, for example, the v-prediction parameterization with a min-SNR weighting value of 5.0 provides a good image quality / resolution / computation tradeoff (e.g., as described in the paper "Efficient Diffusion Training via Min-SNR Weighting Strategy" by Tiankai Hang et al., ICCV 2023).

[0129] The generation / inference stage S200 will now be described in detail. The diffusion model may be configured during inference using various techniques. For example, two different sampling processes may be used: denoising diffusion probabilistic model (DDPM) and denoising diffusion implicit model (DDIM). DDIM, for example, may provide a good balance between inference speed and image quality. When inferring an image of a given 3D abstract scene, generation may take approximately 1 second on an NVIDIA RTX A6000 GPU.

[0130] To better represent the input semantic embedding signal, inference may be performed using classifier-free guidance. The CFG strength may be adjusted by adjusting the guidance scale scalar s (s≧1 means that the conditioning signal is amplified).

[0131] The generation method may involve using a typical implementation of a CFG, i.e., a trained model p θ is a conditioning signal y that includes one or more object semantic embeddings and / or one global style embedding. s and the conditioning signal y where object semantics and global style conditioning are null vectors. u You can create views from both p θ (x|y s ) the noise ε predicted at each sampling step s , and pθ (x|y u ) predicted noise ε u Instead of using one of the two predictions for denoising, another noise ε is used, i.e., ε = ε u +s(ε s- ε u ) Effectively, this implementation reduces the predicted conditional noise ε s may be used to boost the conditioning signal by pushing the denoising process towards regions where

[0132] Examples of results are now described with reference to Figures 4 to 14. In particular, we present qualitative results obtained with a generative method using a 3D-enabled rendering generative model with multimodal and multilevel semantic control.

[0133] The following example illustrates the control of generated images using conditioning signals for a 3D scene. In particular, conditioning signals for a 3D scene (i.e., global input style) can be provided through two different modalities: image and text.

[0134] Figures 4-6 show examples of the use of conditioning signals for image-type 3D scenes. In particular, these figures include various examples of bedroom 401 images generated from various inspiration images 402. The images shown are those used for global style guidance. These examples demonstrate the model's ability to capture semantic and stylistic information from input images and use it as inspiration, while taking into account the input 3D layout. In particular, it is interesting to note that even large deviations from the training distribution can be understood and serve as inspiration for image generation. This is made possible by the highly general CLIP embedding space.

[0135] Figures 7-10 show examples of the use of conditioning signals for text-based 3D scenes. In particular, these figures show various examples of bedrooms generated using various text descriptions as global conditioning.

[0136] These examples qualitatively demonstrate the proposed architecture's ability to generate views of an input 3D scene in a style provided through text prompts. Specifically, Figures 9 and 10 show images generated for the same input 3D scene and viewpoint, but using various conditioning signals. Not only are standard descriptions, such as those provided with color, rendered extremely well in the output images. This achievement is made possible by the shared space between the text and images that make up the CLIP embedding. Figure 11 shows a comparison of global style conditioning. This figure shows images generated from the same 3D layout, but with various conditioning modalities.

[0137] An example is now presented in which a conditioning signal for an object is used to control the generated image.

[0138] Figures 12 and 13 show several examples of object-level control achieved with both text 411 and image 412 inputs. These examples demonstrate the model's ability to, first, allow precise control over an object's semantics (what the object is) and, second, its style (how instances of that object look). While style control primarily affects a specific object, the input style most often leaks through to other objects that may be affected by another object's input style. Such an observation is not a drawback in itself, as it simply means that the model attempts to maintain a coherent style within a scene. This is most likely due to the scene tokenizer's transformer module, which captures correlations between tokens representing individual objects.

[0139] Here, we present a quantitative evaluation metric, specifically one that quantitatively measures the effect of global style conditioning on the generated images.

[0140] CLIP embeddings of an image convey information about its style and semantics through vector representations. In the CLIP embedding space, images or text that share similar style and / or content have high cosine similarity (S C Therefore, the impact of global conditioning can be assessed by analyzing the extent to which this conditioning leads the global style guide image to closely match the style / semantics of the input conditioning image, as opposed to the generation of unconditioned images. To conduct this study, follow the following protocol:

[0141] For each scene in the evaluation set, -generate unconditional and conditional images of the scene; - Calculate the CLIP embeddings of both generated images, - Calculate the cosine similarity Sc between each calculated CLIP embedding and the conditioning signal (which can be text or image) used to generate the conditional, thus Sc(CLIP conditionnedGeneration ,CLIP conditionalSignal ) and Sc(CLIP unconditionnedGeneration ,CLIP conditionalSignal ) and - calculating a ratio between the cosine similarity calculated between the CLIP embedding of the conditional image and the conditioned signal and the cosine similarity calculated between the CLIP embedding of the unconditional image and the conditioned signal;

[0142]

number

[0143] Since the objects in the images are likely to remain unchanged (the same layout is used to generate both), the variation between the two generated images is mostly due to their style. Under these conditions, the cosine similarity S c A higher r indicates that the image's style is stylistically closer to the conditioning image. Therefore, for global style conditioning to be quantitatively successful, we expect the obtained ratio to be greater than 1.

[0144] The results obtained from evaluating 4000 scenes are presented in Figure 14. For global conditioning, we utilized a combination of randomly generated captions ("A*color*bedroom.") and selected inspiration styles (HomeByMe renderings, real-life photos, non-interior design-related photos).

[0145] These results demonstrate that global style conditioning indeed positively influences the style of an image (ratio greater than 1) by bringing its style (characterized by the CLIP embedding) closer to the input style. These results also show that text and image conditioning are not precisely equivalent, with the signal being much stronger with text guidance than with image guidance (average ratio of 1.08 with image guidance vs. 1.25 with text guidance). Such results can be explained by the fact that the text captions used for conditioning are extremely short, with a single image token in the vision transformer (used in CLIP) being equivalent to a long sentence as described in the original Vision Transformer paper by Ryan Po et al., "Compositional 3D Scene Generation using Locally Conditioned Diffusion," arXiv 2023, incorporated herein by reference.

[0146] The same observation can be made for object-level conditioning signals.

[0147] FIG. 15 shows an example of a system, where the system is a client computer system, for example a user's workstation.

[0148] The client computer in this example also includes a central processing unit (CPU) 1010 connected to an internal communication BUS 1000, and a random access memory (RAM) 1070 also connected to the BUS. The client computer is further provided with a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and the like. Any of the foregoing may be supplemented by or incorporated in specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a keyboard, or other haptic device 1090. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. The cursor control device also allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse, and the buttons on the mouse are used to generate the signals. Alternatively or additionally, the client computer system may be equipped with a sensitive pad and / or a sensitive screen.

[0149] A computer program may include computer-executable instructions, which include means for causing the system to perform a method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, for example, an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions that performs the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation or update program. Applying the program to a system provides, in either case, instructions for performing the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment, the server being in communication with one or more clients over a network, in which case the processing unit executes the instructions contained in the program, whereby the method is performed in the cloud computing environment.

Claims

1. 1. A computer-implemented method for generating a 2D image of a 3D scene, comprising: configuration data comprising a layout of the 3D scene and at least one conditioning signal, each conditioning signal having one type from a predetermined set of at least two types; ○ The viewpoint of the 3D scene and obtaining applying a machine learning function to the captured geometry data and viewpoint, the function comprising a scene encoder and a generative image model, the scene encoder taking as input the captured geometry data and viewpoint and outputting a scene encoding tensor, the generative image model taking as input the scene encoding tensor output by the scene encoder and outputting the generated 2D image; 1. A computer-implemented method for generating a 2D image of a 3D scene, comprising:

2. The method of claim 1 , wherein obtaining the configuration data comprises selecting, upon user interaction, the type of each conditioning signal from among a predetermined set of the at least two types.

3. The method of claim 1 or 2, wherein the predetermined set of at least two types includes an image type and a text type.

4. the layout of the 3D scene includes bounding boxes each representing a respective object in the 3D scene, and the at least one conditioning signal comprises: one or more conditioning signals for the 3D scene; and / or A method according to claim 1 , further comprising one or more conditioning signals for each of the objects represented by at least one of the portions of the bounding boxes.

5. The method of claim 4 , wherein obtaining the positioning data comprises selecting the one or more conditioning signals for the 3D scene upon user interaction.

6. 6. The method of claim 4, wherein obtaining the positioning data comprises, for each given bounding box of the at least some of the bounding boxes, selecting, upon user interaction, one or more respective conditioning signals for the object representing the given bounding box.

7. The method of claim 1 , wherein the scene encoder comprises a multi-modal encoder, the multi-modal encoder configured to project each conditioning signal into a single latent space.

8. 8. A computer-implemented method for machine learning a function for use in the generation method of any one of claims 1 to 7, comprising: acquiring a dataset including training samples each including a 2D image, configuration data, and viewpoints, wherein the configuration data of at least some of the training samples include conditioning signals, and the conditioning signals of the at least some of the training samples include at least one first conditioning signal having a first type from a predetermined set of at least two types and / or at least one second conditioning signal having a second type from the predetermined set of at least two types; training the function based on the acquired data set; A computer-implemented method for machine learning a function used in the generation method of any one of claims 1 to 7, comprising:

9. 9. The machine learning method of claim 8, further comprising replacing a predetermined portion of the conditioning signals of the dataset with conditioning signals having predetermined values ​​before or during the training step.

10. 10. The machine learning method of claim 8 or 9, wherein the step of obtaining configuration data comprises determining at least one conditioning signal for an object from the 2D images of the training samples.

11. 11. The machine learning method of claim 8, wherein the first type is an image type, and the obtaining step includes modifying at least a portion of the at least one first conditioning signal, and the function is trained taking into account each modified first conditioning signal.

12. the first type is an image type and the obtaining step comprises determining at least one first conditioning signal by applying an image generator; and / or 12. The machine learning method of claim 8, wherein the second type is a text type and the obtaining step comprises determining at least one second conditioning signal by applying a text generator.

13. 13. A computer program comprising instructions, which when executed by a computer, cause the computer to perform the generation method of any one of claims 1 to 7 and / or the machine learning method of any one of claims 8 to 12.

14. A computer-readable storage medium having the computer program according to claim 13 recorded thereon.

15. 14. A system comprising a processor coupled to a memory, the memory having recorded thereon the computer program of claim 13.

Images