Generating a plurality of 3D images of a 2d scene

Abstract

To provide a computer-implemented method for generating a plurality of 3D images of a 2D scene.SOLUTION: The method comprises obtaining placement data and a machine-learning model for generating 2D images, generating a plurality of first 2D images of a 3D scene each having a respective viewpoint, calculating a first latent vector for each generated first 2D image, and calculating a second latent vector as a weighted combination of the calculated first latent vectors for each generated first 2D image; For each given viewpoint of the first 2D image, a plurality of second 3D images of the 2D scene is generated by applying the model to the computed second latent vector, forming an improved solution for generating the plurality of 3D images of the given 2D scene.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems and programs for generating multiple 2D images of a 3D scene. [Background technology]

[0002] Numerous systems and programs are available on the market for designing, engineering, and manufacturing objects. CAD is an acronym for Computer-Aided Design, which relates to software solutions for designing objects, for example. CAE is an acronym for Computer-Aided Engineering, which relates to software solutions for simulating the physical behavior of future products, for example. CAM is an acronym for Computer-Aided Manufacturing, which relates to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, the graphical user interface plays a key role in the efficiency of the technology. These technologies can be incorporated into product lifecycle management (PLM) systems. PLM refers to a business strategy that supports sharing enterprise product data, applying common processes, and leveraging enterprise knowledge to develop products from conception to the end of their lifespan across the extended enterprise. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA, 3DVIA, and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to both the Engineering and Manufacturing Hubs. Overall, the systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this context, applications for 3D scene creation have been developed. These applications generally propose to create, manipulate, and present 3D scenes, especially (though not exclusively) for touch-sensitive devices (e.g., smartphones or tablets). One task of these applications is the generation of realistic 2D images of the 3D scene.

[0004] Solutions for generating 2D images of 3D scenes have been developed in recent years, for example, using generative deep learning models. However, these solutions do not fully consider the entire 3D environment of the imaged scene. In particular, these solutions cannot leverage knowledge of the 3D structure and relationships of objects in the scene or environment. In particular, they cannot account for, for example, perspective, occlusion, or lighting factors, and therefore cannot create accurate and natural immersive content. Furthermore, current solutions cannot generate multiple such realistic 2D images of a given 3D scene that are visually and functionally consistent across various viewpoints.

[0005] In this context, there remains a need for improved solutions for generating multiple 2D images of a 3D scene. Summary of the Invention

[0006] Accordingly, a computer-implemented method for generating multiple 2D images of a 3D scene (hereinafter referred to as the generation method) is provided. The method includes obtaining geometry data including a layout of the 3D scene and a machine learning model configured to generate the 2D images. The model takes as input a viewpoint, a 3D scene layout, and a latent vector. The model includes a scene encoder and a generative image model. The scene encoder takes as input the 3D scene layout, the viewpoint, and the latent vector, and outputs a scene encoding tensor. The generative image model takes as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. The method includes generating multiple first 2D images of the 3D scene, each having a respective viewpoint, by applying the model to the viewpoint, the 3D scene layout, and the respective initial latent vector of the first 2D image, for each first 2D image. The method includes calculating a first latent vector for each generated first 2D image by applying a global projector to the generated first 2D image. The method includes calculating, for each generated first 2D image, a second latent vector as a weighted combination of the calculated first latent vectors. The method includes, for each given viewpoint of the first 2D image, generating a plurality of second 2D images of the 3D scene by applying a model to the given viewpoint, the layout of the 3D scene, and the calculated second latent vectors.

[0007] The method of production may include one or more of the following: Calculating the second latent vector includes calculating a viewpoint overlap measure for each pair of the first 2D images, wherein the second latent vector is a weighted combination of the first latent vector with the viewpoint overlap measure. a viewpoint overlap measure for each pair i,j of the first 2D image is calculated based on the following formula:

number

[0008] Also provided is a computer-implemented method for machine learning a model for use in the generating method (hereinafter referred to as the machine learning method). The machine learning method includes obtaining a dataset including training samples each including a 2D image, a layout, a viewpoint, and a respective latent vector. The machine learning method includes training a model based on the obtained dataset.

[0009] The machine learning methods may include one or more of the following: The machine learning method further includes, before or during training, replacing a predetermined portion of each latent vector of the dataset with a latent vector having a predetermined value; and / or The machine learning method further includes training a global projector simultaneously with training the model.

[0010] There is also provided a computer program comprising instructions, which when executed by a computer cause the computer to perform the generation method and / or the machine learning method.

[0011] There is also provided a computer readable storage medium having a computer program recorded thereon.

[0012] There is also provided a system comprising a processor coupled to a memory, the memory having a computer program stored thereon, and optionally a graphical user interface coupled to the processor.

[0013] There is further provided a device comprising a data storage medium having a computer program recorded thereon.

[0014] The device may form or function as a non-transitory computer-readable medium, for example, in a Software as a Service (SaaS) or other server or cloud-based platform. Alternatively, the device may comprise a processor coupled to a data storage medium. The device may thus form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a graphical user interface coupled to the processor. [Brief explanation of the drawings]

[0015] Non-limiting examples will now be described with reference to the accompanying drawings. [Figure 1] FIG. 1 shows a flowchart of an example of a generation method and a machine learning method. [Figure 2] FIG. 1 shows a flowchart of an example of a generation method and a machine learning method. [Figure 3] FIG. 1 illustrates an example of a scene encoder. [Figure 4] FIG. 10 illustrates an example of an implementation of a generation method. [Figure 5] FIG. 10 is a diagram illustrating an example of a viewpoint overlap measure. [Figure 6] 1A and 1B show examples of results obtained by the generation method and the machine learning method. [Figure 7] 1A and 1B show examples of results obtained by the generation method and the machine learning method. [Figure 8] 1A and 1B show examples of results obtained by the generation method and the machine learning method. [Figure 9] FIG. 1 illustrates an example of a system. DETAILED DESCRIPTION OF THE INVENTION

[0016] Referring to the flowchart of FIG. 1 , a computer-implemented method for generating multiple 2D images of a 3D scene (hereinafter referred to as the generation method) is proposed. The method includes obtaining S10 configuration data including a layout of the 3D scene and a machine learning model configured to generate the 2D images. The model takes as input a viewpoint, a 3D scene layout, and a latent vector. The model includes a scene encoder and a generative image model. The scene encoder takes as input the 3D scene layout, a viewpoint, and a latent vector, and outputs a scene encoding tensor. The generative image model takes as input the scene encoding tensor output by the scene encoder and outputs a generated 2D image. The method includes S20 generating multiple first 2D images of the 3D scene, each having a respective viewpoint, by applying the model to the viewpoint, the 3D scene layout, and the respective initial latent vector of the first 2D image, for each first 2D image. The method includes S30 calculating a first latent vector for each generated first 2D image by applying a global projector to the generated first 2D image. The method includes calculating S40, for each generated first 2D image, a second latent vector as a weighted combination of the calculated first latent vectors. The method includes generating S50, for each given viewpoint of the first 2D image, a plurality of second 2D images of the 3D scene by applying a model to the given viewpoint, the layout of the 3D scene, and the calculated second latent vectors.

[0017] Such a generation method forms an improved solution for generating multiple 2D images of a given 3D scene.

[0018] In particular, the generative method enables the automatic and efficient generation of multiple 2D images of a given 3D scene. In particular, applying the model enables the generation of (various and realistic) 2D images from high-level, abstract, and proxy representations of the 3D scene (and therefore is easy to define). Indeed, upon training, the model enables the generation of 2D images of a 3D scene from only the 3D scene's layout, latent vectors, and viewpoints. From the layout, latent vectors, and viewpoints, the trained model enables the generation of 2D images, which are particularly useful and interesting for representing objects within the 3D scene. In particular, providing these two inputs to the trained model is much easier for a user than providing accurate object models for each object in the 3D scene and then using traditional rendering methods. Thus, the trained model enables the easy and rapid generation of multiple 2D images of the 3D scene the user is building by simply defining their layout and providing viewpoints for these images.

[0019] In particular, the model generates particularly realistic and relevant 2D images of a 3D scene. Indeed, the model is equipped with a scene encoder that can take into account the layout in the generated 2D image. Thus, the model can generate 2D images that take into account the perspective of the 3D scene and its lighting, as well as occlusions between objects (the scene encoder can take this information into account during the generation of the 2D image). In other words, the 2D images generated by the trained model are 3D-aware, i.e., they take into account the 3D environment of the 3D scene. In particular, the method allows for the consideration of off-screen objects in the 3D scene when generating the 2D image, i.e., objects that are not within the camera's field of view affect the generated image (e.g., light from a window).

[0020] Furthermore, the method uses a 3D-aware, style-conditioned generative model to enable the generation of multiple 2D images of a given scene that are visually and functionally consistent across various viewpoints. Indeed, the method generates each second 2D image by applying a model conditioned by a second latent vector calculated as a weighted combination of the calculated first latent vectors, thereby enabling improved visual and functional consistency across the generated second 2D images. Visual and functional consistency means that the overall appearance of the generated second 2D images of the scene and its elements is as close to each other as possible, thus reinforcing the impression that a single scene has been rendered from multiple viewpoints.

[0021] Furthermore, the generation method has the significant advantage of being computationally and implementation-efficient. Indeed, the inference method, first, does not rely on costly optimizations to maintain coherence across multiple views, and second, does not require (re)training a dedicated neural network for the task and therefore does not rely on multi-view datasets, which are typically difficult to obtain. It can be easily implemented to be used with existing models in inference. Furthermore, the generation of multiple coherent 2D images of a scene can be parallelized in inference by leveraging batch processing.

[0022] Furthermore, the proposed machine learning method is trained end-to-end for the task in a single training phase: it does not rely on large-scale pre-trained image generation models or pre-trained depth estimators.

[0023] The generation method and / or machine learning method are computer-implemented. This means that the steps (or substantially all steps) of the generation method and / or machine learning method are performed by at least one computer, or any similar system. Thus, the steps of the generation method and / or machine learning method are performed by a computer, possibly fully automatically or semi-automatically. In examples, triggering of at least some of the steps of the generation method and / or machine learning method may be performed through user-computer interaction. The level of user-computer interaction required depends on the expected level of automation and may be balanced against the need to implement user preferences. In examples, this level may be user-defined and / or predefined.

[0024] For example, in the generation method, step S10 of obtaining placement data may include, upon user interaction (e.g., performed by a user currently designing the 3D scene), executing a user-generated process for generating a layout of the 3D scene that is taken as input by the model. For example, the user-generated process may include determining one or more (e.g., all) bounding boxes that the 3D scene will contain. Determining the given bounding box may be performed in any manner. For example, determining the given bounding box may include setting a size of the given bounding box and placing the sized bounding box inside the 3D scene. The sizing and placing steps may be performed manually by the user. For example, the sizing step may include the user inputting a width, depth, and height (e.g., through user interaction using a keyboard). The placing step may include the user inputting coordinates of a point on the bounding box (e.g., a corner or its center) and its direction, or the user moving the bounding box to its position within the 3D scene (e.g., the bounding box may be displayed on a screen and moved by the user using a mouse). In an example, the step of setting the size may be performed semi-automatically. For example, the step of setting the size may include a user selecting a category of object to be represented by a given bounding box and automatically suggesting a size (e.g., width, depth, and height) of the given bounding box according to the selected category (e.g., from a database that stores default sizes for different categories of objects). The suggested size may be accepted by the user or may be adjusted by the user (e.g., manually, as described above). For example, if a user wants to add a couch to their 3D layout, the user-generated process may suggest a default bounding box that matches the object category entered by the user, with the user then modifying these dimensional values.

[0025] Alternatively or additionally, the user-generated process may include determining a boundary of the 3D scene. Determining the boundary of the 3D scene may be performed in any manner. For example, determining the boundary of the 3D scene may include determining a set of respective points that represent the boundary of the 3D scene. For example, determining the boundary may include determining some of the points in the set (e.g., representing corners of the 3D scene) and then sampling other points on the boundary between these points (i.e., along the walls of the 3D scene) that represent the corners of the 3D scene.

[0026] In another example, obtaining the configuration data S10 may include retrieving the configuration data to be taken as input by the model, for example, from a database. In that case, the configuration data may have been generated before the generation method was performed, for example, using the user-generation process described above, or may have been recorded in the database after the user-generation process was performed. Similarly, obtaining the machine learning model may include retrieving the machine learning model from a database. The model may have been trained by performing the machine learning method and then recorded in the database.

[0027] In an example, the generation method may also include acquiring a respective viewpoint for each of the first 2D images (each of the respective viewpoints for each of the first 2D images forming a set of viewpoints). Acquiring this set of viewpoints may include determining each of the viewpoints in this set, for example, during user interaction. Determining each viewpoint may include setting parameters of a camera from which the first 2D images are generated. These parameters may include camera position, field of view, and / or pitch. Determining these parameters may include a user manually entering them and, optionally, suggesting at least some of them to the user (e.g., with default values). For example, determining the viewpoints may be performed by a user, for example, by entering viewpoint coordinates and / or direction or by selecting this information on a screen displaying the 3D scene. Alternatively, determining the set of viewpoints may be performed automatically, for example, by another function that predicts a set of relevant viewpoints for the 3D scene taking into account its layout. Further alternatively, these two methods of determining viewpoints may be combined, and some viewpoints may be predicted while others are manually entered by a user.

[0028] In an example, the configuration data may also include at least one conditioning signal. The at least one conditioning signal may include one or more conditioning signals for a 3D scene. In this case, each conditioning signal for the 3D scene may be associated with a respective viewpoint, and each initial latent vector taken as input by the model to generate the first 2D image from the respective viewpoint may be a projection of this conditioning signal of the 3D scene by a global projector. Each conditioning signal for the 3D scene may condition the 3D scene as a whole, for example, conditioning the overall atmosphere or feel of the 3D scene. The conditioning signal for the 3D scene may be of an image type or a text type.

[0029] Alternatively or additionally, the at least one conditioning signal may include one or more conditioning signals for each object represented by at least one of the at least some of the bounding boxes. For example, the placement data may include one or more conditioning signals for at least some (e.g., all) of the objects present in the 3D scene (each represented by a respective bounding box in the layout). In that case, the layout encoder of the scene encoder may further take as input projections of one or more conditioning signals for the objects of the 3D scene. In other words, the projections of one or more conditioning signals for the objects may be included in the parameters taken as input by the layout encoder to condition the appearance / behavior of these objects.

[0030] A conditioning signal for the 3D scene and / or one or more conditioning signals for each object may be selected. For example, acquiring the configuration data may include selecting at least one conditioning signal during user interaction. Each selected conditioning signal may be stored, for example, in a database, and the input may include a user selection of the stored conditioning signal. The type of each conditioning signal may be among a predetermined set of at least two types, which may include an image type and a text type. In that case, the global projector may be a multimodal encoder. The multimodal encoder may be any model, such as a contrastive language-image pre-training (CLIP) model, configured to project several types of conditioning signals into a single latent space. An image-type conditioning signal may be a 2D image of a 3D scene or object. A text-type conditioning signal may be free text (e.g., a dialog box) describing the atmosphere of a room or object.

[0031] A typical example of a computer implementation of the generation method and / or machine learning method is to perform the generation method and / or machine learning method using a system adapted for this purpose. The system may comprise a processor coupled to a memory and a graphical user interface (GUI), and the memory stores a computer program including instructions for performing the generation method and / or machine learning method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally comprise several physically separate parts (e.g., one for the program, and possibly one for the database).

[0032] The datasets considered by machine learning methods to train models may be stored in a database. A "database" refers to any collection of data (i.e., information) organized for search and retrieval (e.g., a relational database based on a predefined structured language, e.g., SQL). When stored in memory, a database allows for rapid search and retrieval by a computer. In practice, databases are structured to facilitate the storage, retrieval, modification, and deletion of data, along with various data processing operations. A database may consist of a file or a set of files that may be divided into records, each of which consists of one or more fields. A field is the basic unit of data storage. Users may retrieve data primarily through queries. Using keywords and sorting commands, users can quickly explore, rearrange, group, and select fields within many records to search or create reports on specific aggregations of data according to the rules of the database management system they are using.

[0033] Generative and / or machine learning methods generally manipulate modeled (3D) objects. A modeled object is any object defined by data stored, for example, in a database. By extension, the term "modeled object" refers to the data itself. Depending on the type of system, a modeled object may be defined by various types of data. In fact, a system may be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these various systems, a modeled object is defined by corresponding data. Thus, it can be referred to as a CAD object, a PLM object, a PDM object, a CAE object, a CAM object, CAD data, PLM data, PDM data, CAM data, or CAE data. However, since a modeled object may be defined by data corresponding to any combination of these systems, these systems are not exclusive of one another. Thus, a system may be both a CAD system and a PLM system.

[0034] A CAD system also refers to any system at least adapted for designing a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, data defining the modeled object includes data enabling the representation of the modeled object. A CAD system may provide a representation of a CAD modeled object, for example, using edges or lines, and in certain cases, faces or surfaces. Lines, edges, or surfaces may be represented in various ways, for example, with non-uniform rational B-splines (NURBS). In particular, a CAD file contains specifications from which geometry can be generated, thereby enabling the generation of a representation. The specifications of a modeled object may be stored in a single CAD file or in multiple CAD files.

[0035] In an example, each 3D scene may represent a real room, e.g., an indoor real room. For example, the rooms represented by the 3D scenes may be rooms in a residential (e.g., house or apartment) such as a kitchen, bathroom, bedroom, living room, garage, laundry room, attic, office (e.g., private or shared), conference room, children's room, nursery, hallway, dining room, and / or library (this list may include other types of rooms). Alternatively, the rooms represented by the 3D scenes may be other indoor rooms, such as in a factory, museum, and / or theater. Alternatively, the rooms represented by the 3D scenes may be outdoor scenes, such as a garden, terrace, or amusement park.

[0036] Each object in each 3D scene may represent the geometry of a real object located in the real room that the 3D scene represents. This real object may be manufactured in the real world after completion of its virtual design (e.g., using a CAD software solution or system). The 3D scene may comprise one or more furniture objects, such as one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more sideboards, one or more nightstands, one or more desks, and / or one or more wardrobes. Alternatively or additionally, the 3D scene may comprise one or more decorative objects, such as, for example, one or more accessories, one or more plants, one or more books, one or more frames, one or more kitchen accessories, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors, and / or one or more electronic objects (e.g., a refrigerator, a freezer and / or a washing machine).

[0037] The ability to generate multiple 2D images of a given scene that are both realistic and consistent opens up a wide range of practical applications. Indeed, these practical applications may seek to generate several views of the same scene without removing or adding visual elements or modifying the visual elements represented on the several generated views. Moreover, multi-view consistency is a key requirement for achieving successful 3D reconstruction, i.e., reconstructing a 3D model from multi-view images of a scene. Examples of such practical applications that the generation method may be involved in are now provided in the following paragraphs.

[0038] The generation method may be included in a real-life room design (i.e., effective placement) process, which may include using the generated plurality of second 2D images to present the room to be placed after performing the generation method. For example, the presentation may be to a user, such as an owner of a home in which the room is located. The generated plurality of second 2D images may be used to help the user decide whether to acquire one or more objects in a 3D scene and may assist the user in their selection by showing the objects in the room from various viewpoints. Because the plurality of second 2D images includes several 2D images of a 3D scene that are visually and functionally consistent across various viewpoints, they may be used to present a complete virtual interior of a room (i.e., including several 2D images of the room).

[0039] Alternatively or additionally, the real-life room design may include using the generated second 2D images to perform a similarity-based search for 3D objects from a catalog that are placed at the locations of the bounding boxes. The trained model enables this due to the realism of the generated second 2D images. For example, the real-life room design may include a user defining a layout of a given 3D scene by placing 3D bounding boxes. The real-life room design may then include generating multiple 2D images of the given 3D scene using a generation method. The user may then particularly understand the appearance and / or style of the 3D scene presented by the generated 2D images and wish to populate the given 3D scene with the most similar 3D objects from the catalog (i.e., replace the bounding boxes with real 3D objects). In that case, real-life room design may include: for each object in the 3D scene, deriving a position of each object in a plurality of generated 2D images from the defined layout; cropping the object in one of the generated 2D images; calculating an image embedding of the object (e.g., using a pre-trained language-image model, hereinafter referred to as CLIP, such as that described in the paper "Learning transferable visual models from natural language supervision" by Radford et al., International Conference on Machine Learning, PMLR2021); and comparing this image embedding of the object with one from the catalog to obtain a most similar object. The real-life room design may include replacing a bounding box in the 3D scene with the most similar object from the obtained catalog for each object.

[0040] Alternatively or additionally (e.g., prior to presentation), the real-life room design process may include adding one or more new objects to a 3D scene (which may initially be, e.g., partially empty) representing a room by modifying the layout of the 3D scene. Adding may include repeating, for each new object, the steps of sizing and positioning a bounding box representing the new object as discussed above. Thus, the generated 2D image may include the new object added to the 3D scene by modifying the layout. The real-life room design process enables the creation of rich and comfortable environments (for animation, advertising, and / or for generating virtual environments, e.g., for simulation). The real-life room design process may also be used to generate virtual environments. The real-life room design process may be included in a general process that may include repeating the real-life room design process for several 3D scenes, thereby presenting several 3D scenes with objects.

[0041] Alternatively or additionally, the real-life room design process may include physically arranging a (i.e., actual) room after the method is performed so that its design matches the 3D scene presented using the generated plurality of second 2D images. For example, a room (without the objects represented by the input 3D scene) may already exist in the real world, and the real-life room design process may include placing an actual object (i.e., an object represented by one of the bounding boxes of the layout) represented by one of the objects of the 3D scene within the already-existing (i.e., real-world) room. The bounding box of this object may be added by the user to the layout of the 3D scene. The actual object may be placed according to the position of its bounding box within the 3D scene. The real-life room design process may repeat this process to place various actual objects within the already-existing room. Alternatively, the room may not yet exist at the time the method is performed. In that case, the real-life room design process may include building a room (i.e., adding real objects to the room) according to the generated 2D image of the 3D scene (i.e., by placing real objects in the positions of bounding boxes representing the real objects in the layout of the 3D scene). Because the method improves the placement of 3D objects in the 3D scene, the method also improves the construction of a room corresponding to the 3D scene, and therefore improves the productivity of the real-life room design process.

[0042] We now describe the steps for obtaining a dataset of training samples.

[0043] The dataset includes multiple training samples (e.g., over 50,000 training samples including 2D images of the same type of room), each including a 2D image. The dataset also includes, for each training sample, configuration data (e.g., the layout of the 3D scene and / or conditioning signals) of the 3D scene captured (e.g., partially) within the 2D image included in the training sample, and the viewpoint from which the 2D image is captured (e.g., coordinates of the viewpoint within the 3D scene). In the dataset, some (e.g., all) of the conditioning signals for objects may be 2D images of these objects stored in a database (e.g., first retrieved using a reference to the object). Each 2D image included in the training sample may be captured for a respective (i.e., different) 3D scene. Alternatively, the dataset may include training samples including 2D images of the same 3D scene captured, for example, from different viewpoints and / or with different lighting. The dataset may also include, for each training sample, information indicating the layout of the 3D scene that the 2D image presents and its viewpoint (e.g., a table with rows containing the 2D image reference, a reference to the corresponding 3D scene layout, and the coordinates of the viewpoint of the 2D image, respectively).

[0044] At least some of the training samples of the dataset (e.g., more than 50 or 75% of the training samples) may include conditioning signals for 3D scenes and / or for objects within the 3D scenes. The conditioning signals of at least some of the training samples may include at least one first conditioning signal having a first type from a predetermined set of at least two types and at least one second conditioning signal having a second type from the predetermined set of at least two types. For example, the dataset may include at least one conditioning signal of a text type and at least one conditioning signal of an image type for most of the training samples (e.g., more than 75% or more than 80% of the training samples), which allows the model to be trained to consider both the first type and the second type. Alternatively, the dataset may include conditioning signals of only the same type (text or image). In that case, to train the function to take into account both the first and second types, the machine learning method may include generating the other type of conditioning signal before training (e.g., using an image or text generator as described below), or alternatively, the scene encoder may comprise a multi-modal encoder (such as CLIP) that can take another type of conditioning signal as input.

[0045] The 2D images included in the training samples of the dataset may be realistic 2D images of the 3D scene created prior to execution (e.g., by a designer) of the method. These 2D images may include, for example, perspective, occlusion, and / or lighting factors. To achieve such rendering, the 2D images of the 3D scene in the dataset may have been manually reworked by the designer (e.g., at least partially in places where rendering is difficult due to perspective, occlusion, and / or lighting factors).

[0046] The dataset may be stored in a database. Obtaining the dataset may include retrieving the dataset from the database. Obtaining the dataset may then include storing the retrieved dataset in a memory. After recording, the machine learning method may train a model based on the recorded dataset. Alternatively, obtaining the dataset may include providing access to the dataset in a database. In that case, the machine learning method may use this access to train a model.

[0047] The rooms represented by the 3D scenes in the dataset may or may exist in the real world (either already at the time of acquisition of the dataset or in the future). For example, the rooms may be actual, real rooms in the real world (in terms of layout), and objects may be placed inside these real rooms as specified in the layout of the 3D scenes contained in the dataset. The 3D scenes may represent rooms designed (e.g., by an interior designer) and then implemented in the real world (i.e., the multiple 3D scenes correspond to virtually designed rooms that have been or could be recreated in people's homes). In an example, each room represented in the dataset is of the same type. For example, all rooms represented in the dataset and the 3D scenes may be kitchens, bathrooms, bedrooms, living rooms, garages, laundry rooms, attics, offices (e.g., private or shared), conference rooms, children's rooms, nurseries, hallways, dining rooms, or libraries (this list may include other types of rooms). In that case, the layout obtained during execution of the generation method may be a layout of a 3D scene of the same type as that in the dataset, which allows for the generation of realistic 2D images and improves the stability of the generation method. Alternatively, the dataset may include a variety of room types, in which case the output domain of the generative image model may be large, the number of rooms represented in the dataset may be large, and the training time of the generative image model may also be long.

[0048] In an example, the layout for each 3D scene may include a set of bounding boxes representing objects in the 3D scene. Each bounding box may be rectangular in space and may encapsulate the outer envelope of the object it represents. The layout may include, for each bounding box, parameters representing the position, size, and orientation within the 3D scene of the object represented by the bounding box. For example, the layout may include, for each bounding box, parameters representing the position of the bounding box (e.g., coordinates of the corners or center of the bounding box), parameters representing the size of the bounding box (e.g., width, depth, and height of the bounding box), and parameters representing the orientation of the bounding box (e.g., rotation about each axis of the global reference frame). Optionally, the layout may include, for each bounding box, parameters representing the class of the object represented by the bounding box. The object classes may be predetermined or may correspond to the types of objects they represent. The object classes may be the decorative object and functional object types described above.

[0049] Each 3D scene layout may also include a boundary of the 3D scene. For example, the boundary of the 3D scene may be represented by a respective set of points, e.g., corresponding to corners of the 3D scene or sampled along a wall of the 3D scene. The 3D scene layout may include coordinates of the respective set of points.

[0050] Training the model may include training (e.g., in a supervised manner) a scene encoder and a generative image model to generate 2D images of the dataset when taking as input corresponding layouts and viewpoints included in the dataset. For example, the scene encoder and the generative image model may each include respective parameters (e.g., weights), and supervised training may further comprise determining values ​​for these respective parameters such that the scene encoder and the generative image model suitably reproduce the 2D images of the dataset when taking as input corresponding layouts, viewpoints, and latent vectors included in the dataset. Supervised training of the model may include jointly training the scene encoder and the generative image model (i.e., it may determine their respective parameters simultaneously or during the same process). In an example, a global projector may also be trained along with the model. For example, the global projector may also include respective parameters (e.g., weights), and training may include determining values ​​for these respective parameters of the global projector. In that case, the dataset may include conditioning signals for at least some of the training samples, and further, during training, the latent vectors considered may be the projections of these conditioning signals by a global projector.

[0051] In an example, the machine learning method may further include replacing a predetermined portion of each latent vector of a dataset with a latent vector having a predetermined value before or during training. This predetermined value may represent a null value (i.e., a value used in the absence of a conditioning signal). For example, at each iteration of training, the replacing may include determining whether to retain or drop a latent vector of a training sample (e.g., with a given probability). Alternatively, determining whether to retain or drop a latent vector may be performed before training. Performing the replacement before training would always result in the same conditioning settings for each training sample in the dataset, so performing the replacement at each iteration improves variability (and therefore the accuracy of the trained function). Replacing a predetermined portion of each latent vector of a dataset improves the flexibility of the trained model. Indeed, when a model is trained using conditioning dropout of latent vectors, the model can be used both conditioned (given input latent vectors) or unconditioned (without latent vectors). When the training also includes training a global projector, permuting the predetermined portions of each latent vector of the dataset may include permuting the conditioning signals of the dataset that correspond to these latent vectors.

[0052] In an example, the scene encoder may include a layout encoder configured to encode a set of bounding boxes. For each of the set of bounding boxes (e.g., visible or invisible from a viewpoint), the layout encoder may take as input parameters that describe the position, size, and orientation within the 3D scene of the object represented by the bounding box. Optionally, for each bounding box, the layout encoder may further take as input parameters that describe the class of the object represented by the bounding box. These parameters may be included in the layout of the 3D scene, as described above. The layout encoder may estimate these parameters from the layout taken as input by the model. The layout encoder may be configured to output a vector (hereinafter referred to as a "layout vector") that embeds the parameters.

[0053] For example, the layout encoder may include a positional encoding module that takes bounding box parameters as input and outputs a layout vector. The positional encoding module may be configured to deterministically increase the dimensionality of the scalar values ​​of the parameters taken as input. For example, the positional encoding module may be configured to output, for each bounding box, a position vector representing the position and size of the bounding box and a direction vector representing the orientation of the bounding box. Optionally, the layout encoder may further include a first multi-layer perceptron encoder configured to increase the dimensionality of the direction vectors output by the positional encoding module. The layout encoder may also include a concatenation layer configured to concatenate the vectors output by the positional encoding module and / or the first multi-layer perceptron encoder and output the layout vector.

[0054] In an example, the model may be included in a function further including a global projector. The global projector may be configured to project each generated first 2D image into a latent space. Each latent vector may be included in this latent space. In an example, the global projector may be a multimodal encoder. The multimodal encoder may be configured to project each conditioning signal into a single latent space. The multimodal encoder may be any model, such as a contrastive language-image pre-training (CLIP) model, configured to project several types of conditioning signals into a single latent space. When the layout data includes one or more conditioning signals for objects, the layout encoder may further take as input, for at least some of the bounding boxes, a projection of each conditioning signal for the object represented by the bounding box. In other words, the projection of one or more conditioning signals for the objects of the 3D scene may be included in the parameters taken as input by the layout encoder. In that case, the layout encoder may further comprise a second multi-layer perceptron that takes as input the conditioning signal projections for the object and outputs a conditioning signal vector. A concatenation layer of the layout encoder may be configured to concatenate the output conditioning signal vector with other vectors output by the position encoding module and / or the first multi-layer perceptron encoder.

[0055] In an example, the scene encoder may further include a floor encoder configured to encode a boundary of the 3D scene. As described above, the boundary of the 3D scene may be represented by a respective set of points, and the floor encoder may take the respective sets of points as input and output a floor vector. For example, the floor encoder may include a PointNet model configured to encode the respective sets of points (e.g., as described in the paper by Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, "Pointnet: Deep learning on point sets for 3D classification and segmentation," CVPR 2017), and optionally a multi-layer perceptron configured to take the output of the PointNet model as input.

[0056] In an example, the scene encoder may further include a camera encoder configured to encode a viewpoint. For example, the viewpoint may include parameters of a camera from which the 2D image is generated (e.g., camera position, field of view, and pitch). The scene encoder may be configured to take these camera parameters as inputs and output a camera vector. For example, the scene encoder may comprise a position encoder configured to take the camera parameters as inputs and, optionally, a multi-layer perceptron.

[0057] In the example, for each given 2D image of a given 3D scene in the dataset, the size and position of an object represented by a bounding box in the layout of the given 3D scene are defined in a coordinate system based on the position and orientation of the camera from which the given 2D image is captured. Therefore, the camera position is already encoded in the layout vector output by the layout encoder. Therefore, the viewpoint may include only two scalar values ​​representing the field of view and pitch of the camera, respectively. This allows for a reduction in the number of learned parameters, improving robustness and thus facilitating model convergence.

[0058] In an example, the scene encoder may further include a transformer encoder. The transformer encoder takes as input a concatenation of the set of bounding boxes encoded by the layout encoder (i.e., layout vector), the viewpoints encoded by the camera encoder (i.e., camera vector), and the boundary of the 3D scene encoded by the floor encoder (i.e., floor vector). When the configuration data includes one or more conditioning signals for the 3D scene, projections of the one or more conditioning signals for the 3D scene by the multimodal encoder may be included in the concatenation taken as input by the transformer encoder. The transformer encoder outputs a scene encoding tensor. For example, the transformer encoder may comprise a transformer model configured to take as input a vector to form a sequence of tokens represented as a tensor (the scene encoding tensor). The vector taken as input by the transformer model is the concatenation of all the vectors mentioned above (i.e., layout vector, camera vector, floor vector, and optionally projection of one or more conditioning signals for the 3D scene), optionally padded (or supplemented) with one or more "zero" tokens to form a vector of a fixed (e.g., predetermined) size.

[0059] The generative image model may be any generative image model capable of generating 2D images conditioned with an output scene encoding tensor. The generative image model may be a type of deep neural network trained on a large image dataset to learn the underlying distribution of training images. By sampling from the learned distribution, such a model may be configured to create novel images with properties from those in the training dataset. Examples of generative image models for generating 2D images conditioned with an output scene encoding tensor include generative adversarial networks (GANs), variational autoencoders (VAEs), and diffusion models.

[0060] In an example, the generative image model may be a diffusion model. The diffusion model may be configured to generate an output 2D image by iteratively removing noise from an initial noisy image based on a scene encoding tensor output by a scene encoder. Examples of diffusion models include a cascade model or a latent diffusion model. A cascade model is a model that includes several diffusion modules, such as a diffusion module for outputting an image conditioned with a scene encoding tensor, and a super-resolution model that then upscales the image to a higher resolution. During inference, the diffusion model may generate an output 2D image by iteratively removing noise from the initial noisy image. Each iteration of noise removal may include determining a new version of the initial noisy image that is less noisy than a previous version of the initial noisy image determined during a previous iteration. Determining the new version may be based on a prediction of the noise of the previous version.

[0061] We now describe training of a diffusion model. The training of the diffusion model may be based on generated noisy versions of the 2D images included in the dataset. For example, a machine learning method may include generating noisy versions of the 2D images of the dataset (by adding noise to these 2D images), and the training of the diffusion model may be based on the generated noisy versions of the 2D images. The diffusion model may be trained to remove noise added to the 2D images of the dataset by taking into account the scene encoding tensor output by the scene encoder. In that case, the training of the diffusion model and the scene encoder may take into account a training loss that penalizes the distance between the predicted noise in the generated noisy version and the actual noise in the generated noisy version.

[0062] In an example, the diffusion model may have an architecture that includes a denoiser. The denoiser may comprise several blocks configured to create a generated 2D image. At least one of these blocks may be enhanced by cross-attention using a scene encoding tensor. The cross-attention mechanism may be an attention mechanism applied between elements from various sequences. The cross-attention may be applied between the representations returned by the transformer encoder for various tokens and the visual features calculated within the denoiser. This improves the model's ability to learn the visual and spatial dependencies / relationships that exist between the scene features (encoded by the transformer encoder) and their visual representations in the image (created by the denoiser). Therefore, this mechanism is particularly suitable for generating 2D images of 3D scenes, where the spatial location / concept of objects within the 2D image is of utmost importance.

[0063] In an example, the diffusion model may be configured to operate in a latent space (i.e., it may be a latent diffusion model). In that case, during training, the diffusion model may be trained to denoise compressed latent representations of 2D images of the dataset. In that case, training the model may include compressing 2D images of the dataset in a latent space (e.g., one with small dimensions), thereby obtaining compressed latent representations. Training may be performed based on these compressed latent representations (rather than the 2D images directly). During inference, the diffusion model may take as input, instead of the initial noisy tensor, a compressed initial noisy tensor of the same dimension as the compressed latent representation. The diffusion model may iteratively remove noise from this compressed initial noisy tensor, given a scene encoding tensor output by a scene encoder. Decompression may then be applied to the result to obtain the generated 2D image. An example of such a compression / decompression implementation includes a variational autoencoder (VAE).

[0064] Applying the model will now be described. Applying the model may first include forming an input for the generative image model. When the generative image model does not operate in latent space, this step may include, for example, sampling an initial noisy tensor having the shape of the 2D image to be generated. This initial noisy tensor may be taken as an input by the generative image model. When the generative image model operates in latent space, this step may include sampling a compressed initial noisy tensor (i.e., having the same dimensions as the compressed latent representation). This compressed initial noisy tensor may be taken as an input by the generative image model.

[0065] Applying the model may then include applying a scene encoder to the obtained layout, viewpoint, and latent vectors, thereby outputting a scene encoding tensor. Applying the model may then include using a generative image model conditioned on the output scene encoding tensor to generate a 2D image of the 3D scene. The generative image model may iteratively remove noise from the initial noisy tensor sampled when not operating in latent space or otherwise compressed representation thereof. When operating in latent space, applying the model may further include decompressing the clean latent (obtained by iteratively applying the denoiser) back to image space to obtain a generated 2D image.

[0066] In step S20, the machine learning method generates a plurality of first 2D images by applying the model as described above. In particular, for each set of considered given viewpoints, the machine learning method generates each first 2D image by applying the model to the given viewpoints, each initial latent vector (e.g., projections of conditioning signals) associated with the given viewpoints, and a layout included in the configuration data. Thus, the machine learning method generates each first 2D image for each viewpoint (by conditioning the model with each initial latent vector). Similarly, in step S50, the machine learning method generates a plurality of second 2D images by applying the model as in step S20, but instead of applying the model to each initial latent vector, the model is applied to the second latent vector calculated for each first 2D image generated in step S40. Thus, for each generated first image, the machine learning method generates each second 2D image having the same viewpoint and layout, but this time by conditioning the model with the second latent vector calculated in step S40.

[0067] In step S30, the machine learning method calculates a first latent vector for each first 2D image generated in step S20 by applying a global projector to the generated first 2D image. Thus, the machine learning method calculates a first latent vector for each first 2D image generated in step S20. All calculated first latent vectors may be included in the same latent space. For each first 2D image generated, the global projector is configured to take the generated first 2D image as an input and output a first latent vector.

[0068] Next, in step S40, the machine learning method calculates a second latent vector for each first 2D image generated in step S20 as a weighted combination of the calculated first latent vectors. Thus, the machine learning method calculates a second latent vector for each generated first 2D image. This second latent vector is taken as input by the model for generating the second 2D image, instead of the respective initial latent vector, as described above. Each calculated second latent vector is also contained in the same latent space as the first latent vector.

[0069] Next, calculation of a given second latent vector for a given first 2D image will be described. These details apply equally to each second latent vector calculated in step S40. Calculating a given second latent vector may include calculating a set of respective weights (e.g., real numbers between 0 and 1) for each other first latent vector. Each weight may represent a proximity between the two first 2D images. Calculating a given second latent vector may include calculating a weighted combination of the calculated first latent vectors using the calculated set of respective weights. The weighted combination may be a weighted sum of the calculated first latent vectors, each weighted by its respective weight. The calculated given second latent vector may correspond to the calculated sum.

[0070] In an example, the step S40 of calculating the second latent vector may include calculating a viewpoint overlap measure for each pair of first 2D images. The second latent vector may be a weighted combination of the first latent vector and the viewpoint overlap measure. The viewpoint overlap measure calculated for each pair of first 2D images may correspond to a weight representing the proximity between the two first 2D images of the pair. In particular, the viewpoint overlap measure between the two first 2D images may represent the proximity with respect to the viewpoints of the two first 2D images. In particular, the viewpoint overlap measure may measure the overlap of the viewpoints of the two first 2D images. The viewpoint overlap measure may be calculated in any manner. For example, the viewpoint overlap measure of each pair i,j of first 2D images may be calculated based on the following formula:

[0071]

number

[0072] Here, similarity i,j is the similarity score, and v i and v j are unit vectors representing the camera direction of the first 2D image i, j, respectively, and Area inter and Area union are the intersection and union areas between the viewpoints of the pair i,j of the first 2D images, respectively. The unit vector v i may represent the orientation of the camera associated with the first 2D image (i.e., from which the first 2D image is acquired). i may be directed from the camera position (where the first 2D image is acquired). Area inter and Area union Area can be calculated by projecting the volume visible from each viewpoint of each first 2D image onto a 2D plane representing the floor of the 3D scene. inter and Area union may be the intersection and union of the projections of the viewpoints of the first 2D images i, j, respectively. Each set of weights may be estimated from the viewpoint overlap measure, for example, by normalizing the viewpoint overlap measure between 0 and 1.

[0073] In other examples, the machine learning method may use other viewpoint overlap measures. For example, the viewpoint overlap measure for each pair i,j of the first 2D image may correspond to the ratio of the number of objects present at the viewpoint of pair i,j of the first 2D image to the number of objects present at each viewpoint of pair i,j. The number of objects present at a given viewpoint may be the number of objects visible (e.g., partially or completely) in the first 2D image acquired from this given viewpoint. Regarding the aforementioned example of viewpoint overlap measures, the respective sets of weights may be estimated from the viewpoint overlap measure, for example, by normalizing the viewpoint overlap measure between 0 and 1.

[0074] In an example, a machine learning method may include iteratively calculating first and second latent vectors and generating a next plurality of 2D images. Each iteration may consider the multiple 2D images generated in the previous iteration to calculate the first and second latent vectors for the next plurality of 2D images. Each iteration may include, for each first 2D image generated in the previous iteration, calculating a new first latent vector (in the same manner as step S30) by applying a global projector to the first 2D image generated in the previous iteration. Each iteration may then include, for each first 2D image generated in the previous iteration, calculating a new second latent vector (in the same manner as step S40) as a weighted combination of the calculated new first latent vectors. Each iteration may then include, for each given viewpoint of the first 2D image, generating a plurality of new second 2D images of the 3D scene (in the same manner as step S50) by applying a model to the given viewpoint, the layout of the 3D scene, and the calculated new second latent vector. Iteration of these steps allows for improved visual and functional consistency across different viewpoints of the multiple 2D images produced in each iteration.

[0075] In an example, the steps may be repeated until a criterion is achieved. This criterion may, for example, take into account the variation of the second latent vector during the iteration. For example, in each iteration, the calculated new second latent vector may be compared to the second latent vector calculated in the previous iteration, and the machine learning method may stop when the variation between the previous second latent vector and the new second latent vector is zero or close to zero. The variation between two versions of the second latent vector may be evaluated using any measure of the distance between the two second latent vectors.

[0076] Here, implementation forms of the generation method and machine learning method will be described with reference to FIGS. The trained model is conditioned (i.e., 3D-aware) with the output scene encoding tensor. This allows for the creation of accurate and natural-looking immersive content, thanks to the exploitation of knowledge about the 3D structure and relationships of objects in a scene or environment and, therefore, improved consideration of, for example, perspective, occlusion, or lighting factors. In particular, the trained model includes only a scene encoder and a single conditional diffusion model specifically trained end-to-end for this task. It does not utilize large-scale, general-purpose pre-trained image synthesis priors, does not involve several training stages to train modules separately, and does not require training a neural volume renderer, or NeRF, for each generated scene. Because 3D awareness is built in through the layout of the training samples, it does not rely on a separate depth estimator, which is usually flawed and propagates errors, and does not require a multi-view dataset for training. The generation method also enhances the level of user interaction and controllability.

[0077] Machine learning and generative methods solve the problem of generating multiple renderings of a given scene that are visually and functionally consistent across different viewpoints using 3D-enabled, style-conditioned generative models, such as those disclosed in European Patent Application No. 24305100, incorporated herein by reference. "Visually and functionally consistent" means that the overall appearance of the renderings of the scene and its elements is as close as possible to each other, thus reinforcing the impression that a single scene has been rendered from multiple viewpoints. When generating multiple images using such a forward-bound (i.e., no per-scene optimization stage) generative backbone, each generation is performed independently, and ensuring consistency across multiple results is extremely difficult.

[0078] Machine learning and generative methods solve this technical problem using deep learning-based techniques. The pipeline can be divided into two main stages: During the inference stage, generative methods perform view-dependent latent manipulation of the conditioning signal to dramatically improve visual consistency across generated views.

[0079] In the first stage, the machine learning method performs supervised training of a 3D correspondence generative model conditioned on a latent vector η that encodes the 3D layout, the target camera pose, and the visual content and semantics of the target rendering. The trained module that computes η is denoted as τ and can take at least an image as input. τ may be a pre-trained multimodal foundation model, and conditioning dropout may be performed on η during training (i.e., η is randomly replaced by a null vector during training, so that inference can be performed by providing η in conditioned generation or not providing it in unconditioned generation).

[0080] The goal of the supervised training phase is to give the model the ability to reproduce (or generate) 2D reference images when given as input the corresponding scene annotations (i.e., layout) of the 3D scene.

[0081] In the second phase (inference), the generation method includes invoking the iterative denoising process at least twice. The generation method first generates a batch of renderings of a given scene from various viewpoints. Then, the generation method uses τ to calculate a semantic latent embedding η of the generated images. Next, the generation method includes calculating a similarity score between each pair of camera viewpoints. Such a similarity score measures the number of elements represented in the field of view of both cameras. For each camera viewpoint, the generation method includes calculating a centroid combination of the semantic latent embeddings η weighted by the similarity scores between the considered viewpoint and the other viewpoints. Finally, new batches of renderings are generated for the various viewpoints, and each viewpoint generation is conditioned on its respective centroid latent embedding.

[0082] In an example, this may be applied several times, for example, until the similarity between the centroid embeddings of subsequent generation stages and the semantic embeddings of the current generated image for different viewpoints reaches a threshold value.

[0083] The main advantages of machine learning and generative methods include: Generate consistent views at low cost The generation method has the important advantage of being computationally and implementation-wise efficient. Indeed, the inference method, first, does not rely on costly optimizations to maintain coherence across multiple views, and second, does not require (re)training a dedicated neural network for the task and therefore does not rely on multi-view datasets, which are typically difficult to obtain. It can be easily implemented to be used with existing models in inference. Furthermore, the generation of multiple coherent views of a scene can be parallelized in inference by leveraging batch processing. ·Flexibility The generative method can be used to generate consistent renderings of 3D scenes. When the diffusion backbone is trained with conditioning dropout on style latent embeddings, this generation can be performed both conditioned (given an input style vector) or unconditioned. Improved consistency across generated views The ability to generate several renderings of a given scene that are both realistic and consistent opens up many practical applications. Indeed, the generation method can be used to generate several views of the same scene without removing or adding visual elements or modifying the visual elements represented on the several generated views. Moreover, multi-view consistency is a key requirement for achieving successful 3D reconstruction, i.e., reconstructing a 3D model from multi-view images of a scene.

[0084] We now provide definitions of certain terms.

[0085] Deep neural networks (DNNs) are a powerful set of techniques for learning in neural networks, a biologically inspired programming paradigm that enables computers to learn from observed data. In object recognition, the success of DNNs is attributed to their ability to learn rich mid-level media representations, as opposed to the manually engineered low-level features (e.g., Zernike moments, HOG, bag-of-words, SIFT) used in other methods (e.g., min-cut, SVM, boosting, random forests). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move away from feature engineering as much as possible by achieving end-to-end optimization starting from raw features and ending with labels.

[0086] A generative image model is a type of deep neural network that is trained on a large image dataset to learn the underlying distribution of training images. By sampling from the learned distribution, such a model can create novel images with properties from those in the training dataset. GANs (generative adversarial networks), VAEs (variational autoencoders), and diffusion models are widely recognized as the most common generative image models, with diffusion models currently considered the state-of-the-art in the field.

[0087] Diffusion models are a type of deep learning model that can be used for image generation. They aim to learn the structure of a dataset by modeling how data points diffuse through a latent space. Diffusion models consist of three components: a forward process, a backward process, and a sampling phase. In the forward process, Gaussian noise is added to the training data through a Markov chain. The goal of training a diffusion model is to teach the model how to undo the noise addition step by step. This is done in the backward process, where the diffusion model reverses the noise addition performed in the forward process and thus recovers the data. During the sampling phase, the image-generating diffusion model starts with random Gaussian noise images. After being trained to reverse the diffusion process on images from the training dataset, the model can generate novel images similar to those in the dataset. This is achieved by starting with pure Gaussian noise and reversing the diffusion process until a clear image is obtained.

[0088] In the context of generative AI models for image synthesis, conditioning refers to the process of injecting additional information into the image generation process to obtain results that match user-driven constraints. Conditioning can take a variety of forms, including, for example, text (e.g., DALL-E 2, Midjourney, or Stable Diffusion) or images (e.g., ControlNet or semantic segmentation).

[0089] The bounding box of a three-dimensional (3D) object is the smallest rectangular cuboid that encloses the object. Its position, its dimensions, and its orientation characterize the 3D bounding box.

[0090] A "viewpoint" represents the perspective or "camera" from which a rendering is captured. It may include four components: position, orientation, and field of view and pitch. The position and orientation of the viewpoint and the position and orientation of the bounding box may be defined within a single reference frame.

[0091] The term "3D abstract scene" refers to a list of labeled bounding boxes (the layout of the 3D scene) representing objects in the scene (labels correspond to classes of objects), a viewpoint, and optionally other elements that may enhance the description of the environment (e.g., information about the shape of the room). The adjective "abstract" emphasizes that the objects in the scene have no visual representation and are not defined beyond the properties of their bounding boxes and their labels.

[0092] A scene encoder is a specialized deep neural network that learns to extract a comprehensive representation from a 3D scene, which may include spatially arranged objects, layout, or viewpoints. A scene encoder takes in a variety of inputs depending on its specific architecture and the needs of the user, and produces a high-dimensional vector output. This encoded representation should capture important features of the scene and serve as valuable input for subsequent stages of a deep learning model.

[0093] FIG. 2 shows a flowchart of an example machine learning method and an example implementation of a 3D-enabled, style-conditioned image generation model that can be used to sample consistent multi-view renderings (2D images) in a generative manner.

[0094] The pipeline consists of a (latent) diffusion model 100 conditioned by a novel 3D scene encoder 200. Similar to other deep learning models, it features an offline stage S100 (training the model by running machine learning methods) and an online stage S200 (generating 2D images by applying the model, also called the inference stage).

[0095] We now describe the offline training stage S100 in detail. The purpose of this stage is to simultaneously (i) train the scene encoder 200 to create a comprehensive mathematical representation that can be used for conditioning, and (ii) train the diffusion model 100 to generate images from noise. The scene encoder 200 takes as input a set of elements characterizing a 3D scene (the layout of the 3D scene) and outputs a scene encoding tensor. The diffusion model 100 takes as input a noisy version of the image to be generated and the scene encoding tensor, and outputs a denoised version of the input image. This training is end-to-end, i.e., a single loss value is calculated and back-propagated to adjust the weights of both the diffusion model and the scene encoder. The setup of the training stage may include the following subtasks: Data preprocessing steps The data samples of the dataset, particularly the scene annotations (i.e., layout), may be processed so that they can be passed to a scene encoder. The preprocessing stage may optionally include computing object semantic embeddings of the training dataset offline (prior to training experiments), as well as global style embeddings, using, for example, a pre-trained encoder from multimodal base model 250. Computing and storing these embeddings offline helps reduce memory usage and computation during training. ·Architecture definition step The scene encoder 200 may return a single fixed-size tensor embedding for the entire scene being rendered. The diffusion model 100 may take as input a noisy version of the image and a scene encoding vector. It may return an estimate of the noise added to the image and use it to suggest a less noisy version of the input image. Definition of conditioning dropout method Depending on the available modalities representing each object in the dataset, a multinomial probability distribution may be designed to define the probability of selecting a modality for which the object semantic embedding is calculated or a null token, respectively. The probability of selecting a null token is the dropout rate. In each training iteration, the machine learning method may include sampling from this probability distribution independently for each object. Similarly, a separate dropout rate may be set to perform conditioning dropout for the scene-global style embedding. Training loss definition step The training loss function may measure the distance between the predicted noise in the input image and the true noise in the image added through the forward process. Training Steps Training can be performed by iterating over the dataset (image and scene annotation pairs) several times.

[0096] We now describe in detail the image generation / inference stage S200, which aims to output a rendering that matches the viewpoint of an abstract 3D scene. At this stage, the generation method may include the following subtasks: Step to determine scene embedding vectors The scene encoder 200 and the input abstract 3D scene are used to compute a scene embedding vector. The abstract 3D scene does not necessarily have to be part of a database (e.g., it can be created by the user or generated using other techniques). Optionally, object semantic embeddings 204 (conditioning signals for objects), which can be computed from user-provided prompts such as images or text descriptions, can be associated with any object in the scene to guide the generation. Similarly, scene style embeddings 205 (conditioning signals for the 3D scene) can also be optionally passed. Generate a random Gaussian noise image The Gaussian noise image 301 may have the size of the desired final image when training in image space, or the size of the latent space of the VAE when training a latent diffusion model. Iteratively denoising the generated images Using the diffusion model 100, a random Gaussian-noised image 301 is first denoised, and the output of the diffusion model is then iteratively denoised. A U-Net denoiser (the DNN backbone of the diffusion model) can be conditioned with the scene embedding vectors using cross-attention between the layers of the U-Net and the scene embedding vectors. After a fixed number of denoising steps, a final sharp image 302 is produced. Because the diffusion model is trained with conditioning dropout, classifier-free guidance is implemented during denoising to push predictions toward the conditioned model output and away from the unconditioned model output, and thus may better represent the conditioning signal, e.g., the scene style embedding computed from the text description.

[0097] We now describe an example implementation of the general framework described above, focusing on interior scene generation.

[0098] We now present details regarding the acquisition and contents of the dataset used to train the model. The data used may be extracted from user-made HomeByMe® renderings (i.e., 2D images created by actual users). Whenever a high-quality rendering is performed in the application, a rich annotation file is saved along with the image. The raw data from this annotation includes information about the rendering (semantic segmentation maps and / or 2D bounding boxes of visible objects) and information about the scene from which the rendering was performed (3D bounding boxes of objects, room geometry and / or viewpoint). The dataset also includes object-level annotations, i.e., 3D meshes, object categories, image thumbnails of objects, material data, text annotations describing the objects, etc. From this raw data, three elements can be extracted: Annotated Objects For each object in the scene (not necessarily visible in the rendering captured by the user), the annotation file contains a list of various features that describe the object. In particular, it defines the object's 3D bounding box, with two 3D points corresponding to two opposite vertices of the bounding box. This also indicates the object category from a total of 174 possible classes in the HomeByMe dataset. ·perspective The user's rendering perspective can be saved in an annotation file. In particular, the camera position, its orientation, and its field of view are captured and used later in the pipeline. Room shape The room geometry is stored in the annotation file as a list of 2D points representing the corners of the room.

[0099] Figure 3 shows an example of the scene encoder 200 of Figure 2. The scene encoder 200 includes a layout encoder 210 configured to encode a set of bounding boxes 201 and a conditioning signal 204 for the objects represented by the set of bounding boxes 201. For each set of bounding boxes 201, the layout encoder 210 takes as input parameters representing the position, size, and orientation of the object represented by the bounding box within the 3D scene, and, when a conditioning signal is input for the object represented by the bounding box, a projection of this conditioning signal by a multimodal encoder 250 (also referred to as a global style encoder). The scene encoder 200 further includes a floor encoder 230 configured to encode a boundary 203 of the 3D scene. The scene encoder 200 further includes a camera encoder 220 configured to encode a viewpoint 202. As shown in Figure 2, the scene encoder 200 further includes a transformer encoder 240. The transformer encoder takes as input a concatenation of a set of bounding boxes 201 encoded by the layout encoder 210, a viewpoint 202 encoded by the camera encoder 220, a 3D scene boundary 203 encoded by the floor encoder 230, and a projection of the conditioning signal of the 3D scene 205 by the multimodal encoder 250. The transformer encoder 240 outputs a scene encoding tensor.

[0100] The offline training stage S100 will now be described in detail.

[0101] The machine learning method may include a data processing step prior to the training step to process the layout and viewpoint of each 2D image in the dataset.

[0102] The data processing step may include a first step for processing 3D bounding boxes for each 3D scene. The first step may include converting raw 3D bounding boxes from a representation based on two opposing vertices to a representation with their positions (x, y, z), their dimensions (width w, height h, depth d), and their orientations. Objects present in the scene may have only one rotational degree of freedom, i.e., rotation around a vertical axis. As a result, machine learning methods may use only a single angle θ to define the orientation of the bounding box. In practice, machine learning methods may use a different representation, encoding the orientation of the 3D bounding box by the corresponding pair (cos(θ), sin(θ)). Such a parameterization is mathematically equivalent to the single-value parameterization, but strengthens the continuity of the deep learning model between θ=0 and θ=2π. This is beneficial for model convergence. As a result, the processed 3D bounding box is defined by a list of eight parameters (x, y, z, w, h, d, cos(θ), sin(θ)).

[0103] The data processing step may include a second step for processing the conditioning signals of objects of text type (class or description). Each object from the HomeByMe dataset may be described by a class that provides a broad description (chair, table, or door). There may be a total of 174 classes in the HomeByMe dataset.

[0104] The data processing step may include a third step to process the layout boundaries. The third step may include increasing the dimensions of the floor points. The raw points from the data annotations are 2D points (x,y) because their Z coordinate is implicitly 0. The 2D points are changed to 3D points by using 0 as the Z coordinate. This step is necessary because the 3D points are affected by the transformations described below.

[0105] The data processing step may include a fourth step to process the bounding box coordinates, particularly from world coordinates to camera coordinates. The raw positions and orientations found in the annotation file use the world coordinates defined in HomeByMe. To reduce the number of learned parameters, promote robustness, and therefore ease convergence, the data processing step may implement a coordinate system change from the original world coordinates to a viewpoint-based coordinate system. In the new coordinate system, the world origin is set to the camera position, and the basis vectors are selected as follows: the "Z" basis vector is unchanged, the "Y" basis vector is the projection of the viewpoint's forward vector onto a plane perpendicular to the "Z" vector, and the "X" vector is perpendicular to the two mentioned above. With this basis change, the viewpoint can be described purely by two scalar values: field of view (FOV) and pitch (the angle its forward vector makes with the "Y" basis vector). This basis change affects the position and rotation of all objects and points in the scene. While this basis change is optional, it aids in model convergence.

[0106] We now describe the scene encoder architecture in detail.

[0107] The scene encoder consists of four components: a layout encoder, a camera encoder, a floor encoder, and a transformer module (or transformer encoder) (see Figure 3).

[0108] Next, we describe the layout encoder. The scalar values ​​(x, y, z, w, h, d, cos(θ), sin(θ)) describing each bounding box in the scene may be passed through a position encoding module (PE) that deterministically increases the dimensionality of the scalar values. In this example, the scalar values ​​are

number

[0109] After the position encoding module, the position and dimensions of a bounding box, each originally described by three scalar values, are described by a 192-dimensional vector (3 x 64 = 192). On the other hand, a rotation, originally described by a pair of scalar values, is described by a 128-dimensional vector after position encoding. To ensure that the position, dimensions, and rotation are weighted equally by the model, the high-dimensional version of the rotation is

number

[0110] The object semantic embeddings computed by CLIP's pre-trained text and image encoders are

number

number

[0111] All of the previously calculated vectors are

number

[0112] We now describe the camera encoder. A camera or viewpoint is completely described by two scalar values: field of view and pitch. Both of these values ​​are encoded in a high-dimensional space using position encoding.

number

number

[0113] We now describe a floor encoder. The floor is represented solely by an unordered set of 3D points corresponding to its corners. Such a representation is ambiguous and cannot be easily interpreted by a deep neural network. Alternatively, the data processing step may involve densely sampling points along the walls of the room, thereby generating a set of sampled points along the boundary, so that the room boundary is represented by a 3D point cloud. This 3D point cloud is then

number

number

[0114] We now describe a multimodal encoder (or global style encoder). Similar to object semantic embeddings, a global style embedding of a scene obtained by applying a trained CLIP image encoder to a target HQ rendering is

number

number

[0115] We now describe the Transformer module. 3D object tokens, camera tokens, floor tokens, and scene semantic tokens are all concatenated to form a sequence of tokens. These tokens are independent of each other. To capture the relationships between the various elements of this sequence, the Transformer module is used. Due to its inherent architecture, the operation of the Transformer module can be improved using a fixed input size. However, the sequence constructed through the concatenation of the outputs of the layout encoder, camera encoder, floor encoder, and multimodal encoder can have variable length since the number of 3D bounding boxes in a scene can vary from scene to scene. To be compatible with the Transformer architecture, the concatenation of vector sequences is performed using a "zero" token so that the sequence is of fixed length.

number

number

[0116] We now describe the architecture of the diffusion model. The model may include one of two versions of the diffusion model: one that operates directly in image space, and the other that operates in the latent space of a pre-trained VAE to increase the final image dimensionality. In the first case, diffusion occurs directly on pixels from the image, while in the second case, diffusion occurs on a latent version of the image, which is then decoded using a VAE decoder. The two approaches are not fundamentally different and do not require significant changes beyond the introduction of the VAE.

[0117] The diffusion model has a conditional generation architecture featuring a U-Net backbone with four down blocks and four up blocks. In particular, the last two down blocks and the first two up blocks can be augmented with cross-attention using scene embedding vectors. The number of up / down blocks and the number of blocks augmented with cross-attention can be varied depending on the needs and user resources. This configuration provides a good compromise between image quality and training time.

[0118] We now describe the definition of the conditioning dropout scheme. In particular, we first describe conditioning dropout for object semantic embeddings. For each object and at each training iteration, the machine learning method involves drawing an object semantic embedding that is passed to the layout encoder according to the following probability law (see Figure 3): CLIP encoding calculated from the string representation of the object's class. P=0.1. CLIP encoding calculated from object thumbnails. P=0.3 CLIP encoding calculated from the crop during extended rendering of the object. Null token. Machine learning methods use the "zero" token to indicate no object semantic conditioning.

number

[0119] Other probabilities may be set, however, this redistribution allows the architecture to correctly interpret the CLIP latent space while also providing good results, especially when no object semantic conditioning is provided.

[0120] Next, we describe conditioning dropout in the global style embedding of a scene. At each training iteration, the global style embedding computed by applying the CLIP encoder to the target rendering is conditioned to a dropout rate of 0.2, i.e., "zero" tokens.

number

[0121] We now describe the training loss used in training. Diffusion models can be trained using a variety of losses / parameterizations. As introduced in the paper "Denoising Diffusion Probabilistic Models" by Jonathan Ho et al., NeurIPS 2020, which is incorporated herein by reference, at each training iteration, and for each training image, the time step t is uniformly sampled t~U({1,...,T}) and a distribution schedule β is applied. t Gaussian noise is added to the image according to the following: The diffusion model tries to predict the noise added to the image. The loss used is the true noise ε and the predicted noise ε θ where θ indicates that the prediction is based on the parameters of the model.

[0122]

number

[0123] Alternatively, other commonly used diffusion training parameterizations / losses may be utilized interchangeably, for example, the v-prediction parameterization with a min-SNR weighting value of 5.0 provides a good image quality / resolution / computation tradeoff (e.g., as described in the paper "Efficient Diffusion Training via Min-SNR Weighting Strategy" by Tiankai Hang et al., ICCV 2023).

[0124] The generation / inference stage S200 will now be described in detail. The diffusion model may be configured during inference using various techniques. For example, two different sampling processes may be used: denoising diffusion probabilistic model (DDPM) and denoising diffusion implicit model (DDIM). DDIM, for example, may provide a good balance between inference speed and image quality. When inferring an image of a given 3D abstract scene, generation may take approximately 1 second on an NVIDIA RTX A6000 GPU.

[0125] FIG. 4 shows an example of an implementation of the generation method.

[0126] We first describe the offline training phase of a 3D-enabled, style-conditioned image generation model. The generation method applies an image generation model that can be trained according to machine learning methods, for example, with conditioning dropout. In particular, The generative diffusion model can be conditioned with style / semantic embeddings (latent vectors) that capture the style / semantics of the entire scene. Such embeddings can be computed from images, e.g., renderings of the target scene. This conditioning can be performed using conditioning dropout during training, so that inference can be performed both conditioned and unconditioned. In the latter case, classifier-free guidance (CFG) can be applied using a global style encoder and its training strategy. Generative diffusion models can also be 3D-aware and pose-conditioned. This means that the scene rendering is conditioned on the generated camera pose and an abstract 3D representation of the scene, consisting of, for example, annotated 3D bounding boxes of objects. Such conditioning can be performed by a layout encoder and a camera encoder. However, a training strategy for the layout encoder that allows guidance of individual object appearance from multimodal input during inference is optional. This guidance can greatly benefit generative methods, as it allows for appropriate constraints on the model regarding the desired appearance and functionality of each object.

[0127] We now describe the generative method, a two-stage inference method with view-dependent manipulation of the conditioning signal. The generative method may be implemented in the following five steps using a trained model, dramatically improving the coherence between several views of a scene.

[0128] (1) The first step S20 involves a first generation stage, which involves using a trained 3D-enabled, style-conditioned generative model to generate a batch of n renderings (or 2D images) of a 3D scene from n camera poses 401 (or viewpoints). These camera poses 401 may be user-defined or suggested based on the scene layout. If the model has been trained using conditioning dropout on the style latent η, then conditioning the generation on such embeddings is optional and a CFG may be applied. Here, the conditioning style latent η (0) may be calculated from the conditioning images 205, or other modalities depending on the nature of τ.

[0129] (2) The second step S30 involves a latent embedding calculation, which uses τ to compute the style latent representation (η i (1) ) 1≦i≦nThis includes calculating:

[0130] (3) The third step (the first part of step S40) includes a camera similarity score calculation, which includes calculating a similarity score between each pair of camera viewpoints.

number

[0131] (4) The fourth step (the second part of step S40) involves a latent embedding centroid operation, which computes, for each of the n camera poses, a latent vector η weighted by its respective similarity score with the camera of interest. (1) By calculating the centroid combination of i (2) ) 1≦i≦n This includes obtaining

[0132] (5) The fifth step S50 involves a second generation stage, which involves generating a new batch of n renderings, each of which is generated based on the respective style embedding η for that camera pose. (2) It is conditioned with.

[0133] We now present an example implementation of the aforementioned framework. This use case focuses on interior scene generation. The 3D-aware, style-conditioned diffusion model used in inference was obtained following the training procedure, dataset acquisition, and preprocessing detailed above. We now provide an example implementation of the generation method. Note that in this example, τ is the trained CLIP model, and therefore the style embedding η is in the shared CLIP latent space between the text and the image.

[0134] (1) The first generation stage (step S20), which uses a single abstract 3D layout and floorplan as input, in the same way as before, but for n camera viewpoints. It uses the CLIP latencies η obtained from text prompts or inspiration images. (0) It can be conditioned with or without global style conditioning.

[0135] (2) The latent embedding calculation step (S30), i.e., the generation method, calculates the style latent vector η by applying the trained CLIP image encoder to the generated rendering. (1) This includes calculating:

[0136] (3) The step of camera similarity score calculation (first part of step S40), i.e., the generation method, involves measuring the field of view intersection 510 between all distinct pairs of camera viewpoints using the 2D approach summarized in FIG. 5 and the following formula:

[0137]

number

[0138] In Figure 5, the camera areas (viewpoints) are represented here as isosceles triangles 510, 520. The figure shows the field of view intersection 530 measured between viewpoints 510 and 520. The camera range distance r considered for the similarity calculation is i , r j may be derived from the wall / floor plan of the room, or may be fixed to a reasonable predetermined value (e.g., 5 meters for a bedroom scene).

[0139] These scores are then calculated using an n × n similarity matrix

number

[0140] This matrix is ​​symmetric (S i,j=S j,i ) and using the formula proposed above, (S i,i = 1). This similarity matrix has a unit diagonal of

number

[0141] This formulation can be viewed as a 2D approximation and can be effectively evaluated for several pairs of viewpoints. However, accurate similarity scores may be calculated in 3D space in a similar manner by computing a 3D volume intersection of the camera views. The 2D approximation significantly reduces complexity, both in terms of computation and implementation effort, and is sufficiently accurate for most common camera poses.

[0142] (4) Latent Embedding Centroid Operation Step (Second Part of Step S40): For each camera viewpoint, the centroid latent is obtained by calculating the average weighted by the similarity score.

[0143]

number

[0144] (5) The second generation stage (step S50), i.e., the generation method generates a new batch of renderings similar to the first stage, but with η as the conditioning style latent for each style latent vector. (2) This includes using

[0145] Example results are now described with reference to Figs. 6-8. In particular, qualitative generation results are presented. For a given abstract 3D scene, the generation method generates a (0) Generate 2D images (renderings) from multiple camera poses without specifying the camera position.

[0146] 6 shows the first 2D images acquired from the multiple viewpoints generated in step S20. These are images from the first generation stage. Even though the input 3D layout is the same for each view, the visual appearance of the scene can change dramatically from view to view, thus leaving the impression that different scenes have been rendered.

[0147] Figure 7 shows the multiple second 2D images generated in step S50 for the multiple first 2D images shown in Figure 6. Figure 7 shows that the overall appearance of the scene is preserved across the various generated second 2D images.

[0148] Here we present details on the generation time using our generative method. Using the generative backbone presented previously, it took 37 seconds on an RTX A6000 GPU to generate 30 904x512 views of a scene. By comparison, generating the same number of views without applying our generative method would take 18 seconds, which makes sense since our method generates twice. Therefore, our generative method does not take the time to generate multiple 2D images twice, and is therefore much faster than other methods that require expensive optimization stages.

[0149] Here, quantitative metrics are presented. In particular, we present the results of an evaluation conducted to quantitatively evaluate the ability of our generation method to improve visual / semantic consistency across several generated views of a given scene. The evaluation includes the following steps: In a first step, nine evaluation 3D scenes are selected, and for each of them at least 16 renderings from different camera views are generated by repeating step S20 (first generation stage). Then, the CLIP embedding calculated from the generated image after the first generation stage is stored (step S30). Steps S40 and S50 of the generation method are then performed to calculate the CLIP embedding from the final generated image (after the second generation stage). For both sets of CLIP embeddings (obtained after the first and second generation stages), the average of the cosine similarities between all pairs of CLIP embeddings in the set is calculated.

[0150] The obtained values ​​are plotted in Figure 8. The plot shows that the average cosine similarity scores between pairs of generated renderings are consistently high when applying our novel generative technique to all test scenes, indicating that visual / semantic information is well preserved across different views. The dashed horizontal line indicates the average score.

[0151] FIG. 9 shows an example of a system, where the system is a client computer system, for example a user's workstation.

[0152] The client computer in this example also includes a central processing unit (CPU) 1010 connected to an internal communication BUS 1000, and a random access memory (RAM) 1070 also connected to the BUS. The client computer is further provided with a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and the like. Any of the foregoing may be supplemented by or incorporated in specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a keyboard, or other haptic device 1090. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. The cursor control device also allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse, and the buttons on the mouse are used to generate the signals. Alternatively or additionally, the client computer system may be equipped with a sensitive pad and / or a sensitive screen.

[0153] A computer program may include computer-executable instructions, which include means for causing the system to perform a method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, for example, an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions that performs the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation or update program. Applying the program to a system provides, in either case, instructions for performing the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment, the server being in communication with one or more clients over a network, in which case the processing unit executes the instructions contained in the program, whereby the method is performed in the cloud computing environment.

Claims

1. 1. A computer-implemented method for generating a plurality of 2D images of a 3D scene, comprising: configuration data including the layout of said 3D scene, and a machine learning model configured to generate a 2D image, the model taking as input a viewpoint, the layout of the 3D scene, and a latent vector, the model comprising a scene encoder and a generative image model, the scene encoder taking as input the layout of the 3D scene, the viewpoint, and the latent vector and outputting a scene encoding tensor, the generative image model taking as input the scene encoding tensor output by the scene encoder and outputting the generated 2D image. a step (S10) of acquiring generating (S20) a plurality of first 2D images of the 3D scene, each having a respective viewpoint, by applying, for each first 2D image, the model to the respective viewpoint of the first 2D image, the layout of the 3D scene, and a respective initial latent vector; For each generated first 2D image, calculating a first latent vector by applying a global projector to the generated first 2D image (S30); calculating (S40) a second latent vector as a weighted combination of the calculated first latent vectors for each generated first 2D image; generating (S50), for each given viewpoint of the first 2D image, a plurality of second 2D images of the 3D scene by applying the model to the given viewpoint, the layout of the 3D scene, and the calculated second latent vector; 1. A computer-implemented method for generating multiple 2D images of a 3D scene, comprising:

2. 2. The method of claim 1, wherein calculating the second latent vector (S40) comprises calculating a viewpoint overlap measure for each pair of first 2D images, and the second latent vector is a weighted combination of the first latent vector with the viewpoint overlap measure.

3. The viewpoint overlap measure for each pair i, j of the first 2D image is calculated based on the following formula: [Equation 1] Here, similarity ｉ，ｊ is the similarity score, and v ｉ and v ｊ are unit vectors representing the camera directions of the first 2D images i and j, respectively, and Area ｉｎｔｅｒ and Area ｕｎｉｏｎ 3. The method of claim 2, wherein x, j, y, y, and y are the intersection and union areas, respectively, between the viewpoints of the pair i, j of the first 2D image.

4. 3. The method of claim 2, wherein the viewpoint overlap measure for each pair i,j of the first 2D image corresponds to a ratio of the number of objects present at the viewpoint of the pair i,j of the first 2D image to the number of objects present at each viewpoint of the pair i,j.

5. 5. The method of claim 1, wherein the positioning data includes a respective conditioning signal for at least a portion of the first 2D image, and wherein generating the plurality of first 2D images includes calculating, for the at least a portion of the first 2D image, the respective latent vectors that are taken as input by the model of the first 2D image by applying the global projector to the respective conditioning signals of the first 2D image.

6. 6. The method of claim 5, wherein each conditioning signal has one type from a predetermined set of at least two types.

7. The method of claim 6 , wherein the predetermined set of at least two types includes an image type and a text type.

8. 8. The method of claim 1, further comprising repeating the steps of calculating the first and second latent vectors and generating a next plurality of 2D images, wherein each iteration takes into account the plurality of 2D images generated in a previous iteration to calculate the first and second latent vectors for the next plurality of 2D images.

9. The method of claim 8 , wherein the steps are repeated until a criterion is achieved, the criterion taking into account variations in the second latent vector during the iterations.

10. 10. A computer-implemented method for machine learning a model for use in the generation method of any one of claims 1 to 9, comprising: obtaining a dataset including training samples each including a 2D image, a layout, a viewpoint, and a respective latent vector; training the model based on the acquired dataset; 10. A computer-implemented method for machine learning a model for use in the generation method of any one of claims 1 to 9, comprising:

11. 11. The machine learning method of claim 10, further comprising replacing a predetermined portion of the respective latent vectors of the dataset with latent vectors having predetermined values ​​before or during the training step.

12. The machine learning method according to claim 10 or 11, further comprising the step of training the global projector simultaneously with the step of training the model.

13. 13. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the generation method of any one of claims 1 to 9 and / or the machine learning method of any one of claims 10 to 12.

14. A computer-readable storage medium having the computer program according to claim 13 recorded thereon.

15. 14. A system comprising a processor coupled to a memory, the memory having recorded thereon the computer program of claim 13.

Images