Method, program, storage medium and system for determining machine learning function

Abstract

The present disclosure particularly relates to a computer-implemented method for determining a machine learning function configured for taking an input 3D scene and for outputting one or more camera viewpoints, each camera viewpoint for generating a respective 2D rendering of the 3D scene. The method includes obtaining a library comprising a 3D scene. The method includes forming a first data set for training a first neural network configured to output a camera position based on the library, and forming a second data set for training a second neural network configured to output a camera orientation. The method includes training the first neural network based on the first data set and training the second neural network based on the second data set. Each camera viewpoint output by the machine learning function includes a camera position and a camera orientation. Such a method forms an improved solution for outputting one or more camera viewpoints of a 3D scene.

Description

Technical Field

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems and programs for determining a machine learning function configured to take an input 3D scene and to output one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. Background Art

[0002] Many systems and programs are provided on the market for the design, engineering and manufacture of objects. CAD is an acronym for computer-aided design, for example, it relates to a software solution for designing an object. CAE is an acronym for computer-aided engineering, for example, it relates to a software solution for simulating the physical behavior of a future product. CAM is an acronym for computer-aided manufacturing, for example, it relates to a software solution for defining manufacturing processes and operations. In such computer-aided design systems, a graphical user interface plays an important role in the efficiency of the technology. These technologies can be embedded in a product lifecycle management (PLM) system. PLM refers to a concept across an extended enterprise that helps companies share product data, apply common processes, and utilize company knowledge to develop a business strategy for a product from concept to the end of its life. The PLM solution provided by Dassault Systèmes (trademarks CATIA, ENOVIA, 3DVIA, and DELMIA) provides an engineering center that organizes product engineering knowledge, a manufacturing center that manages manufacturing engineering knowledge, and an enterprise center that allows enterprises to integrate and connect to both engineering and manufacturing centers. Together, the system delivers an open object model that links products, processes, resources to enable dynamic, knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] Interior design increasingly involves creating virtual indoor 3D scenes that represent real rooms. To illustrate these indoor 3D scenes, images can be taken from specific viewpoints of the 3D scene. Automatic methods for determining the viewpoint of a virtual visit of an indoor scene are generally unknown in the literature. The literature tends to study viewpoint inference for single objects.

[0004] Referring to the list of prior art references provided below, the method of reference [1] aims to accelerate viewpoint inference for a single object by learning to infer viewpoints that maximize certain metrics (such as the number of faces of the object's mesh that are visible from the inferred viewpoint). The proposed method strongly relies on the assumption that only a single object is observed. For example, the found viewpoint is determined by finding coordinates on a sphere centered on the object. This necessarily simplifies the problem (finding coordinates on a surface rather than a volume), but cannot be applied to 3D scene viewpoint inference at all for obvious reasons (the size of the search space and restrictions with respect to the room topology). Furthermore, even if some metrics can be computed to determine good viewpoints for a single object, those metrics may be more difficult to define (or even more implicit) for 3D scenes.

[0005] Furthermore, some approaches (e.g., the approach in [2]) attempt to infer an assessment of the aesthetics of an image based on learning from datasets made by users. However, such approaches are apparently not yet mature enough to automatically determine the viewpoint for a virtual visit of an indoor scene. In particular, one of the problems is that the constructed datasets are not operational. Second, this is too different from the field of indoor scenes, and even if these approaches are applied to this field, it is not clear how such approaches can construct relevant datasets (e.g., the problem of random cameras versus user-placed cameras or the problem of grading user renderings). Third, such approaches only give a score for a given image and do not address at all the problem of finding the right camera parameters (e.g., position and orientation) to obtain that image.

[0006] Few rule-based methods can also be considered to define reasonable camera positions (e.g., the camera is placed in front of each center of each wall, with the line of sight looking towards the center of the corresponding wall). However, those methods can easily fail in complex room architectures.

[0007] In the literature that does not use neural networks, the literature includes the method of reference

[10] , which relies on heuristic-based methods such as computing the triangle density of several views and keeping the viewpoints with higher density (assuming that the higher the density, the more information there is in the image). The method of reference

[11] explores a similar approach. However, these methods are more suitable for dataset construction purposes (mainly because they can be very slow).

[0008] Reference [9] proposes a data-driven method for selecting viewpoints for 3D scenes. In this article, the distribution of a set of well-rendered examples is modeled using n probability density functions representing the x, y, and depth positions of pixels of each n semantic category in the example set. Candidate positions are then generated by sampling viewpoints according to the above estimated probability density functions. Afterwards, a submodular maximization-based algorithm is used to select a set of candidate views that approximately matches the entire probability density field as a set. However, this approach has some failure cases and requires a lot of computation (distribution estimation, candidate attribute estimation, and selection). It is not based on deep learning and is more suitable for dataset construction, such as in references

[10] and

[11] .

[0009] The literature also includes reference [8], which has also been published for single objects (and is not applicable to 3D scenes), but attempts to learn the optimal viewpoint from human preferences from human-labeled viewpoints.

[0010] List of prior art references:

[0011] [1] Schelling, M., Hermosilla, P., Vázquez, PP, & Ropinski, T. (2021, May). Enabling Viewpoint Learning through Dynamic Label Generation. In ComputerGraphics Forum (Vol.40, No.2, pp.413-423).

[0012] [2] Talebi, H., & Milanfar, P. (2018). NIMA: Neural image assessment. IEEE transactions on image processing, 27(8), 3998-4011.

[0013] [3]Li,Y.,Bu,R.,Sun,M.,Wu,W.,Di,X.,&Chen,B.(2018).Pointcnn:Convolutionon x-transformed points.Advances in neural information processing systems,31.

[0014] [4]Wang,K.,Lin,Y.A.,Weissmann,B.,Savva,M.,Chang,A.X.,&Ritchie,D.(2019).Planit:Planning and instantiating indoor scenes with relation graphand spatial prior networks.ACM Transactions on Graphics(TOG),38(4),1-15.

[0015] [5]Corso,G.,Cavalleri,L.,Beaini,D.,Liò,P.,& P.(2020).Principal neighbourhood aggregation for graph nets.Advances in NeuralInformation Processing Systems,33,13260-13271.

[0016] [6]Paschalidou,D.,Kar,A.,Shugrina,M.,Kreis,K.,Geiger,A.,&Fidler,S.(2021).Atiss:Autoregressive transformers for indoor scene synthesis.Advancesin Neural Information Processing Systems,34,12013-12026.

[0017] [7]Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,...&Polosukhin,I.(2017).Attention is all you need.Advances in neuralinformation processing systems,30.

[0018] [8]Hartwig,S.,Schelling,M.,Onzenoodt,C.V.,Vázquez,P.P.,Hermosilla,P.,&Ropinski,T.(2022,September).Learning Human Viewpoint Preferences fromSparsely Annotated Models.In Computer Graphics Forum(Vol.41,No.6,pp.453-466).

[0019] [9]Genova,K.,Savva,M.,Chang,A.X.,&Funkhouser,T.(2017).Learning whereto look:Data-driven viewpoint set selection for 3d scenes.arXiv preprintarXiv:1704.02393.

[0020]

[10] Roberts,M.,Ramapuram,J.,Ranjan,A.,Kumar,A.,Bautista,M.A.,Paczan,N.,...&Susskind,J.M.(2021).Hypersim:A photorealistic synthetic dataset forholistic indoor scene understanding.In Proceedings of the IEEE / CVFinternational conference on computer vision(pp.10912-10922).

[0021]

[11] Zhang,Y.,Song,S.,Yumer,E.,Savva,M.,Lee,J.Y.,Jin,H.,&Funkhouser,T.(2017).Physically-based rendering for indoor scene understanding usingconvolutional neural networks.In Proceedings of the IEEE conference oncomputer vision and pattern recognition(pp.5287-5295).

[0022] Against this background, there remains a need for improved solutions for outputting one or more camera viewpoints of a 3D scene. Summary of the invention

[0023] Thus, a computer-implemented method for determining a machine learning function is provided, the machine learning function being configured to take an input 3D scene and to output one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. The method comprises obtaining a library, the library comprising 3D scenes and one or more camera viewpoints for each 3D scene, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. The method comprises forming a first data set based on the library for training a first neural network, the first neural network being configured to obtain a first input representation of the 3D scene and to output information corresponding to a camera position in the 3D scene. The method comprises forming a second data set based on the library for training a second neural network, the second neural network being configured to obtain a second input representation of the 3D scene and to output information corresponding to a camera orientation in the 3D scene. The method comprises training the first neural network based on the first data set and training the second neural network based on the second data set. Each camera viewpoint output by the machine learning function comprises a camera position corresponding to the information output by the first neural network and a camera orientation corresponding to the information output by the second neural network.

[0024] The method may include one or more of the following:

[0025] - forming the first data set comprises, for each 3D scene:

[0026] ○ Determine a set of regularly distributed starting positions on a plan view of the 3D scene;

[0027] o for each camera viewpoint of the 3D scene, determining an offset vector between the camera position of the camera viewpoint and the closest starting position,

[0028] - The first neural network is configured to further take as input a starting position in the 3D scene.

[0029] The information output by the first neural network includes an offset vector between a camera position and the starting position;

[0030] - Each set of said starting positions are substantially equidistant from each other. The distances between said starting positions are substantially the same for each 3D scene.

[0031] - Each set of said starting positions is distributed to form a regular 2D grid on a floor plan of said 3D scene.

[0032] Each starting position corresponds to a respective vertex of the regular 2D grid.

[0033] - said first input representation and / or said second input representation comprises a set of geometrical features of 3D objects of said 3D scene.

[0034] - the first input representation and / or the second input representation comprises a graph comprising nodes each representing a respective object of the 3D scene and arcs each linking respective pairs of nodes. The 3D objects represented by the pairs of nodes are arranged according to a mutual arrangement relationship from a predetermined set of one or more mutual arrangement relationships;

[0035] - the predetermined set of one or more mutual arrangement relationships comprises an adjacency relationship, a proximity relationship, a distance relationship and / or a vertically stacked relationship;

[0036] - the second neural network also takes as input the camera position output by the first neural network;

[0037] -The first neural network has a graph neural network (GNN) model and a multi-layer perceptron (MLP)

[0038] The architecture of the model. The MLP model takes as input the concatenation of the output of the GNN model and the coordinates of the starting position;

[0039] -The second neural network has a graph neural network (GNN) model and a multi-layer perceptron (MLP)

[0040] The architecture of the model, the MLP model takes as input the concatenation of the output of the GNN model and the coordinates of the camera position output by the first neural network;

[0041] - the machine learning function comprises a post-processing block. The post-processing block is configured to select the output one or more camera viewpoints from a set of camera viewpoints determined by applying the first neural network and the second neural network to each of a set of starting positions regularly distributed on a plan view of the input 3D scene; and / or

[0042] -Select one or more camera viewpoints to be output including:

[0043] o For each camera viewpoint, determining the 3D objects of the input 3D scene that are visible in a 2D rendering generated from the camera viewpoint; and

[0044] o Selecting one or more of the determined camera viewpoints such that the 3D objects visible in one or more 2D renderings generated from the one or more selected camera viewpoints represent at least a predetermined percentage of all 3D objects present in the 3D scene.

[0045] A computer program comprising instructions for performing the method is also provided.

[0046] A computer-readable storage medium having a computer program recorded thereon is also provided.

[0047] A system comprising a processor coupled to a memory having a computer program recorded thereon is also provided.The system may include a graphical user interface.

[0048] Also provided is a device comprising a data storage medium having a computer program recorded thereon. The device may form or be used as a non-transitory computer readable medium, such as on a SaaS (Software as a Service) or other server or cloud-based platform, etc. The device may alternatively include a processor coupled to the data storage medium. Thus, the device may form a computer system in whole or in part (e.g., the device is a subsystem of the entire system). The system may also include a graphical user interface coupled to the processor.

[0049] A method of using a machine learning function determined according to the method is also provided. The method of using includes using the machine learning function. The use of the machine learning function includes obtaining an input 3D scene and applying the machine learning function to the obtained input 3D scene, thereby outputting one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the obtained input 3D scene.

[0050] A machine learning function determined according to the method is also provided. The machine learning function is configured to take an input 3D scene and to output one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Non-limiting examples will now be described with reference to the accompanying drawings, in which:

[0052] Figure 1 a flow chart illustrating an example of a method;

[0053] Figure 2 a flow chart showing examples of offline and online phases;

[0054] Figure 3 An example of a GNN model of a first neural network is shown;

[0055] Figures 4 to 8 showing examples of results of machine learning functions;

[0056] Fig. 9 and 10 An example showing the architecture of a machine learning function;

[0057] Fig.11 An example of a group showing a starting position; and

[0058] Fig.12 An example of a system is shown. DETAILED DESCRIPTION

[0059] refer to Figure 1 A computer-implemented method for determining a machine learning function is provided, the machine learning function being configured to take an input 3D scene and to output one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. The method comprises: obtaining S10 a library, the library comprising 3D scenes and one or more camera viewpoints for each 3D scene, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene. The method comprises: forming S20 a first data set for training a first neural network based on the library, the first neural network being configured to obtain a first input representation of the 3D scene and to output information corresponding to a camera position in the 3D scene. The method comprises: forming S30 a second data set for training a second neural network based on the library, the second neural network being configured to obtain a second input representation of the 3D scene and to output information corresponding to a camera orientation in the 3D scene. The method comprises: training S40 a first neural network based on the first data set, and training a second neural network based on the second data set. Each camera viewpoint output by the machine learning function comprises a camera position corresponding to the information output by the first neural network and a camera orientation corresponding to the information output by the second neural network.

[0060] Such an approach forms an improved solution for outputting one or more camera viewpoints of a 3D scene.

[0061] It is noteworthy that the method allows training a function for outputting one or more camera viewpoints. In particular, the method allows training a function based on a library of 3D scenes. In fact, the method comprises forming two data sets (one for each neural network) for training the function. Thus, starting directly from the library of 3D scenes, the method allows the training of a function for outputting camera viewpoints to be performed automatically.

[0062] Furthermore, the use of a first neural network for camera position and a second neural network for camera orientation improves the optimality of the output camera viewpoints. In fact, forming the first data set and the second data set based on a library (including the optimal camera viewpoint for each 3D scene) allows training two neural networks to infer the optimal camera viewpoints. Optimal means that there are a large number of objects in each output camera viewpoint. The method improves the training of each neural network and therefore improves the optimality of the camera viewpoints output by the neural network after training.

[0063] Furthermore, a first neural network considering the camera position and a second neural network considering the camera orientation are particularly relevant. Indeed, these two parameters (camera position and orientation) allow defining a unique viewpoint and the training for predicting these two parameters can be separated. It makes it particularly relevant to use a corresponding neural network for each of these parameters. Additionally, considering a specific neural network for each parameter is a good compromise between the specialization of each neural network and the difficulty of training. Indeed, it allows training neural networks each for capturing the specificity of these parameters in a library of 3D scenes while keeping a reasonable number of neural networks to be trained (i.e. two networks: one for camera position and another for camera orientation).

[0064] The method is computer-implemented. This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Therefore, the steps of the method may be performed fully automatically or semi-automatically by a computer. In an example, the triggering of at least some steps of the method may be performed by user-computer interaction. The level of user-computer interaction required may depend on the level of automation foreseen and balanced with the need to achieve the user's wishes. In an example, the level may be user-defined and / or predefined.

[0065] A typical example of a computer implementation of the method is to perform the method using a system suitable for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI); the memory has recorded thereon a computer program, the computer program including instructions for performing the method. The memory may also store a database. The memory is any hardware suitable for such storage, and may include several physically distinct parts (e.g., one for the program and possibly one for the database).

[0066] The 3D scene library, the first data set and / or the second data set may be stored in a database. "Database" refers to any data (i.e., information) collection organized for search and retrieval (e.g., a relational database, e.g., based on a predetermined structured language, such as SQL). When stored on a memory, a database allows a computer to quickly search and retrieve. The database is indeed constructed to facilitate the storage, retrieval, modification, and deletion of data in conjunction with various data processing operations. A database may consist of files or sets of files that can be decomposed into records, each record consisting of one or more fields. A field is the basic unit of data storage. Users can retrieve data primarily through queries. Using keywords and sorting commands, users can quickly search, rearrange, group, and select fields in many records to retrieve or create reports on specific data aggregations according to the rules of the database management system being used.

[0067] The method typically manipulates modeled (3D) objects. A modeled object is any object defined by data, for example stored in a database. By extension, the expression "modeled object" specifies the data itself. Depending on the type of system, modeled objects may be defined by different kinds of data. For example, the system may be a CAD system, and modeled objects may be defined by corresponding data, so to speak, CAD objects.

[0068] A CAD system further means any system that is at least suitable for designing a modeled object based on a graphical representation of the modeled object (such as CATIA). In this case, the data defining the modeled object includes data that allow the representation of the modeled object. The CAD system can, for example, use edges or lines (in some cases, by faces or surfaces) to provide a representation of the CAD modeled object. Lines, edges or surfaces can be represented in various ways, for example, non-uniform rational B-splines (NURBS). Specifically, a CAD file contains specifications from which geometric shapes can be generated, which in turn allows the generation of a representation. The specifications of the modeled object can be stored in a single CAD file or in multiple CAD files. The typical size of a file representing a modeled object in a CAD system is in the range of one megabyte per component. And a modeled object can typically be an assembly of thousands of components.

[0069] In the context of CAD, a modeled object may typically be a 3D modeled object, for example, representing a product such as a part or an assembly of parts, or possibly an assembly of a product. A "3D modeled object" refers to any object that is modeled by data that allows its 3D representation. The 3D representation allows the component to be viewed from all angles. For example, when represented in 3D, a 3D modeled object can be manipulated and can be rotated about any of its axes, or about any axis in the screen on which the representation is displayed. This specifically excludes 2D icons that are not modeled in 3D. The display of a 3D representation facilitates design (i.e., increases the speed at which the designer statistically completes his task). This speeds up the manufacturing process in industry, as product design is part of the manufacturing process.

[0070] CAD systems can be history-based. In this case, the modeled object is further defined by data including the history of geometric features. The modeled object can actually be designed by a physicist (i.e., designer / user) using standard modeling features (e.g., extrusion, rotation, cutting and / or rounding) and / or standard surface treatment features (e.g., sweeping, blending, lofting, filling, deformation and / or smoothing). Many CAD systems that support such modeling functions are history-based systems. This means that the creation history of the design features is usually saved by a non-cyclic data flow that links the geometric features together via input and output links. The history-based modeling paradigm has been well known since the early 1980s. The modeled object is described by two permanent data representations: history and B-rep (i.e., boundary representation). B-rep is the result of the calculation defined in the history. The shape of the part displayed on the screen of the computer when the modeled object is represented is B-rep (e.g., its tessellation). The history of the part is the design intent. Basically, history collects information about the operations that the modeled object has undergone. B-rep can be saved with history to make it easier to display complex parts. The history can be saved with the B-rep to allow design changes to be made to the part according to the design intent.

[0071] Each 3D scene can represent a real room. Each 3D scene of the library can include one or more 3D objects. Some (e.g., all) of the 3D scenes of the library and / or the input 3D scenes can represent complex rooms, that is, they can each include several objects (e.g., more than ten or one hundred objects) and / or can each be non-rectangular in shape (e.g., having more than four walls and / or including one or more circular walls). Each 3D object of the 3D scene can represent the geometry of a real object positioned (or to be positioned) in a real room represented by the 3D scene. The real object can be manufactured in the real world after its virtual design is completed (e.g., using a CAD software solution or a CAD system). The 3D scenes of the library can each include one or more furniture objects and / or one or more decorative objects. Any furniture object herein can have a furniture function in the room where they are placed. For example, the 3D scenes of the library can each include one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more side panels, one or more bed cabinets, one or more desks and / or one or more wardrobes. Any decorative object herein can have a decorative function in the room where they are placed. For example, a 3D scene of a library can each include one or more ornaments, one or more plants, one or more books, one or more frames, one or more kitchen utensils, one or more mats, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors and / or one or more electronic objects (e.g., a refrigerator, a freezer and / or a washing machine).

[0072] The method may be used in a generative process, which may include, after executing the method, generating one or more camera viewpoints of the room using the machine learning function determined by the method (inference phase). In this case, the method may be included in the generative process (i.e., the formation of the data set and the training of the neural network may be performed during the generative process). Alternatively, the method may alternatively be performed before executing the generative process (i.e., the machine learning function may have been trained when executing the generative process).

[0073] One or more camera viewpoints may be generated for a user, such as an owner of a home in which the room is located. The generation process may include obtaining an input 3D scene representing a room to be arranged (e.g., an input 3D scene provided by a user). The generation process may then include applying the determined machine learning function to the obtained input 3D scene, thereby automatically generating one or more camera viewpoints of the input 3D scene (i.e., one or more viewpoints output by the function). In an example, the generation process may then include recording the generated one or more camera viewpoints in a database that is, for example, accessible online to the user.

[0074] In an example, a generation process can be used for a virtual visit process. For example, the generation process can be included in the virtual visit process, or alternatively can be performed before the virtual visit process (the virtual visit process is therefore performed based on one or more camera viewpoints generated by executing the generation process). The virtual visit process can include determining an image and / or video of a room based on one or more camera viewpoints generated. For example, the virtual visit process can include generating a corresponding 2D rendering (i.e., image and / or video) of each determined camera viewpoint. The corresponding 2D rendering can include renderings of objects, walls, and / or windows / doors visible from the viewpoint. The virtual visit process can then include assembling the generated 2D renderings, such as in the form of a video showing various 2D renderings generated continuously. The virtual visit process can include or be based on repetition of the generation process. For example, the generation process can be repeated for each room in a house or apartment. In this case, the virtual visit process can generate a virtual visit of all rooms in the house or apartment (i.e., including 2D renderings of all rooms).

[0075] The virtual visit process may be included in the real room design process, which may include: after performing the virtual visit process, using the generated 2D rendering to decide whether to acquire one or more new real objects of the room. Each new real object may have a corresponding object located within the input 3D scene (representing the new real object, i.e., a virtual model of the real object). For example, the real room design process may include: before the virtual visit process, a step of positioning the corresponding object representing each new real object within the input 3D scene (e.g., using a virtual directory including virtual representations of the new objects or by creating a corresponding object representing each new object from scratch). The corresponding object representing the new real object may be visible in the generated 2D rendering. The generated 2D rendering can help the user make a selection by showing one or more new objects located within the room (i.e., positioned relative to other objects in the room).

[0076] Alternatively or additionally, the real room design process may include: after performing the virtual visit process, physically arranging the (i.e., real) room so that its design matches the input 3D scene. For example, the room (without one or more new objects) may already exist in the real world, and the real room design process may include positioning one or more new real objects within the already existing room (i.e., in the real world). The new real objects may be located in the same position as the corresponding objects representing them in the input 3D scene. Alternatively, when performing the method, the room may not yet exist. In this case, the real room design process may include building the room based on the provided 3D scene (i.e., starting from scratch and by placing objects in the provided 3D scene). Because the method improves the acquisition of new real objects for the room, the method also improves the positioning of new real objects and / or the construction of the room, and thus improves the productivity of the real room design process.

[0077] The method may include a step of forming a library (library forming step). In this case, obtaining S10 may include a library forming step. The library forming step may include providing a 3D scene and / or a camera viewpoint of the 3D scene. The provision of the 3D scene and / or the camera viewpoint may be performed upon user interaction. For example, each 3D scene may have been created by a user (e.g., using a software solution for creating a 3D scene of a room). The creation of the 3D scene may include defining the walls / doors of the room and positioning the 3D objects within the defined walls. The provision of the camera viewpoint may also be performed by the user. For example, providing each camera viewpoint may include determining the camera position and camera orientation of the camera viewpoint. After the library forming step, obtaining S10 may include recording the calculated library in a database, and steps S20 and S30 may be performed by accessing the recorded library.

[0078] Alternatively, the method may comprise retrieving a library of 3D scenes in a database. The library of 3D scenes may optionally in this case have been obtained according to the library forming step described above (before performing the method). In that case, obtaining S10 may comprise providing access to the library of 3D scenes in the database. The method may use this access to form (during steps S20 and S30) the first and second data sets.

[0079] The forming S20 of the first data set may be performed in any manner. The forming S20 may include determining training samples of the first data set. For example, the forming S20 may include, for each 3D scene of the library, associating a corresponding input representation of the 3D scene with information corresponding to a camera position of each camera viewpoint of the 3D scene (each association corresponding to a corresponding training sample of the first data set). Each training sample of the first data set may include an input representation associated with information corresponding to a camera position.

[0080] The information corresponding to the camera position may include a vector having values ​​representing the coordinates of a point in a plan view, which represents the camera in the plan view of the 3D scene. For example, the coordinates may be defined in a reference coordinate system (e.g., arbitrarily selected). Alternatively, the coordinates may be defined from a starting position in the 3D scene. In this case, the vector may be an offset vector defined between the camera position and the starting position. The 3D scene may include a plan view and a set of starting positions regularly distributed on the plan view. An offset vector may be defined between the camera position and the closest starting position (i.e., between all starting positions in the set).

[0081] Similarly, the formation S30 of the second data set can be performed in any manner and can include determining training samples of the second data set. For example, the formation S30 can include, for each 3D scene of the library, associating the corresponding input representation of the 3D scene with information corresponding to the camera orientation of each camera viewpoint of the 3D scene (each association corresponds to a corresponding training sample of the second data set). The formation S30 can also include: for each 3D scene of the library, associating the corresponding input representation of the 3D scene with information corresponding to the camera orientation of each camera viewpoint of the 3D scene. In this case, each training sample of the second data set may include an input representation associated with information corresponding to the camera orientation and information corresponding to the camera position. When the first data set is formed first, the formation S30 of the second data set can be performed based on the already formed first data set. For example, the formation S30 of the second data set can include adding an association of information corresponding to the camera orientation of the camera viewpoint represented by the training sample to each training sample of the first data set (i.e., in addition to the association of information corresponding to the camera position already included in the first data set).

[0082] In an example, the method may form a first data set and a second data set through the same 3D scene of the library (i.e., each data set may include training samples determined for the same 3D scene), for example, through all 3D scenes of the library. In this case, the first and second data sets may take the form of a single table. For example, the single table may include a row representing each training sample and a column representing the value of each training sample. For example, a single table may include a first column for input representation, a second column for camera position, and a third column for camera orientation. In this case, the first data set may be formed by the first column and the second column, and the second data set may be formed by the first column, the second column, and the third column. Alternatively, the method may form a first data set based on a first subset of the 3D scenes of the library, and may form a second data set based on a second subset of the 3D scenes of the library. In this case, each subset may include 3D scenes shared with other subsets in the library, and other 3D scenes not shared with other subsets. In this case, the two data sets may each take the form of a corresponding table, or alternatively may also be combined in a single table using, for example, null values, for example, null values ​​indicating that the 3D scene does not exist in the two data sets.

[0083] The camera orientation may be a description of the direction in which the camera is pointing in the 3D scene space. The camera orientation may be defined relative to a reference coordinate system (e.g., including three vectors). The information corresponding to the position of the camera may include angle values, each of which represents a rotation between the direction pointed by the camera and a corresponding vector of the reference coordinate system.

[0084] In the example, the second neural network also takes as input the camera position output by the first neural network. In this case, each training sample of the second data set may also include information corresponding to the camera position of the camera viewpoint (in addition to the information corresponding to the camera orientation). Forming S30 may include, for each 3D scene of the library, associating the corresponding input representation of the 3D scene with information corresponding to the camera position of each camera viewpoint of the 3D scene. Each training sample of the second data set may include an input representation associated with information corresponding to the camera orientation and the camera position.

[0085] In an example, the forming S20 of the first data set and the forming S30 of the second data set may be performed simultaneously. For example, the first data set may correspond to a portion of the second data set (the first data set includes each training sample of the second data set, but has no information about the camera orientation of each training sample). In this case, the forming S30 of the second data set may be performed as described above, and the forming S20 of the first data set may include extracting the portion of the second data set corresponding to the first data set (i.e., for each camera viewpoint, the input representation is associated with information only about the camera position).

[0086] In an example, during the inference phase, the first neural network may be configured to further take a starting position as input. The starting position may be in a set of starting positions regularly distributed on a plan view of the 3D scene taken as input by the first neural network. In this case, forming S20 may include: for each 3D scene, determining S21 a set of starting positions regularly distributed on a plan view of the 3D scene. The starting positions of each group may be substantially equidistant from each other. For each 3D scene, the distances between the starting positions may be substantially the same. For example, the distances between adjacent starting positions may be lower than 3 meters and / or higher than 50 centimeters, for example, between 75 centimeters and 1.50 meters (e.g., approximately 1 meter). The starting positions of each group may be randomly distributed (e.g., in an accessible space of a room, i.e., without an object). Alternatively, the starting positions of each group may be regularly distributed so as to form a regular pattern (e.g., a pattern including concentric circles or a grid pattern) on a plan view of the 3D scene. For example, the starting positions of each group may be distributed to form a regular 2D grid. Each starting position may correspond to a corresponding vertex of the regular 2D grid. During the inference phase, the information output by the first neural network may include an offset vector between the camera position and the starting position taken as input.

[0087] After determining S21, forming S20 may include: for each camera viewpoint of the 3D scene, determining S22 an offset vector between a camera position of the camera viewpoint and a closest starting position. Determining S22 the offset vector may include: calculating a distance between the camera position and each starting position in the group, the closest starting position being the starting position with the smallest distance. Then, determining S22 the offset vector may include determining a coordinate of the offset vector in a reference coordinate system (each coordinate represents a distance between the camera viewpoint and the closest starting position along a corresponding direction of the reference coordinate system).

[0088] The group of starting positions and the offset vectors improve the training of the neural networks and therefore improve the inference of these neural networks on the camera viewpoint. In fact, the group of starting positions allows the abstraction of local patterns and decontextualized learning. The group of starting positions allows to consider parts of the 3D scene, and the offset vectors allow to consider that each ground truth applies to the corresponding part of the 3D scene. Each starting position represents a perimeter within which each neural network is trained to predict the values ​​present in the training set. Therefore, they allow to consider the fact that the ground truth applies to the corresponding part of the 3D scene, and other ground truths may exist at other points in the 3D scene.

[0089] Any input representation of a 3D scene may include a set of geometric properties of (e.g., all) 3D objects of the 3D scene. For each 3D object, the geometric properties may include the position, orientation, and / or size of the 3D object. The position may include the coordinates of the center of the 3D object in a reference coordinate system. The orientation of the 3D object may be defined relative to a reference position. For example, the orientation may include angle values, each of which represents a rotation of a 3D object positioned in the 3D scene relative to a reference position of the 3D object along a corresponding vector of the reference coordinate system.

[0090] In an example, any input representation of a 3D scene may include voxelization of the 3D scene. Voxelization may include a regular grid of voxels on the 3D scene space. Each voxel may include information about the presence of an object in the area defined by the voxel (e.g., 0 when there is no object, otherwise 1). Voxelization may provide a set of geometric properties (i.e., position, orientation, and / or size) for each 3D object. Alternatively, any input representation of a 3D scene may include a graph including nodes each representing a corresponding object of the 3D scene and arcs each linking corresponding pairs of nodes. The 3D objects represented by a pair of nodes may be arranged according to a mutual arrangement relationship in a predetermined set of one or more mutual arrangement relationships. Each edge may include a label indicating the mutual arrangement relationship between the two objects represented by the nodes linked by the edge. Each node of the graph may include a set of geometric features (i.e., position, orientation, and / or size) of the 3D object represented by the node.

[0091] In an example, a predetermined set may include an adjacency relationship. When two objects are spaced apart from each other by less than a first predetermined distance (e.g., 5 cm), the two objects may be arranged according to the adjacency relationship. Alternatively or additionally, a predetermined set may include a proximal relationship. When two objects are spaced apart from each other by more than a first predetermined distance and less than a second predetermined distance (e.g., 50 cm), the two objects may be arranged according to the adjacency relationship. Alternatively or additionally, a predetermined set may include a distance relationship. When two objects are spaced apart from each other by more than a second predetermined distance, the two objects may be arranged according to the distance relationship. For each relationship, the distance may be the distance between the bounding boxes representing the two objects in the 3D scene (e.g., the distance defined along one of the axes of one of the two objects). Alternatively or additionally, a predetermined set may include a vertical stacking relationship. When two objects are positioned above and below each other (e.g., a lamp on the ceiling above a table), the two objects may be arranged according to the adjacency relationship.

[0092] In an example, the method may include determining corresponding input representations of one or more (e.g., all) 3D scenes of the library. The determination of the input representation may be performed during the formation of the data set. The determination of the corresponding input representation of the 3D scene (representation formation step) may include determining geometric properties of 3D objects of the 3D scene. For example, the representation formation step may include determining the position, orientation, and / or size information (optionally, a bounding box) of each 3D object of the 3D scene. In an example, the representation formation step may also include determining a graph of the 3D scene. Determining the graph may include determining each node and each arc of the graph based on the objects of the 3D scene. Each node of the graph may include the position, orientation, and / or size information (optionally, a bounding box) determined for the object of the 3D scene represented by the node in the graph.

[0093] Alternatively, the library may include a corresponding input representation for each 3D scene. In this case, the method may include retrieving the corresponding input representation of the 3D scene during the formation of each data set (steps S20 and S30). The input representations included in the 3D scene library may optionally have been obtained in this case according to the above-mentioned representation formation step (before performing the method).

[0094] The acquisition of the library (step S10), the formation of the first and second data sets (steps S20 and S30) and / or the training of the first and second neural networks (step S40) may be performed during an offline phase. In this case, the acquired library, the first and second data sets and / or the determined machine learning functions (including the trained first and second neural networks) may be recorded in a database, for example, during this offline phase. After this offline phase, the determined machine learning functions may be used during an online phase (inference phase). For example, the determined machine learning functions may be used by client computers, each client computer being configured to provide an input 3D scene to a server, the server being able to access the database and being configured to apply the machine learning function to the provided input 3D scene, and providing the camera viewpoints output by the function for each input 3D scene to the client computer, the client computer providing the input 3D scene (or, for example, a 2D rendering also generated by the server for each camera viewpoint).

[0095] The method may include (e.g., periodically) updating the first and second data sets. For example, the library may be extended by a user providing newly created 3D scenes with camera viewpoints, and the method may include updating the first and second data sets to include these newly created 3D scenes (e.g., by adding the newly created 3D scenes and their corresponding camera viewpoints to the first and second data sets). In this case, the method may include, after the update, retraining the first neural network based on the updated first data set, and retraining the second neural network based on the updated second data set.

[0096] Reference now Figures 2 to 12 The method is discussed further.

[0097] The method may be composed in a fully automatic way to infer viewpoints of a 3D room that may be used to define a virtual visit of the 3D room. In case of inferring viewpoints of a virtual visit of a 3D scene, the method may allow to present the complexity of the 3D scene, i.e. the objects of the scene, information about their relative positioning may be given from the user's perspective, and all this information may be given with a minimal set of inferred viewpoints.

[0098] The method may include a method of training a neural network (training phase) to infer 3D viewpoints of a 3D scene (or 3D room) from a library (dataset) of 3D rooms with some (few) 3D viewpoint annotations (camera viewpoints for each 3D room). Optionally, the method may also include a selection step to filter those viewpoints to select a minimum set of viewpoints that can be used to build a virtual tour of the 3D room (inference phase).

[0099] The method comprises a training phase. The training phase comprises forming S20 a first data set. Forming S20 comprises, for each 3D room in the data set, optionally defining S21 a set of reasonable camera positions (starting position set), optionally finding S22, for each ground truth viewpoint position (camera position) present in the data set of the 3D room, the closest reasonable camera position from the ground truth viewpoint position, and storing the offset vector from the reasonable camera position to the ground truth viewpoint position. Then, the training phase comprises training S40. Training S40 comprises optionally providing a first neural network, which takes the 3D room (e.g., a 3D graphical representation of the 3D room) and the initial camera position as input and outputs a 3-dimensional vector (the offset vector from the initial camera position). Then, training S40 comprises applying a learning algorithm to the first neural network so as to train it to output the correct offset vector given each 3D room in the data set and the corresponding initial camera position. Training S40 also includes optionally providing a second neural network that takes as input the 3D room (e.g., a 3D graphical representation of the 3D room) and the camera position and outputs a 3-dimensional vector (e.g., one vector for each angle of orientation). Training S40 then includes applying a learning algorithm to the second neural network so as to train it to output the correct ground truth orientation vector given each 3D room and the corresponding ground truth viewpoint position.

[0100] The inference phase (method of use) includes, given a 3D room, defining a set of plausible camera positions. The inference phase includes applying an offset vector inferred by a first trained neural network for each camera position in the set of plausible camera positions to each corresponding camera position in the set of plausible camera positions. The inference phase includes inferring a heading vector for each new camera position by a second trained neural network. The inference phase includes applying a post-processing step to maintain a minimal set of inferred viewpoints that see a particular number of objects relative to the total number of objects that can be seen from each inferred viewpoint.

[0101] The method allows training a function for automatically inferring a set of viewpoints that can be linked together to build a virtual tour of a given 3D room, maximizing the visibility of key elements in the room (i.e., primarily objects therein). The method is fully automatic, suitable for any room type, and uses a very light representation of the room.

[0102] The method belongs to the field of 3D viewpoint inference of 3D rooms. The purpose of automatic viewpoint inference is to calculate a set of camera viewpoints for a given room, which gives the camera position and orientation that can be used to construct a virtual visit of the given room. The camera viewpoint may include a camera position and a camera orientation. The camera position and the camera orientation may be external parameters of the camera viewpoint. The camera viewpoint may also include intrinsic parameters (e.g., magnification, rendering window width, rendering window length, rendering color properties). Generating a 2D rendering for a given camera viewpoint may take into account the external parameters and internal parameters of the given camera viewpoint. A virtual visit is a navigation between viewpoints inferred for a given room. The goal of a virtual visit is to observe the maximum number of objects in the scene.

[0103] The method may be included in a general process that may include two phases. In a first phase, the general process may include constructing and training two models, one for viewpoint position inference and the other for viewpoint orientation inference (i.e., performing the method, also referred to as an offline phase). In a second phase, the general process may include viewpoint inference for any room based on the two trained models (also referred to as an online phase).

[0104] refer to Figure 2 , we now discuss a flowchart of an example of offline and online phases. This figure shows the distinction between the online and offline phases of a general process.

[0105] The offline stage S110 aims to produce a model (neural network) that implements the following tasks: (i) viewpoint position inference, (ii) viewpoint orientation inference for a given room based on the proposed learning scheme for a given training dataset. The training dataset is a dataset consisting of 3D rooms, each with one or more viewpoints. These rooms are considered to be well-arranged and of good viewpoint quality (rendering is purchased by the user). The first neural network (also called the viewpoint position inference model) takes the 3D room and the starting position of the dataset as input, and learns to infer a 3D vector (called offset) so that the 3D translation corresponding to the 3D prediction vector applied to the starting position matches the viewpoint position of the dataset for the 3D room. The second neural network (also called the viewpoint orientation inference model) takes the 3D room of the dataset and the camera position for the viewpoint position of the 3D room from the dataset as input, and learns to infer the viewpoint orientation of the dataset for the 3D room.

[0106] Given a 3D room, the online stage S120 comprises defining a set of starting positions. For each position, the online stage S120 comprises applying the trained viewpoint position inference model to infer the offset and obtain a predicted position. Then, for each predicted position, the online stage S120 comprises using the viewpoint orientation inference model to infer the correct orientation of each predicted position. Then, the online stage S120 comprises optionally applying post-processing steps to keep a minimum set of viewpoints that ensure a certain coverage of the 3D room in the corresponding rendering and construct a virtual visit with them.

[0107] The offline phase S110 is now discussed in more detail.

[0108] The library consists of 3D rooms, each with one or more viewpoints. These rooms can be considered well laid out and with good viewpoint quality (renderings can be selected and purchased by the user). The number of viewpoints per room may indeed be unbalanced with most 3D rooms having only one viewpoint. The method is designed in such a way that even from an unbalanced dataset with only one viewpoint per room, it can learn to generate several viewpoints for a single room. The elements of the first and / or second dataset may correspond to the pair of 3D room representations and the viewpoints of the pair of 3D rooms during the next part (referred to as ground truth).

[0109] The offline phase S110 comprises the determination S111 of an input representation of the 3D scene. The method may use any representation of a 3D room. For example, using a point cloud representation, the method may use a backbone to compute a 3D room representation. With reference to the list of prior art references provided above, this may be performed according to the teachings of reference [3]. In the example of this embodiment, the method uses a very compact scene graph representation based on the PlanIT idea. With reference to the list of prior art references provided above, this may be performed according to the teachings of reference [4]. The graph structure for the scene representation is constructed as described below.

[0110] Any scene graph consists of:

[0111] A list of nodes, each node corresponds to a 3D_Elem in the room with the following properties:

[0112] ○Node ID;

[0113] ○3D bounding box of 3D_Elem (length, width, height);

[0114] ○3D_Elem's transformation matrix (gets the absolute position and orientation of 3D_Elem in the 3D room);

[0115] ○ a list of input edges; and

[0116] ○ Output a list of edges.

[0117] A list of edges (undirected) representing the relationships between 3D_Eems in the room, each with the following properties:

[0118] ○ Edge ID.

[0119] The method may obtain a scene graph from an already existing original scene graph (computed prior to the method), and may include extracting a scene graph from an already existing initial scene graph. Extraction of the scene graph may include removing any semantic information in the graph (e.g., no category attributes for nodes and edges). This results in a very light representation, as it contains less information than the original 3D room representation. It also allows the model to strongly generalize furniture categories and room types.

[0120] For each input representation, determining S111 may include applying normalization to each node attribute for stability purposes. The method may include determining the applied normalization factor relative to the dimensions of the room dimensions as follows:

[0121] Let us define x0, y0, z0, x1, y1, z1 as the minimum and maximum coordinates of the room along each axis and v = (x1-x0) + (y1-y0) + (z1-z0) as the volume of the 3D room. The method may include normalizing the position of each object and the ground truth camera viewpoint by the following formula: p = (p-p0) / v. For the x coordinate, it becomes: x = (x-x0) / v. Scale the bounding box parameters of each object by 1 / v.

[0122] The offline phase S110 comprises the formation of a dataset S112. The strategy of learning to infer several viewpoints of a given 3D room based on an unbalanced (or imbalanced) dataset relies on learning optimal local viewpoints and is described in detail below. An unbalanced dataset means that there is no dataset with all viewpoint annotations for all rooms, i.e., the dataset may include one or more viewpoint annotations for each 3D room. This also means that the method can learn to infer the optimal set of viewpoints for a virtual visit of a given 3D room (according to the defined goal), even if the number of viewpoints in the dataset does not cover the entire room complexity (i.e., see a certain percentage of objects in the room).

[0123] For each room, forming S211 comprises defining a regular 2D grid to determine a set of starting points. Note that this represents a convenient way to obtain the set of starting points, but the method may rely on other ways of obtaining them (e.g., randomly, in the circulation area of ​​the room). The step size on each axis depends on the size of the room. The method may, for example, choose the following sizes as step sizes:

[0124] Sx=Lx / N, where Lx is the room size along the x-axis, N is the selected parameter, and Sx is the step size on the x-axis.

[0125] Sy=Ly / M, where Ly is the room size along the y-axis, M is the selected parameter, and Sy is the step size on the x-axis.

[0126] Axes x and y may be orthogonal axes in which the room is defined.

[0127] After defining N and M (e.g., N=M=5) and obtaining an M*N number of starting points, determining the set of starting positions may include filtering the vertices of the defined regular 2D grid to keep only vertices within the 3D space of the room (defined by the floor plan in this example).

[0128] Forming S112 includes determining an offset vector for each camera viewpoint. This selection of the offset vector for the learning phase of the position inference model is now discussed.

[0129] During the learning phase, determining the offset vector includes determining the ground truth camera position (the camera position of the camera viewpoint in the library) and searching for the closest starting position (based on x and y coordinates) from the vertices of the mesh retained after filtering. Once found, the determination includes calculating the 3D offset vector from the starting position to the ground truth camera position, considering that the offset vector has a base height of H (e.g., H=1700mm).

[0130] The x and y coordinates of the offset vector are normalized (in the absolute system) by Sx and Sy, respectively. The z coordinate is normalized by the room height and is limited to the [-0,5; 0,5] domain. In the event that the x and y offset vector coordinates are not in the finite [-0,5; 0,5] domain, the method may include applying the same scaling factor to them such that max(llxll,llyll) = 0.5.

[0131] The offline stage S110 includes training S113 of a first neural network (position inference model). The first neural network may have an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model. The MLP model may take as input a concatenation of the output of the GNN model and the coordinates of the starting position. Since the method uses a scene graph representation for 3D rooms, the first neural network may be a graph neural network (GNN) model to obtain a meaningful embedding for each 3D room. The embedding is then concatenated with the starting position coordinates and fed into a multi-layer perceptron model (MLP). The output of the multi-layer perceptron (MLP) includes three coordinates of the offset vector.

[0132] In this example, the method uses the PNAConv architecture. Referring to the list of prior art references provided above, the use of the PNAConv architecture can be performed according to the teachings of reference [5] (in the prior art of GNN architecture). The method includes based on Figure 3 The PNAConv block shown creates a unique network to obtain scene graph embeddings.

[0133] Figure 3 The computation of the new node representation with the aid of the PNAConv architecture is shown. The node of interest is node 110 and its neighbor (linked by edge 121 ) is node 120 .

[0134] The architecture used is a combination of four PNAConv with a dimension of seven channels per node, where a batch normalization layer and a ReLU activation are placed after each of them. The final graph representation is obtained by global pooling (summing) over the node features.

[0135] MLP130 has three linear layers with the following [input, output] dimensions: [7+3,5], [5+3,3], [3+3,3]. The +3 specified at each layer corresponds to the concatenation of the input position to the output of the previous layer and can be viewed as a skip connection mechanism. ReLU activation is applied on each non-final layer output. The activation function of the final output layer is f(x)=Sigmoid(x)-0.5 to match the target [-0.5; 0.5] domain.

[0136] This is an example implementation and all blocks can be adjusted by changing the number and type of layers, channel sizes, activation layers, pooling layers, etc. The main idea is to have a first neural network that is able to infer a high-level graph representation and a second neural network that infers from this high-level representation a 3D vector representing an offset vector and a 3D vector representing an input position.

[0137] The training loss can measure the difference between the predicted offset vector and the ground truth offset vector. Many different losses can be used, such as the well-known L1 or L2 losses. In the example of this embodiment, the L1 loss is used.

[0138] After this step, the method obtains a trained location inference model.

[0139] The offline stage S110 includes training S113 of a second neural network (heading inference model). The second neural network may have an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model. The MLP model may take as input a cascade of the output of the GNN model and the coordinates of the camera position output by the first neural network. The heading inference network also uses a scene graph representation of the 3D room. Therefore, the architecture is similar to the position inference model. The first GNN is used to obtain a meaningful embedding of the 3D room. This embedding is then concatenated with the 3 coordinates of the camera position under consideration and fed into a multi-layer perceptron model (MLP). The output of the MLP is the 3 coordinates of the heading vector. During training, the camera position considered as input by the second neural network for its heading prediction is the ground truth position. During the inference stage, the second neural network takes as input the predicted position obtained from the prediction of the first neural network.

[0140] The architecture used is a combination of four PNAConv with a dimension of seven channels per node, where a batch normalization layer and a ReLU activation are placed after each of them. The final graph representation is obtained by global pooling (summing) over the node features.

[0141] The MLP has two linear layers with the following [input, output] dimensions: [7+3, 5], [5, 3]. The +3 specified in the first layer corresponds to the concatenation of the input position to the output of the previous layer. A ReLU activation is applied on the output of the first layer.

[0142] For the first neural network, this is an example implementation and all blocks can be adjusted by changing the number and type of layers, channel sizes, activation layers, pooling layers, etc. The main idea is to have a first neural network that is able to infer a high-level graph representation and a second neural network that infers from this high-level representation a 3D vector representing the orientation vector and a 3D vector representing the input position.

[0143] The training loss can measure the difference between the predicted heading vector and the ground truth heading vector. Many different losses can be used, such as cosine or L2 loss. Cosine loss can be used in examples of this embodiment.

[0144] After this step, the method obtains a trained orientation inference model.

[0145] The online phase S120 is now discussed.

[0146] Given a 3D room, the online phase S120 comprises a first step S121. The first step S121 comprises first extracting its graphical representation (e.g., this step can be performed during the offline phase as described above). Then, the first step S121 comprises calculating a set of starting (input) positions (e.g., this step can also be performed during the offline phase as described above).

[0147] Then, the online stage S120 comprises an inference step S122. For a given 3D scene, the online stage S120 comprises determining a set of camera viewpoints by applying the first neural network and the second neural network to each of a set of starting positions regularly distributed on a plan view of the input 3D scene. The first neural network is applied first. The inference step S122 comprises using the trained position inference model (the first neural network) to infer a final position for each starting position in the calculated set, the final position being obtained by adding the inferred offset vector of the starting position to the starting position, thereby obtaining an inferred set of final positions.

[0148] Then, the second neural network is applied. For each final position in the obtained set of inferred final positions, the inference step S122 infers the orientation vector (camera orientation) by applying the trained orientation inference model (second neural network) to the final position, thereby obtaining a set of camera viewpoints (each camera viewpoint consists of a final position and an orientation vector). The method obtains the set of camera viewpoints by applying the first neural network and the second neural network to each starting position in a set of starting positions regularly distributed on a plan view of the input 3D scene.

[0149] Even though the results obtained at the previous steps may have been used to define the virtual visit, the online step may also include a post-processing step to avoid too much redundancy in the suggested viewpoints. This post-processing step may be useful, in particular when the number of starting positions is quite high (e.g., higher than 5), in order to improve the user experience (e.g., avoid too much redundancy in the suggested viewpoints). Optionally, the machine learning function may therefore include a post-processing block. The post-processing block may be configured to select one or more camera viewpoints for output from a set of camera viewpoints determined by applying the first neural network and the second neural network to each starting position in a set of starting positions regularly distributed on a plan view of the input 3D scene.

[0150] Selection of one or more camera viewpoints for output may be performed so as to optimize the number of objects visible in the camera viewpoints. For example, the selection may include determining, for each camera viewpoint, 3D objects of the input 3D scene that are visible in a 2D rendering generated from the camera viewpoint. The selection may then include selecting one or more of the determined camera viewpoints so that the 3D objects visible in the one or more 2D renderings generated from the one or more selected camera viewpoints represent at least a predetermined percentage of all 3D objects present in the 3D scene. The visibility of an object in a 2D rendering may refer to the visibility of vertices or faces of a bounding box of the object in the 2D rendering. Alternatively, an object may be visible in a 2D rendering when at least one of the following criteria is met:

[0151] The proportion of pixels in the 2D rendering (image) corresponding to the object is greater than a threshold (eg, 1% of the number of pixels in the 2D rendering); and / or

[0152] • The proportion of the object's faces visible from the viewpoint is greater than a threshold (eg, 10%).

[0153] In order to select the minimum number of predicted viewpoints for a given 3D room, the function may include a post-processing step that aims to select the minimum number of viewpoints that present the best coverage that can be obtained from the inferred viewpoint group. The definition of coverage is discussed below. The post-processing step can be performed by executing the following example algorithm to select the inferred viewpoints. In a first step, the algorithm may include instructions for storing the group of objects seen for each viewpoint and for storing a list of objects seen by all viewpoints together (e.g., in a database). The algorithm may then include instructions for sorting the viewpoint list in descending order relative to the number of objects seen for each viewpoint. The algorithm may then include instructions for adding the first element of the list to the selected viewpoint list and for removing it from the viewpoint list, now referred to as the remaining viewpoints. The algorithm may then include instructions for the following operations: when the number of objects seen in the selected viewpoint list is lower than a certain percentage of the list of objects seen by all viewpoints (calculated in the first step), the following steps are performed: traverse all lists of the remaining viewpoints and select one that contains the largest number of unseen objects relative to the list of objects visible from the group of the selected viewpoint, and add it to the selected viewpoint list and remove it from the remaining viewpoint list.

[0154] Now let's discuss examples of the results of machine learning functions.

[0155] The results are analyzed using the following metrics: a first metric called relative coverage and a second metric called total coverage. Relative coverage is the ratio between the number of objects visible from the selected inference viewpoint and the total number of objects visible from the inference viewpoint. Total coverage is the ratio between the number of objects visible from the selected inference viewpoint and the total number of objects in the scene. The results show the efficiency of the method.

[0156] The network was trained on about 50k 3D bedrooms and, as a proof of generalization, has been evaluated on 200 living rooms to obtain the following results. The results demonstrate that the machine learning function is relevant to automatically infer the viewpoints of any given 3D room. In particular, the machine learning function provides the possibility to infer the minimum number of viewpoints to observe the main furniture of the room (a post-processing step). The average number of objects per scene is 72.16. The average number of selected renderings is 2.07. The average relative coverage is 0.95. The average total coverage is 0.74. Figures 4 to 8 2D renderings obtained from the camera viewpoint inferred by the function (inference step) are shown. These results have been obtained for different room types for a model trained on 50k 3D bedrooms. Specifically, Figure 4 A first living room used in this first example is shown, and Figure 5 Three 2D renderings obtained from three camera viewpoints inferred from the function of this first living room are shown. Figure 6The bedroom used in this second example is shown, and Figure 7 Three 2D renderings obtained from three camera viewpoints inferred from the function of the bedroom are shown. Figure 8 A second living room used in this second example is shown, along with two 2D renderings obtained from two camera viewpoints inferred by functions of the second living room.

[0157] The first data set and / or the second data set may have the following characteristics: The first data set and / or the second data set may include an average area greater than 10m 2 and / or less than 50m 2 (For example, the average area is 16,7m 2 ). The first dataset and / or the second dataset may include training samples of 3D scenes with an average number of objects per scene higher than 10 and / or lower than 50 (e.g., the average number of objects per scene is 32.2). The first dataset and / or the second dataset may include training samples of 3D scenes with an average total coverage higher than 0.25 and / or lower than 0.75 (e.g., 0.51). Optionally, the camera position may be defined in height (e.g., 2m) and may always be in the room boundary. For example, the method may include filtering 3D scenes from the library with camera positions that do not meet those criteria.

[0158] Fig. 9 An example of the architecture of a machine learning function is shown. The machine learning function includes a first neural network (position model) and a second neural network (rotation model). The first neural network has an architecture that includes a graph neural network (GNN) model 210 and a multi-layer perceptron (MLP) model 212. The MLP model 212 takes as input the output of the GNN model 210 and a cascade 211 of the coordinates of the starting position. The MLP model 212 outputs an offset vector 215. The final position 213 is calculated by adding the output offset vector 213 to the initially considered starting position 214. The first neural network takes as input a graphic of the room 200 and an initial position and predicts the position of the camera 213. It is trained using an L1 loss and attempts to learn f(G,pos init ), so that pos pred =pos init +f(G,pos init ).

[0159] The second neural network has an architecture including a graph neural network (GNN) model 220 and a multi-layer perceptron (MLP) model 222. The MLP model 222 takes as input a cascade 221 of the output of the GNN model 220 and the coordinates of the camera position output by the first neural network (i.e., the final position 213, which is the result of adding the offset vector 213 output by the first neural network to the initial starting position 214). The second neural network 220 takes as input the graph and the position to predict the rotation of the camera. It is trained using cosine distance (1-cosine similarity) and attempts to learn h(G,pos) such that rot pred =h(G,pos).

[0160] Fig.10 An alternative example of the architecture of the function is shown. In this alternative, an alternative scene representation is used. Instead of a graph, the alternative scene representation is an unordered collection of 3D_Elems (i.e., object bounding boxes and positions). For example, each 3D_Elem can be represented as:

[0161] its 3D bounding box (length, width, and height); and

[0162] ● Its transformation matrix (to obtain the absolute position and orientation of 3D_Elem in the 3D room).

[0163] Therefore, the neural network can be based on a Transformer architecture instead of a GNN to obtain a scene representation that is position-concatenated with both a position inference model and an orientation inference model. Fig.10 Such a converter is shown.

[0164] In the figure, PE 310 is a positional encoding for projecting each scalar into a higher dimensional space. Referring to the list of prior art references provided above, the positional encoding can be performed according to the teachings of reference [6]. x, y, z represent the position of the center of the bounding box, l, w, h represent its size and θ represents the rotation around the z-axis. The transformer can be implemented as in https: / pytorch.org / docs / stable / modules / torch / n / modules / (this can be performed according to the teachings of reference [7]).

[0165] Fig.11An example of a group of starting positions 410, 420 is shown. The group of starting positions 410, 420 is regularly distributed on a plan view 400 of a 3D scene taken as input by a first neural network. The groups of starting positions 410, 420 are substantially equidistant from each other. The first neural network is configured to take as input, for each starting position, an input representation of the 3D scene and a starting position in the 3D scene, and output an offset vector between the camera position and the starting position. For example, for starting position 420, the first neural network is configured to take as input the input representation of the 3D scene and the starting position 420, and output an offset vector 430 between the camera position 440 and the starting position 420. The method can infer a camera position 440 inferred by the first neural network for the starting position 420 by adding the output offset vector 430 to the starting position 420 taken as input.

[0166] Fig.12 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0167] The client computer of the example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, a random access memory (RAM) 1070 also connected to the bus. The client computer is also provided with a graphics processing unit (GPU) 1110, which is associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device such as a hard disk drive 1030. Mass storage devices suitable for tangibly implementing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks. Any of the foregoing may be supplemented or incorporated by a specially designed ASIC (Application Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a tactile device 1090, such as a cursor control device, a keyboard, and the like. A cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. In addition, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a plurality of signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, the buttons of which are used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive board and / or a sensitive screen.

[0168] The computer program may include instructions executable by a computer, the instructions including means for causing the above-mentioned system to perform the method. The program may be recorded on any data storage medium, including the memory of the system. The program may be implemented, for example, in a digital electronic circuit, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, such as a product tangibly implemented in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing an instruction program to perform the functions of the method by operating on input data and generating output. Therefore, the processor may be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to send data and instructions to the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in an assembly or machine language if necessary. In any case, the language may be a compiled or interpreted language. The program may be a fully installed program or an update program. The application of the program on the system results in any case of instructions for executing the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, and the server communicates with one or more clients via a network. In this case, the processing unit executes instructions included by the program, thereby causing the method to be performed on the cloud computing environment.

Claims

1. A computer-implemented method for determining a machine learning function configured to take an input 3D scene and output one or more camera viewpoints, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene, the method comprising: ● obtaining ( S10 ) a library, the library comprising 3D scenes and one or more camera viewpoints for each 3D scene, each camera viewpoint being used to generate a corresponding 2D rendering of the 3D scene; ●Based on library: o forming (S20) a first data set for training a first neural network, the first neural network being configured to take a first input representation of a 3D scene and output information corresponding to a camera position in the 3D scene; as well as o forming (S30) a second data set for training a second neural network, the second neural network being configured to obtain a second input representation of a 3D scene and output information corresponding to a camera orientation in the 3D scene; as well as ● training (S40) the first neural network based on the first data set, and training the second neural network based on the second data set, Each camera viewpoint output by the machine learning function includes a camera position corresponding to information output by the first neural network, and a camera orientation corresponding to information output by the second neural network.

2. The method according to claim 1, wherein: For each 3D scene, forming (S20) the first data set comprises: ●determine (S21) a set of regularly distributed starting positions on the plane map of the 3D scene; for each camera viewpoint of the 3D scene, determining (S22) an offset vector between the camera position of the camera viewpoint and the closest starting position, The first neural network is configured to further take a starting position in the 3D scene as input, and the information output by the first neural network includes an offset vector between a camera position and the starting position.

3. The method according to claim 2, wherein: Each group of starting positions are substantially equidistant from each other, and the distances between the starting positions are substantially the same for each 3D scene.

4. The method according to claim 2 or 3, wherein: Each set of the starting positions is distributed to form a regular 2D grid on the plan view of the 3D scene, and each starting position corresponds to a respective vertex of the regular 2D grid.

5. The method according to any one of claims 1 to 4, wherein: The first input representation and / or the second input representation comprises a set of geometric features of 3D objects of the 3D scene.

6. The method according to claim 5, wherein: The geometric features include location, orientation and / or size information. Optionally, the geometric features include a plurality of bounding boxes, each of the plurality of bounding boxes representing a corresponding 3D object of the 3D scene.

7. The method according to claim 5 or 6, wherein: The first input representation and / or the second input representation comprises a graph comprising a plurality of nodes each representing a corresponding object of the 3D scene and a plurality of arcs each linking corresponding pairs of nodes, the 3D objects represented by the node pairs being arranged according to a mutual arrangement relationship, the mutual arrangement relationship being selected from a predetermined set comprising one or more mutual arrangement relationships.

8. The method according to claim 7, wherein: The predetermined set of one or more mutual arrangement relationships includes an adjacency relationship, a proximity relationship, a distance relationship and / or a vertical stacking relationship.

9. The method according to any one of claims 1 to 8, wherein: The second neural network also takes as input the camera position output by the first neural network.

10. The method according to any one of claims 1 to 9, wherein: The first neural network has an architecture including a graphical neural network (GNN) model and a multi-layer perceptron (MLP) model, the MLP model taking as input a concatenation of the output of the GNN model and the coordinates of the starting position, and / or The second neural network has an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model, and the MLP model takes as input a concatenation of the output of the GNN model and the coordinates of the camera position output by the first neural network.

11. The method according to any one of claims 1 to 10, wherein: The machine learning function includes a post-processing block configured to: The output one or more camera viewpoints are selected from a set of camera viewpoints determined by applying the first neural network and the second neural network to each starting position in a set of starting positions regularly distributed on a plan view of the input 3D scene.

12. The method according to claim 11, wherein: Select one or more camera viewpoints to be exported including: For each camera viewpoint, determining 3D objects of the input 3D scene that are visible in a 2D rendering generated from the camera viewpoint; and The one or more determined camera viewpoints are selected such that the 3D objects visible in the one or more 2D renderings generated from the one or more selected camera viewpoints represent at least a predetermined percentage of all 3D objects present in the 3D scene.

13. A computer program comprising instructions for executing the method of any one of claims 1 to 12.

14. A computer-readable storage medium having recorded thereon a first neural network machine learned according to any one of claims 1 to 12, a second neural network machine learned according to any one of claims 1 to 12, and / or a computer program according to claim 13.

15. A system comprising a processor coupled to a memory having recorded thereon the computer program of claim 13.

Images