Machine-learning function to output camera viewpoint

Abstract

To provide a computer implementation method to decide a machine-learning function configured to take in an input 3D scene and to output each of one or a plurality of camera viewpoints for generating each 2D rendering for each 3D scene.SOLUTION: This method includes the steps for: forming a first data set for training a first neural network configured to output a camera position based on a library including 3D scenes; forming a second data set for training a second neural network configured to output a camera direction; and training the first neural network based on the first data set, and training the second neural network based on the second data set. Each camera viewpoint output by the machine-learning function includes a camera position and a camera direction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for determining machine learning functions configured to take an input 3D scene and output one or more respective camera viewpoints for generating respective 2D renderings of the 3D scene. [Background technology]

[0002] Several systems and programs for designing, engineering and manufacturing objects are offered on the market. CAD is an acronym that stands for Computer-Aided Design, e.g., it relates to software solutions for designing objects. CAE is an acronym that stands for Computer-Aided Engineering, e.g., it relates to software solutions for simulating the physical behavior of a future product. CAM is an acronym that stands for Computer-Aided Manufacturing, e.g., it relates to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays a key role for the efficiency of the technology. These technologies can be incorporated within a product lifecycle management (PLM) system. PLM refers to a business strategy that helps companies share product data, apply common processes and leverage the company's knowledge for the evolution of products from conception to the end of their life, throughout the concept of an extended enterprise. The PLM solutions offered by Dassault Systèmes (under the brands CATIA, ENOVIA, 3DVIA, and DELMIA) provide an Engineering Hub that codifies product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to both the Engineering and Manufacturing Hubs. Overall, the system delivers an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] Interior design increasingly involves the creation of virtual indoor 3D scenes that represent real rooms. To show these indoor 3D scenes, images may be captured from specific viewpoints of the 3D scene. Automatic methods for determining a viewpoint for a virtual visit of an indoor scene are generally not known in the literature. Rather, the literature studies viewpoint estimation for a single object.

[0004] With reference to the list of prior art references provided below, the method of [1] aims to speed up viewpoint estimation for single objects by learning to estimate a viewpoint that maximizes a certain metric, such as the number of faces of the object's mesh that are visible from the estimated viewpoint. The proposed method strongly relies on the assumption that there is only a single object being observed. For example, the viewpoint to be found is determined by finding coordinates on a sphere centered on the object. Although this certainly simplifies the problem (looking for coordinates on a surface instead of a volume), for obvious reasons (restrictions on the size of the search space, and on the topology of the room), it is quite impossible to apply to viewpoint estimation for 3D scenes. Moreover, even if some metrics are computed and it is possible to determine a good viewpoint for a single object, they may be more difficult to define (even more implicit) for 3D scenes.

[0005] Also, certain methods (e.g., the method in [2]) try to estimate the evaluation of aesthetic features of images based on learning on a dataset created by a user. However, it is clear that such approaches are not yet mature enough to automatically determine viewpoints for virtual visits of indoor scenes. In detail, one of the problems is that the constructed datasets are not operational. Secondly, again unlike the field of indoor scenes, it is not clear how such methods could construct relevant datasets even if they were applied in this field (e.g., for the problem of random cameras vs. cameras placed by a user, or for the problem of step-by-step user rendering). Thirdly, such approaches only give a score for a given image, and do not at all address the problem of finding the correct camera parameters (e.g., position and orientation) to obtain this image.

[0006] A few rule-based methods for defining reasonable camera positions can be envisioned (e.g., a camera is placed in front of each of the centers of each of the walls with a view to the center of the corresponding wall), however, they can easily fail in complex room structures.

[0007] Without using neural networks, the literature includes the method of

[10] , which relies on heuristic-based methods such as computing triangle density for several views and retaining viewpoints where the density is higher (making the assumption that the higher the density, the more information there is in the image). The method of

[11] explores a similar approach; however, those methods are more suited to the purpose of dataset construction (mainly because they can be very slow).

[0008] Non-Patent Document 9 proposes a data-driven approach to selecting viewpoints for 3D scenes. In this document, the distribution of an example set of good renderings is modeled using n probability density functions that represent the x, y, and depth positions of pixels for each of the n semantic classes in the example set. Candidate positions are then generated by sampling viewpoints according to the probability density functions estimated above. After that, a set of candidate views that approximately match the entire probability density field as a set is selected using an algorithm based on submodular maximization. However, this method has several failure cases and requires a lot of computation (distribution estimation, candidate property estimation and selection). It is not deep learning based and is more suitable for dataset construction, such as Non-Patent Documents 10 and 11.

[0009] The literature also includes [8], which is also published for a single object (not applicable to 3D scenes) but attempts to learn the best viewpoint from human preferences from viewpoints labeled by humans. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Schelling, M., Hermosilla, P., Vazquez (a has an accent mark. The same applies to the reference section below), PP, & Ropinski, T. (2021, May). Enabling Viewpoint Learning through Dynamic Label Generation. In Computer Graphics Forum (Vol. 40, No. 2, pp. 413-423). [Non-Patent Document 2] Talebi, H., & Milanfar, P. (2018). NIMA: Neural image assessment. IEEE transactions on image processing, 27(8), 3998-4011. [Non-Patent Document 3] Li, Y., Bu, R., Sun, M., Wu, W., Di, X., &Chen, B. (2018). Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems, 31. [Non-Patent Document 4] Wang, K., Lin, YA, Weissmann, B., Savva, M., Chang, AX, & Ritchie, D. (2019). Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transactions on Graphics(TOG), 38(4), 1-15. [Non-Patent Document 5] Corso, G., Cavalleri, L., Beaini, D., Lio (with grave accent on o), P., & Velickovic (with caron on first c, accent on second c), P. (2020). Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33, 13260-13271. [Non-Patent Document 6] Paschalidou,D.,Kar,A.,Shugrina,M.,Kreis,K.,Geiger,A.,&Fidler,S.(2021).Atiss:Autoregressive transformers for indoor scene synthesis.Advances in Neural Information Processing Systems,34,12013-12026.

Direct Environment 7

Outdoor Tools 8

Outdoor Tools9

Outdoor Tools 10

[0011] Therefore, a computer-implemented method is provided for determining a machine learning function configured to capture an input 3D scene and output one or more respective camera viewpoints for generating a respective 2D rendering of the 3D scene. The method includes obtaining a library including 3D scenes and, for each of the 3D scenes, one or more respective camera viewpoints for generating a respective 2D rendering of the 3D scene. The method includes forming, based on the library, a first data set for training a first neural network configured to capture a first input representation of the 3D scene and output information corresponding to a camera position in the 3D scene. The method includes forming, based on the library, a second data set for training a second neural network configured to capture a second input representation of the 3D scene and output information corresponding to a camera orientation in the 3D scene. The method includes training the first neural network based on the first data set and training the second neural network based on the second data set. Each of the camera viewpoints output by the machine learning function includes a camera position corresponding to the information output by the first neural network and a camera orientation corresponding to the information output by the second neural network.

[0012] The method may include one or more of the following.

[0013] - forming a first data set for each of the 3D scenes, determining a set of starting positions regularly distributed on the floor plan of the 3D scene; determining, for each camera viewpoint of the 3D scene, an offset vector between the camera position of the camera viewpoint and the closest starting position, the first neural network is configured to further take as input a starting position in the 3D scene, and the information output by the first neural network includes an offset vector between the camera position and the starting position; - the start positions of each of the sets are substantially equidistant from each other. The distance between the start positions is substantially the same for each of the 3D scenes. The starting positions of the set are distributed to form a regular 2D grid on the floor plan of the 3D scene, with each starting position corresponding to a vertex of the regular 2D grid. - the first input representation and / or the second input representation comprises a set of geometric properties of 3D objects of the 3D scene; - the first input representation and / or the second input representation comprises a graph including respective nodes representing respective objects of the 3D scene and respective arcs linking respective pairs of nodes, the 3D objects represented by the pairs of nodes being arranged according to a mutual relationship from a predetermined set of one or more mutual relationships; the predetermined set of one or more mutual placement relationships includes an adjacent relationship, a proximate relationship, a distance relationship, and / or a vertical overlap relationship; The second neural network further takes as input the camera position output by the first neural network; The first neural network has an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model. The MLP model takes as input a concatenation of the output of the GNN model and the coordinates of a starting position; the second neural network has an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model, the MLP model taking as input a concatenation of the output of the GNN model and the coordinates of the camera position output by the first neural network; the machine learning functionality comprises a post-processing block, the post-processing block being configured to select the output camera viewpoint or viewpoints from a set of camera viewpoints determined by applying the first neural network and the second neural network to each of a set of start positions regularly distributed on the floor plan of the input 3D scene; and / or - Select one or more camera viewpoints to be output. determining, for each camera viewpoint, the 3D objects of the input 3D scene that are visible in the 2D rendering generated from that camera viewpoint; ○ Selecting one or more of the determined camera viewpoints, whereby the 3D objects visible in one or more 2D renderings generated from the one or more selected camera viewpoints represent at least a predetermined percentage of all 3D objects present in the 3D scene.

[0014] There is further provided a computer program comprising instructions for carrying out this method.

[0015] There is further provided a computer readable storage medium having the computer program recorded thereon.

[0016] There is further provided a system including a processor coupled to a memory, the memory having a computer program recorded thereon. The system may include a graphical user interface.

[0017] Further provided is a device including a data storage medium having a computer program recorded thereon. The device may form or act as a non-transitory computer readable medium, such as in a software as a service (SaaS) or other server or cloud-based platform. The device may alternatively include a processor coupled to the data storage medium. Thus, the device may form a computer system in whole or in part (e.g., the device is a subsystem of an overall system). The system may further include a graphical user interface coupled to the processor.

[0018] There is further provided a method of use of the machine learning function determined according to this method, the method of use comprising the steps of: obtaining an input 3D scene; and applying the machine learning function to the obtained input 3D scene, thereby outputting each of one or more camera viewpoints for generating a respective 2D rendering of the obtained input 3D scene.

[0019] There is further provided a machine learning function determined according to the method, the machine learning function configured to take an input 3D scene and to output each of one or more camera viewpoints for generating respective 2D renderings of the 3D scene. [Brief description of the drawings]

[0020] Non-limiting examples will now be described with reference to the accompanying drawings. [Figure 1] 1 is a flow chart of an example of this method. [Diagram 2] 1 is a flowchart of an example of an offline phase and an online phase. [Diagram 3] FIG. 2 is a diagram showing an example of a GNN model of a first neural network. [Figure 4] FIG. 13 illustrates an example of the results of a machine learning function. [Diagram 5] FIG. 13 illustrates an example of the results of a machine learning function. [Figure 6] FIG. 13 illustrates an example of the results of a machine learning function. [Figure 7] FIG. 13 illustrates an example of the results of a machine learning function. [Figure 8] FIG. 13 illustrates an example of the results of a machine learning function. [Figure 9] FIG. 1 illustrates an example of the architecture of a machine learning function. [Figure 10] FIG. 1 illustrates an example of the architecture of a machine learning function. [Figure 11] FIG. 13 is a diagram showing an example of a set of starting positions. [Figure 12] FIG. 1 illustrates an example of a system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] With reference to the flow chart of FIG. 1, a computer-implemented method is proposed for determining a machine learning function configured to capture an input 3D scene and to output one or more camera viewpoints for generating a respective 2D rendering of the 3D scene. The method includes a step S10 of obtaining a library including 3D scenes and, for each of the 3D scenes, one or more camera viewpoints for generating a respective 2D rendering of the 3D scene. The method includes a step S20 of forming, based on the library, a first data set for training a first neural network configured to capture a first input representation of the 3D scene and to output information corresponding to a camera position in the 3D scene. The method includes a step S30 of forming, based on the library, a second data set for training a second neural network configured to capture a second input representation of the 3D scene and to output information corresponding to a camera orientation in the 3D scene. The method includes a step S40 of training the first neural network based on the first data set and training the second neural network based on the second data set. Each of the camera viewpoints output by the machine learning function includes a camera position that corresponds to the information output by the first neural network and a camera orientation that corresponds to the information output by the second neural network.

[0022] Such a method forms an improved solution for outputting one or more camera viewpoints of a 3D scene.

[0023] In particular, the method allows training a function for outputting one or more camera viewpoints. In particular, the method allows training a function on the basis of a library of 3D scenes. In fact, the method includes the formation of two data sets for training the function, one for each neural network. Thus, starting directly from a library of 3D scenes, the method allows training of the function for outputting camera viewpoints to be performed automatically.

[0024] Moreover, using a first neural network for camera position and a second neural network for camera orientation improves the optimality of the output camera viewpoints. Indeed, forming the first and second data sets based on a library (containing optimal camera viewpoints for each of the 3D scenes) allows training both neural networks to estimate the optimal camera viewpoints. Optimality means that there is a large number of objects at each of the output camera viewpoints. This method improves the training of each of the neural networks and therefore the optimality of the camera viewpoints output by the neural networks after training.

[0025] Moreover, it is particularly important to consider a first neural network for the camera position and a second neural network for the camera direction. In fact, these two parameters (camera position and orientation) allow to define a unique viewpoint, and the training for predicting these two parameters can be separated. That makes the use of a respective neural network for each of these parameters particularly important. In addition, considering a specific neural network for each of the parameters is a good compromise between the respective specialization of the neural network and the difficulty of training. In fact, it allows the neural networks to be trained respectively to capture the specificity of these parameters in the library of 3D scenes, while keeping a reasonable number of neural networks to be trained (i.e. two networks, one for the camera position and another for the camera direction).

[0026] The method is computer-implemented. This means that the steps of the method (or substantially all steps) are performed by at least one computer, or any similar system. The steps of the method are therefore performed by a computer, possibly fully automatically or semi-automatically. In an example, the triggering of at least some of the steps of the method may be performed through user-computer interaction. The level of user-computer interaction required may depend on the level of automation anticipated and may be balanced against the need to implement the user's wishes. In an example, this level may be user-defined and / or predefined.

[0027] A typical example of a computer implementation of the method is to execute the method using a system adapted for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program including instructions for carrying out the method. The memory may also store a database. The memory is any hardware adapted for such storage, possibly including several physically separate parts (e.g. a part for the program and possibly a part for the database).

[0028] The library of 3D scenes, the first data set, and / or the second data set may be stored in a database. By "database" is meant any collection of data (i.e., information) that is organized for search and retrieval (e.g., a relational database based on a predefined structured language, e.g., SQL). When stored in memory, a database allows for fast search and retrieval by a computer. A database is actually structured to facilitate the storage, retrieval, modification, and deletion of data in conjunction with various data processing operations. A database may consist of a file or set of files that may be divided into records, each of which is composed of one or more fields. A field is the basic unit of data storage. A user may retrieve data primarily through queries. Using keywords and sorting commands, a user may quickly search, sort, group, and select fields in many records to retrieve or create reports on specific aggregations of data according to the rules of the database management system being used.

[0029] The methods generally manipulate modeled (3D) objects. A modeled object is any object that is defined by data stored, for example in a database. By extension, the expression "modeled object" refers to the data itself. Depending on the type of system, the modeled object may be defined by different kinds of data. For example, the system may be a CAD system and the modeled object may be defined by the corresponding data (which may be called CAD object).

[0030] By CAD system is further meant at least any system adapted to design a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, the data defining the modeled object includes data enabling the representation of the modeled object. The CAD system may provide a representation of the CAD modeled object, for example, using edges or lines, and in certain cases, faces or surfaces. The lines, edges or surfaces may be represented in various manners, for example, non-uniform rational B-splines (NURBS). In particular, a CAD file contains specifications from which a geometry may be generated, which geometry then enables a representation to be generated. The specifications of the modeled object may be stored in a single CAD file or in multiple CAD files. A typical size of a file representing a modeled object in a CAD system is in the range of one megabyte per part. And the modeled object may typically be an assembly of several thousand parts.

[0031] In the context of CAD, a modeled object may typically be a 3D modeled object, for example representing a product, such as a part or an assembly of parts, or possibly an assembly of a product. By "3D modeled object" is meant any object modeled by data that allows a 3D representation of the object. The 3D representation allows the part to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, can be manipulated and rotated around any of its axes or around any axis in the screen on which the representation is displayed. This notably excludes 2D icons that are not 3D modeled. The display of 3D representations facilitates design (i.e. increases the speed at which designers statistically accomplish their tasks). It speeds up the manufacturing process in industry, since the design of a product is part of the manufacturing process.

[0032] A CAD system may be history-based. In this case, the modeled object is further defined by data including the history of its geometric features. The modeled object may actually be designed by a physical person (i.e., a designer / user) using standard modeling features (e.g., extrude, revolve, cut, and / or round) and / or standard surfacing features (e.g., sweep, blend, loft, fill, deform, and / or smooth). Many CAD systems that support such modeling capabilities are history-based systems. This means that the creation history of the design features is typically preserved through a non-cyclic data flow that links the above geometric features together through input and output links. The history-based modeling paradigm has been well known since the early 80s. A modeled object is described by two persistent data representations: the history and the B-rep (i.e., boundary representation). The B-rep is the result of the calculations defined in the history. The shape of the part displayed on the computer screen when the modeled object is represented is the B-rep (e.g., its tessellation). The history of a part is the design intent. Essentially, the history collects information about the operations that the modeled object has undergone. To make it easier to display complex parts, the B-rep can be saved with the history. To enable design changes of the part according to the design intent, the history can be saved with the B-rep.

[0033] Each of the 3D scenes may represent a real room. Each of the 3D scenes of the library may include one or more 3D objects. Some (e.g., all) of the 3D scenes of the library and / or the input 3D scenes may represent complex rooms, i.e., each may include several objects (e.g., more than 10 or 100), and / or each may be of a shape that is not rectangular (e.g., with more than four walls and / or including one or more circular walls). Each of the 3D objects of the 3D scenes may represent the geometry of a real object that is placed (or will be placed) in the real room represented by the 3D scene. The real object may be manufactured in the real world after completion of a virtual design of the real object (e.g., using a CAD software solution or CAD system). The 3D scenes of the library may each include one or more furniture objects and / or one or more decorative objects. Any furniture objects here may have a furniture function in the room in which they are placed. For example, a 3D scene of a library may include one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more sideboards, one or more nightstands, one or more desks, and / or one or more dressers. Any decorative objects herein may have a decorative function in the room in which they are placed. For example, a 3D scene of a library may include one or more accessories, one or more plants, one or more books, one or more picture frames, one or more kitchen utensils, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors, and / or one or more electronic items (e.g., a refrigerator, a freezer, and / or a washing machine).

[0034] This method may be used in a generation process, which may include generating one or more camera views of a room using the machine learning function (estimation phase) determined by the method after execution of the method. In that case, the method may be included in the generation process (i.e., the formation of the data set and the training of the neural network may be performed during the generation process). Alternatively, the method may be performed prior to execution of the generation process (i.e., the machine learning function may already be trained at the time the generation process is executed).

[0035] The one or more camera viewpoints may be generated for a user, such as a home owner in which the room is located. The generation process may include obtaining an input 3D scene (e.g., an input 3D scene provided by a user) representing the room to be located. The generation process may then include applying the determined machine learning function to the obtained input 3D scene, thereby automatically generating one or more camera viewpoints for the input 3D scene (i.e., one or more viewpoints output by the function). In an example, the generation process may then include recording the generated one or more camera viewpoints, such as in a database accessible online by the user.

[0036] In an example, the generation process may be used in a virtual visit process. For example, the generation process may be included in the virtual visit process or alternatively may be executed prior to the virtual visit process (so that the virtual visit process is executed based on one or more camera viewpoints generated by execution of the generation process). The virtual visit process may include determining images and / or videos of a room based on the generated one or more camera viewpoints. For example, the virtual visit process may include generating a respective 2D rendering (i.e., images and / or videos) of each of the determined camera viewpoints. Each 2D rendering may include renderings of objects, walls, and / or windows / doors visible from the viewpoint. The virtual visit process may then include assembling the generated 2D renderings, for example, in the form of a video showing the various generated 2D renderings in succession. The virtual visit process may include or be based on a repetition of the generation process. For example, the generation process may be repeated for each of the rooms in a house or apartment. In this case, the virtual visit process may generate a virtual visit of all rooms in the house or apartment (i.e., including 2D renderings of all rooms).

[0037] The virtual visit process may be included in a design process of a real room, which may include using the generated 2D rendering to determine whether to obtain one or more new real objects for the room after performing the virtual visit process. Each of the new real objects may have a respective object (representing the new real object, i.e., a virtual model of the real object) placed inside the input 3D scene. For example, the design process of the real room may include a step of placing each of the objects representing each of the new real objects inside the input 3D scene (e.g., using a virtual catalog containing a virtual representation for the new object, or by creating each of the objects representing each of the new objects from scratch) before the virtual visit process. Each of the objects representing the new real objects may be visible in the generated 2D rendering. The generated 2D rendering may help the user make a selection by showing one or more new objects placed inside the room (i.e., placed relative to other objects in the room).

[0038] Alternatively or additionally, the design process of the real room may include physically arranging the room (i.e., real) after the virtual visit process is performed so that its design matches the input 3D scene. For example, the room (without the one or more new objects) may already exist in the real world, and the design process of the real room may include arranging one or more new real objects inside the existing room (i.e., in the real world). The new real objects may be placed in the same position as the respective objects representing the new real objects in the input 3D scene. Alternatively, the room may not yet exist at the time the method is performed. In that case, the design process of the real room may include building the room according to the provided 3D scene (i.e., by starting from scratch and placing the objects as in the provided 3D scene). Since this method improves the acquisition of new real objects for the room, it also improves the placement of the new real objects and / or the building of the room, thus increasing the productivity of the design process of the real room.

[0039] The method may include a step of forming a library (library forming step). In that case, obtaining S10 may include the library forming step. The library forming step may include providing a 3D scene and / or a camera viewpoint of the 3D scene. The providing of the 3D scene and / or the camera viewpoint may be performed based on a user interaction. For example, each of the 3D scenes may be created by a user (for example, using a software solution for creating a 3D scene of a room). The creation of the 3D scene may include defining the walls / doors of the room and placing a 3D object inside the defined walls. The providing of the camera viewpoint may also be performed by a user. For example, the providing of each of the camera viewpoints may include determining a camera position and a camera direction of the camera viewpoint. After the library forming step, obtaining S10 includes recording the calculated library in a database, and steps S20 and S30 may be performed by accessing the recorded library.

[0040] Alternatively, the method may include retrieving a library of 3D scenes in a database. The library of 3D scenes may, in such a case, optionally have been obtained (prior to carrying out the method) in accordance with the library formation step above. In that case, obtaining S10 may include providing access to the library of 3D scenes in the database. The method may use this access to form (during steps S20 and S30) the first and second data sets.

[0041] The formation S20 of the first data set may be performed in any manner. The formation S20 may include determining training samples of the first data set. For example, the formation S20 may include, for each of the 3D scenes of the library, associating each input representation of the 3D scene with information corresponding to a respective camera position of a camera viewpoint of the 3D scene (each of the associations corresponds to a respective training sample of the first data set). Each of the training samples of the first data set may include an input representation associated with information corresponding to a camera position.

[0042] The information corresponding to the camera position may include a vector having values ​​representing the coordinates of a point in a drawing representing the camera in a plan view of the 3D scene. For example, the coordinates may be defined in a (for example, arbitrarily selected) reference coordinate system. Alternatively, the coordinates may be defined from a starting position in the 3D scene. In that case, the vector may be an offset vector defined between the camera position and the starting position. The 3D scene may include a plan view and a set of starting positions regularly distributed on the plan view. The offset vector may be defined between the camera position and the closest starting position (i.e., among all starting positions in the set).

[0043] Similarly, the formation S30 of the second data set may be performed in any manner and may include determining training samples of the second data set. For example, the formation S30 may include, for each of the 3D scenes of the library, associating the respective input representation of the 3D scene with information corresponding to a respective camera direction of a camera viewpoint of the 3D scene (each of the associations corresponds to a respective training sample of the second data set). The formation S30 may further include, for each of the 3D scenes of the library, associating the respective input representation of the 3D scene with information corresponding to a respective camera direction of a camera viewpoint of the 3D scene. In that case, each of the training samples of the second data set may include an input representation associated with information corresponding to a camera direction and information corresponding to a camera position. If the first data set is formed first, the formation S30 of the second data set may be performed based on the already formed first data set. For example, forming the second data set S30 may include adding to each training sample of the first data set an association to information corresponding to the camera direction of the camera viewpoint represented by that training sample (i.e., in addition to the association to information corresponding to the camera position already included in the first data set).

[0044] In an example, the method may form the first and second data sets using the same 3D scenes of the library, such as using all 3D scenes of the library (i.e., each of the data sets may include training samples determined for the same 3D scene). In that case, the first and second data sets may take the form of a single table. For example, the single table may include rows representing each of the training samples and columns representing each value of the training samples. For example, the single table may include a first column for the input representation, a second column for the camera position, and a third column for the camera orientation. In that case, the first data set may be formed by the first and second columns, and the second data set may be formed by the first, second, and third columns. Alternatively, the method may form the first data set based on a first subset of the 3D scenes of the library, and the second based on a second subset of the 3D scenes of the library. In that case, each of the subsets may include 3D scenes from the library that are in common with the other subset, and other 3D scenes that are not in common, in which case the two datasets may each take the form of a respective table, or alternatively may be combined in a single table, e.g., with an empty value if no 3D scenes are present in both datasets.

[0045] A camera orientation may be a description of the direction in which a camera is pointing in a 3D scene space. The camera orientation may be defined relative to a reference coordinate system (e.g., including three vectors). Information corresponding to the camera's position may include respective angle values ​​that represent the rotation between the direction being pointed by the camera and the respective vectors of the reference coordinate system.

[0046] In an example, the second neural network further takes in as input the camera position output by the first neural network. In that case, each of the training samples of the second data set may further include information corresponding to a camera position of a camera viewpoint (in addition to information corresponding to a camera direction). The forming S30 may include, for each of the 3D scenes of the library, associating each of the input representations of the 3D scene with information corresponding to a camera position of each of the camera viewpoints of the 3D scene as well. Each of the training samples of the second data set may include an input representation associated with information corresponding to a camera direction and a camera position.

[0047] In an example, the formation of the first data set S20 and the formation of the second data set S30 may be performed simultaneously. For example, the first data set may correspond to a portion of the second data set (the first data set includes each of the training samples of the second data set, but does not include information about the camera orientation for each of the training samples). In that case, the formation of the second data set S30 may be performed as described above, and the formation of the first data set S20 may consist in extracting the portion of the second data set that corresponds to the first data set (i.e., the association of the input representation to only information about the camera position for each of the camera viewpoints).

[0048] In an example, during the estimation phase, the first neural network may be configured to further take in as input a starting position. The starting position may be among a set of starting positions regularly distributed on the floor plan of the 3D scene taken in as input by the first neural network. In that case, the forming S20 may include a step S21 of determining, for each 3D scene, a set of starting positions regularly distributed on the floor plan of the 3D scene. The starting positions of each of the set may be substantially equidistant from each other. The distance between the starting positions may be substantially the same for each of the 3D scenes. For example, the distance between adjacent starting positions may be less than 3 meters and / or more than 50 centimeters, for example between 75 centimeters and 1.50 meters (for example, approximately 1 meter). The starting positions of each of the set may be randomly distributed (for example, in an accessible space of a room, i.e., a space without objects). Alternatively, the starting positions of each of the set may be regularly distributed to form a regular pattern on the floor plan of the 3D scene (for example, a pattern including concentric circles or a grid pattern). For example, the respective starting positions of the set may be distributed to form a regular 2D grid. Each of the starting positions may correspond to a respective vertex of the regular 2D grid. During the estimation phase, the information output by the first neural network may include an offset vector between the camera position and the starting positions taken as input.

[0049] After the determination S21, the formation S20 may include a step S22 of determining, for each camera viewpoint of the 3D scene, an offset vector between the camera position of the camera viewpoint and a nearest start position. The determination S22 of the offset vector may include calculating the distance between the camera position and each of the set of start positions, the nearest start position being the start position having the smallest distance. The determination S22 of the offset vector may then include determining the coordinates of the offset vector in a reference coordinate system (each of the coordinates represents the distance between the camera viewpoint and the nearest start position along a respective direction of the reference coordinate system).

[0050] The set of starting positions and the offset vectors improve the training of the neural networks and therefore the estimation of the camera viewpoint by these neural networks. In fact, the set of starting positions allows the abstraction of local patterns and learning decoupled from the context. The set of starting positions allows the consideration of a portion of the 3D scene, and the offset vectors allow the consideration of the fact that each of the ground truths fits to each portion of the 3D scene. Each of the starting positions represents a boundary within which each of the neural networks is trained to predict values ​​present in the training set. Thus, they allow the consideration of the fact that the ground truth fits to each portion of the 3D scene and other ground truths may be present at other points in the 3D scene.

[0051] Any input representation of a 3D scene may include a set of geometrical attributes of (e.g., all) 3D objects of the 3D scene. The geometrical attributes may include, for each of the 3D objects, the position, orientation, and / or dimensions of the 3D object. The position may include the coordinates of the center of the 3D object in a reference coordinate system. The orientation of the 3D object may be defined relative to a reference position. For example, the orientation may include each of angle values ​​that represent the rotation of the 3D object when placed in the 3D scene along a respective vector of the reference coordinate system relative to the reference position of the 3D object.

[0052] In an example, any input representation of a 3D scene may include a voxelization of the 3D scene. The voxelization may include a regular mesh of voxels on the 3D scene space. Each voxel may include information about the presence of an object in the region bounded by that voxel (e.g., 0 if the object is not present, or 1 otherwise). The voxelization may provide a set of geometrical attributes (i.e., position, orientation, and / or dimensions) of each of the 3D objects. Alternatively, any input representation of a 3D scene may include a graph including nodes each representing a respective object of the 3D scene and arcs each linking each pair of nodes. The 3D objects represented by the nodes of a pair may be positioned according to a mutual placement relationship from one or more predetermined sets of mutual placement relationships. Each edge may include a label indicating a mutual placement relationship between the two objects represented by the nodes linked by that edge. Each node of the graph may include a set of geometrical attributes (i.e., position, orientation, and / or dimensions) of the 3D object it represents.

[0053] In an example, the predefined set may include an adjacency relationship. If two objects are spaced less than a first predefined distance from each other (e.g., 5 centimeters), the objects may be positioned according to the adjacency relationship. Alternatively or additionally, the predefined set may include a proximity relationship. If two objects are spaced more than a first predefined distance from each other and less than a second predefined distance (e.g., 50 centimeters), the objects may be positioned according to the adjacency relationship. Alternatively or additionally, the predefined set may include a distance relationship. If two objects are spaced more than a second predefined distance from each other, the objects may be positioned according to the distance relationship. For each of the relationships, the distance may be a distance between bounding boxes representing the two objects in the 3D scene (e.g., a distance defined along one of the axes of one of the two objects). Alternatively or additionally, the predefined set may include a vertical overlap relationship. When two objects are placed one above the other (eg, a ceiling lamp placed above a table), the objects may be positioned according to an adjacency relationship.

[0054] In an example, the method may include determining a respective input representation for one or more (e.g., all) of the 3D scenes of the library. The determination of the input representation may be performed during the formation of the dataset. The determination of the respective input representation for the 3D scene (representation formation step) may include determining geometrical characteristics of the 3D objects of the 3D scene. For example, the representation formation step may include determining the placement, orientation, and / or dimensional information (optionally bounding boxes) of each of the 3D objects of the 3D scene. In an example, the representation formation step may also include determining a graph of the 3D scene. The determination of the graph may include determining each of the nodes and each of the arcs of the graph according to the objects of the 3D scene. Each of the nodes of the graph may include the placement, orientation, and / or dimensional information (optionally bounding boxes) determined for the objects of the 3D scene that it represents in the graph.

[0055] Alternatively, the library may include a respective input representation of each of the 3D scenes, in which case the method may include retrieving the respective input representation of the 3D scene during formation of each of the data sets (steps S20 and S30). The input representations included in the library of 3D scenes may optionally have been obtained in such a case in accordance with the representation formation step described above (prior to execution of the method).

[0056] The acquisition of the library (step S10), the formation of the first and second data sets (steps S20 and S30), and / or the training of the first and second neural networks (step S40) may be performed during an offline phase. In that case, the acquired library, the first and second data sets, and / or the determined machine learning function (including the trained first and second neural networks) may be recorded, for example, in a database during this offline phase. After this offline phase, the determined machine learning function may be used during the online phase (estimation phase). For example, the determined machine learning function may be used by each of the client computers configured to provide input 3D scenes to a server, which is configured to access the database and apply the machine learning function to the provided input 3D scenes, and to provide the camera viewpoints output by the function for each of the input 3D scenes to the client computers providing the input 3D scenes (or 2D renderings generated, for example, by the server for each camera viewpoint).

[0057] The method may include updating the first and second datasets (e.g., periodically). For example, the library may be expanded by a user providing camera viewpoints for newly created 3D scenes, and the method may include updating the first and second datasets to include these newly created 3D scenes (e.g., by adding the newly created 3D scenes and their respective camera viewpoints to the first and second datasets). In that case, the method may include, after updating, retraining the first neural network based on the updated first dataset and retraining the second neural network based on the updated second dataset.

[0058] The method will now be discussed further with reference to Figures 2 to 12.

[0059] This method may consist in a fully automatic way to estimate viewpoints of a 3D room that can be used to define a virtual visit of the 3D room. In the case of estimating viewpoints for a virtual visit of a 3D scene, this method may make it possible to expose the complexity of the 3D scene, i.e. objects in the scene may give information about their relative position from the user's viewpoint, and may give all this information with a minimal set of estimated viewpoints.

[0060] The method may consist in a method for training a neural network (training phase) to estimate the 3D viewpoints of a 3D scene (or a 3D room) from a library (dataset) of 3D rooms with several (small number) 3D viewpoint annotations (respective camera viewpoints of the 3D rooms). Optionally, the method may also include a selection step for filtering those viewpoints to select a minimal set of viewpoints that can be used to build a virtual visit of the 3D room (estimation phase).

[0061] The method includes a training phase. The training phase includes forming S20 a first dataset. The forming S20 includes optionally determining S21 a set of plausible camera positions (a set of starting positions) for each 3D room in the dataset, and optionally finding S22 a closest plausible camera position from the ground truth viewpoint position for each ground truth viewpoint position (camera position) present in the dataset for the 3D room, and storing an offset vector from the plausible camera position to the ground truth viewpoint position. The training phase then includes training S40. The training phase S40 optionally includes providing a first neural network that takes as input a 3D room (e.g., a 3D graph representation of the 3D room) and an initial camera position, and outputs a three-dimensional vector (an offset vector from the initial camera position). The training S40 then includes applying a learning algorithm to the first neural network to train the first neural network to output a correct offset vector given every 3D room in the dataset and the corresponding initial camera position. Training S40 also optionally includes providing a second neural network that takes as input a 3D room (e.g., a 3D graph representation of the 3D room) and a camera position and outputs a three-dimensional vector (e.g., a three-dimensional vector for each angle of orientation). Training S40 then includes applying a learning algorithm to the second neural network to train the second neural network to output the correct ground truth direction vector given any 3D room and corresponding ground truth viewpoint position.

[0062] The estimation phase (method of use) includes defining a set of plausible camera positions given a 3D room. The estimation phase includes, for each camera position in the set of plausible camera positions, applying an offset vector estimated by the first trained neural network to each corresponding camera position in the set of plausible camera positions. The estimation phase includes, for each new camera position, estimating a direction vector by the second trained neural network. The estimation phase includes applying a post-processing step to retain a minimal set of estimated viewpoints from which a certain number of objects are visible relative to the total number of objects that can be seen from any estimated viewpoint.

[0063] The method makes it possible to train a function to automatically estimate a set of viewpoints that can be linked together to build a virtual visit of a given 3D room, maximizing the visibility of the main elements in the room (mainly the objects in the room). The method is fully automatic, suitable for any room type, and uses a very light representation of the room.

[0064] The method belongs to the field of 3D viewpoint estimation for 3D rooms. The objective of automatic viewpoint estimation is to compute for a given room a set of camera viewpoints giving camera positions and orientations that can be used to build a virtual visit of the given room. The camera viewpoints may include a camera position and a camera orientation. The camera position and the camera orientation may be extrinsic parameters of the camera viewpoints. The camera viewpoints may also include intrinsic parameters (e.g. magnification, rendering window width, rendering window length, rendering color properties). The generation of a 2D rendering for a given camera viewpoint may take into account the extrinsic and intrinsic parameters of the given camera viewpoint. The virtual visit is a navigation between the viewpoints estimated for a given room. The goal of the virtual visit is to observe the maximum number of objects in the scene.

[0065] This method may be included in a general process that may include two stages. In the first stage, the general process may include building and training two models, one model for estimating the viewpoint position and the other model for estimating the viewpoint direction (i.e., executing the method, also called the offline stage). In the second stage, the general process may include viewpoint estimation for any room based on the two trained models (also called the online stage).

[0066] An example flow chart of the offline and online phases will now be discussed with reference to Figure 2. This figure illustrates the differences between the online and offline phases of the general process.

[0067] The offline stage S110 aims to generate a model (neural network) that accomplishes the tasks of (i) estimating the viewpoint position, (ii) estimating the viewpoint direction for a given room based on the proposed learning scheme given a training dataset. The training dataset is a dataset composed of 3D rooms with one or several viewpoints per room. The rooms are considered to be well-furnished and with good viewpoints (renderings have been purchased by users). The first neural network (also called viewpoint position estimation model) takes as input the 3D rooms of the dataset together with a starting position and learns to estimate a 3D vector (named offset) such that applying a 3D transformation corresponding to the 3D predicted vector to the starting position matches the viewpoint position of the dataset for this 3D room. The second neural network (also called viewpoint direction estimation model) takes as input the 3D rooms of the dataset together with a camera position from the viewpoint position of the dataset for this 3D room and learns to estimate the viewpoint direction of the dataset for this 3D room.

[0068] Given a 3D room, the online phase S120 includes determining a set of starting positions. For each position, the online phase S120 includes applying the trained viewpoint position estimation model to estimate an offset and obtain a predicted position. Then, for each predicted position, the online phase S120 includes using the viewpoint direction estimation model to estimate a correct direction for each of the predicted positions. The online phase S120 then optionally includes applying a post-processing step to retain a minimum set of viewpoints that guarantee a certain range of the 3D room in the corresponding rendering and building a virtual visit with them.

[0069] The offline stage S110 is then discussed in more detail.

[0070] The library consists of 3D rooms with one or several viewpoints per room. The rooms may be considered to be well-furnished and have good quality viewpoints (renderings may be selected and purchased by the user). The number of viewpoints per room may be unbalanced in practice with most of the 3D rooms having only one viewpoint. The method is designed in such a way that the method may learn to generate several viewpoints for a single room even from a dataset that is unbalanced with only one viewpoint per room. Elements of the first and / or second dataset may correspond to pairs of 3D room representations and viewpoints (named ground truth) for the paired 3D rooms in the next section.

[0071] The offline stage S110 includes the determination S111 of an input representation of the 3D scene. The method may use any representation of the 3D room. For example, in a point cloud representation, the method may use a backbone to compute a 3D room representation. With reference to the list of prior art references provided above, this may be performed according to the teachings of [3]. In this example of implementation, the method uses a highly condensed scene graph representation based on the ideas of PlanIT. With reference to the list of prior art references provided above, this may be performed according to the teachings of [4]. The graph structure used for the scene representation is structured as described hereafter.

[0072] Any scene graph can be A list of nodes, each corresponding to a 3D_Elem in the room, with the following attributes: ○ Node ID, ○ The 3D bounding box (length, width, and height) of the 3D_Elem, ○ 3D_Elem's transformation matrix (to get the 3D_Elem's absolute position and orientation in the 3D room) A list of incoming edges, and A list of outgoing edges.

[0073] A list of edges (undirected) representing the relationships between the 3D_Elems in each of the rooms with the following attributes: Edge ID.

[0074] It is composed of:

[0075] The method may obtain a scene graph from an existing original scene graph (computed prior to the method) and may include extracting a scene graph from an existing initial scene graph. Extracting the scene graph may include removing any semantic information in the graph (e.g., no categorical attributes for nodes and edges). This results in a very light representation because it contains less information than the original 3D room representation. It also allows the model to generalize robustly with respect to furniture categories and room types.

[0076] For each input representation, determining S111 may include applying normalization to each of the node's attributes for stability purposes. The method may include determining a normalization factor to be applied for the dimensions of the room dimensions, as described below.

[0077] Define x0, y0, z0, x1, y1, z1 as the minimum and maximum coordinates of the room along each of the axes, and v = (x1-x0) + (y1-y0) + (z1-z0) as the volume of the 3D room. The method may include normalizing each of the object's positions and the ground truth camera viewpoint by p = (p-p0) / v. For x coordinates, that is x = (x-x0) / v. Each of the object's bounding box parameters is scaled by 1 / v.

[0078] The offline stage S110 includes the formation of a dataset S112. The strategy of learning to estimate several viewpoints for a given 3D room based on an imbalanced (or not) dataset relies on learning optimal local viewpoints and is detailed below. An imbalanced dataset means a dataset in which not all viewpoint annotations are present for all rooms, i.e. the dataset may contain one or several viewpoint annotations per 3D room. This also means that the method is able to learn to estimate an optimal set of viewpoints for a virtual visit (according to a defined objective) for a given 3D room, even if the number of viewpoints in the dataset does not cover the complexity of the entire room (i.e. to see a certain percentage of the objects in the room).

[0079] For each room, forming S211 includes defining a regular 2D grid to determine a set of starting points. It should be noted that this represents a convenient way of obtaining this set of starting points, but the method may rely on other ways of obtaining starting points (randomly, in a circular area of ​​the room, ...). The step size on each of the axes depends on the dimensions of the room. The method may, for example, select the following dimensions as step sizes:

[0080] Sx=Lx / N, where Lx is the dimension of the room along the x-axis, N is the selected parameter, and Sx is the step on the x-axis.

[0081] Sy=Ly / M, where Ly is the dimension of the room along the y-axis, M is the selected parameter, and Sy is the step on the y-axis.

[0082] The axes x and y may be orthogonal axes along which a room is defined.

[0083] After defining N and M (e.g., N=M=5) and obtaining M*N starting points, determining the set of starting locations may involve filtering the vertices of the defined regular 2D grid to retain only those vertices that are inside the 3D space of the room (defined by the floor plan in this example).

[0084] Formulation S112 includes determining an offset vector for each of the camera viewpoints. This selection of offset vectors for the training phase of the localization model is discussed next.

[0085] During the learning phase, the determination of the offset vector involves determining the ground truth camera position (the camera position of the camera viewpoint in the library) and searching for the closest start position (based on x and y coordinates) from the vertices of the grid retained after filtering. Once found, the determination involves calculating a 3D offset vector from the start position to the ground truth camera position, taking into account that the offset vector has a based elevation of H (e.g., H=1700 mm).

[0086] The x and y coordinates of the offset vector (in the absolute system) are normalized by Sx and Sy, respectively. The z coordinate is normalized by the room height and restricted to the [-0,5;0,5] domain. In cases where the x and y offset vector coordinates are not in the restricted [-0,5;0,5] domain, the method may involve applying the same proportionality factor to them such that max(||x||,||y||)=0.5.

[0087] The offline stage S110 includes training S113 of a first neural network (location estimation model). The first neural network may have an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model. The MLP model may take as input a concatenation of the output of the GNN model and the coordinates of the starting position. Since the method uses a scene graph representation of the 3D rooms, the first neural network may be a graph neural network (GNN) model to obtain a meaningful embedding for each of the 3D rooms. This embedding is then concatenated with the coordinates of the starting position and fed into a multi-layer perceptron model (MLP). The output of the multi-layer perceptron (MLP) includes the three coordinates of the offset vector.

[0088] In this example, the method uses the PNAConv architecture. Referring to the list of prior art references provided above, the use of the PNAConv architecture can be performed (within the state of the art of GNN architectures) according to the teachings of "PNAConv: A PNAConv-based Approach to Scene Graph Embeddings," Proc. Sci. 1999, 143:1311-1321. The method includes creating a unique network for obtaining scene graph embeddings based on the PNAConv block shown in FIG. 3.

[0089] 3 illustrates the computation of a new node representation by the PNAConv architecture. The node of interest is node 110, whose neighbor (linked by edge 121) is node 120.

[0090] The architecture used is a combination of four PNAConvs with a dimension of seven channels per node, followed by a batch normalization layer and a ReLU activation after each of them. The final graph representation is obtained by global pooling over the node features.

[0091] The MLP 130 has three linear layers, with the associated [input, output] dimensions for the three layers being [7+3,5], [5+3,3], and [3+3,3]. The +3 specified in each of the layers corresponds to the concatenation of the input position to the output of the previous layer, and can be seen as a skip connection mechanism. ReLU activation is applied on each of the non-final layer outputs. The activation function for the final output layer is f(x)=Sigmoid(x)-0.5, matching the target [-0.5;0.5] domain.

[0092] This is an example implementation, all blocks can be tuned by changing the number and type of layers, channel size, activation layers, pooling layers, etc. The main concept is to have a first neural network capable of estimating a high-level graph representation, and a second neural network that estimates a 3-dimensional vector representing the offset vector from this high-level representation, and a 3-dimensional vector representing the input position.

[0093] The training loss may measure the difference between the predicted offset vector and the ground truth offset vector. Many different losses may be used, such as the well-known L1 loss or L2 loss. In this example implementation, the L1 loss is used.

[0094] After this step, the method obtains a trained localization model.

[0095] The offline stage S110 includes the training S113 of a second neural network (orientation estimation model). The second neural network may have an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model. The MLP model may take as input the concatenation of the output of the GNN model and the coordinates of the camera position output by the first neural network. The orientation estimation network also uses a scene graph representation of the 3D room. And this architecture is similar to the position estimation model. The first GNN is used to obtain a meaningful embedding of the 3D room. This embedding is then concatenated with the three coordinates of the considered camera position and fed into a multi-layer perceptron model (MLP). The output of the MLP is the three coordinates of the orientation vector. During training, the considered camera position taken as input by the second neural network for the orientation prediction of the second neural network is the ground truth position. During the estimation phase, the second neural network takes as input the predicted positions obtained from the predictions of the first neural network.

[0096] The architecture used is a combination of four PNAConvs with a dimension of seven channels per node, followed by a batch normalization layer and a ReLU activation after each of them. The final graph representation is obtained by global pooling over the node features.

[0097] The MLP has two linear layers with the associated [input, output] dimensions of [7+3,5], [5,3]. The +3 specified in the first layer corresponds to the concatenation of the input position to the output of the previous layer. ReLU activation is applied on the output of the first layer.

[0098] Regarding the first neural network, this is an example implementation, all blocks can be tuned by changing the number and type of layers, channel size, activation layers, pooling layers, etc. The main concept is to have a first neural network capable of estimating a high-level graph representation, and a second neural network that estimates a 3D vector representing a direction vector from this high-level representation, and a 3D vector representing an input position.

[0099] The training loss may measure the difference between the predicted direction vector and the ground truth direction vector. Many different losses may be used, such as cosine or L2 loss. In this example implementation, cosine loss may be used.

[0100] After this step, the method obtains a trained direction estimation model.

[0101] The online phase S120 is then discussed.

[0102] Given a 3D room, the online phase S120 includes a first step S121, which involves first extracting its graph representation (e.g., this step may be performed as described above during the offline phase). The first step S121 then involves computing a set of starting (input) positions (e.g., this step may also be performed as described above during the offline phase).

[0103] The online stage S120 then includes an estimation step S122. For a given 3D scene, the online stage S120 includes determining a set of camera viewpoints by applying a first neural network and a second neural network to each of the start positions of a set of start positions regularly distributed on the plan view of the input 3D scene. The first neural network is applied first. The estimation step S122 includes using the trained position estimation model (first neural network) to estimate, for each start position of the calculated set, a final position obtained by adding an estimated offset vector for the start position to the start position, thereby obtaining a set of estimated final positions.

[0104] Then, the second neural network is applied. For each final position of the obtained set of estimated final positions, the estimation step S122 estimates a direction vector (camera direction) by applying the trained direction estimation model (second neural network) to the final position, thereby obtaining a set of camera viewpoints (each of the camera viewpoints is at the final position and direction vector). The method obtains this set of camera viewpoints by applying the first neural network and the second neural network to each of the start positions of a set of start positions regularly distributed on the plan view of the input 3D scene.

[0105] Even if the results obtained in the previous step may already be used to define the virtual visit, the online step may include a post-processing step to avoid too much redundancy in the proposed viewpoints. This post-processing step may be useful to improve the user experience (e.g. avoid too much redundancy in the proposed viewpoints), especially when the number of starting positions is very large (e.g. more than 5). Optionally, the machine learning function may therefore include a post-processing block. The post-processing block may be configured to select one or more camera viewpoints to be output from a set of camera viewpoints determined by applying the first and second neural networks to each of the starting positions of the set of starting positions regularly distributed on the floor plan of the input 3D scene.

[0106] The selection of the output camera viewpoint or viewpoints may be performed to optimize the number of objects visible in the camera viewpoints. For example, the selection may include determining, for each camera viewpoint, the 3D objects of the input 3D scene that are visible in the 2D rendering generated from the camera viewpoint. The selection may then include selecting one or more of the determined camera viewpoints, such that the 3D objects visible in the one or more 2D renderings generated from the one or more selected camera viewpoints represent at least a predetermined percentage of all 3D objects present in the 3D scene. The visibility of an object in the 2D rendering may refer to the visibility of vertices or faces of the object's bounding box in the 2D rendering. Alternatively, an object may be visible in the 2D rendering if at least one of the following criteria is met:

[0107] The percentage of pixels in the 2D rendering (image) that correspond to the object is greater than a threshold (e.g., 1% of the number of pixels in the 2D rendering); and / or The percentage of the object's faces that are visible from the viewpoint is greater than a threshold (for example, 10%).

[0108] To select the minimum number of predicted viewpoints for a given 3D room, the function may include a post-processing step that aims to select the minimum number of viewpoints that present the best range that can be obtained from the set of estimated viewpoints. The definition of range is discussed hereafter. The post-processing step may be performed by executing the following exemplary algorithm for selecting estimated viewpoints. In a first step, the algorithm may include instructions for storing a set of objects seen for each of the viewpoints, and for storing together (e.g., in a database) a list of objects seen by all viewpoints. The algorithm may then include instructions for sorting the list of viewpoints in descending order with respect to the number of objects seen per viewpoint. The algorithm may then include instructions for adding the first element of the list to the list of selected viewpoints, and for removing that first element from the list of viewpoints, which are now named remaining viewpoints. The algorithm may then include instructions for going through all the remaining lists of viewpoints while the number of objects visible in the list of selected viewpoints is less than a certain percentage of the list of objects visible by all viewpoints (calculated in the first step), selecting a viewpoint that contains the highest number of invisible objects with respect to the list of objects visible from the set of selected viewpoints, and adding it to the list of selected viewpoints and removing it from the list of remaining viewpoints.

[0109] Example results of the machine learning functions are then discussed.

[0110] The results are analyzed using a first metric named relative coverage and a second metric named total coverage. The relative coverage is the ratio between the number of objects visible from the selected estimated viewpoint and the total number of objects visible from the estimated viewpoint. The total coverage is the ratio between the number of objects visible from the selected estimated viewpoint and the total number of objects in the scene. The results show the efficiency of the method.

[0111] The network was trained on about 50,000 3D bedrooms and evaluated on 200 living rooms as proof of generalization, obtaining the following results. The results prove that the machine learning function is suitable to automatically estimate the viewpoints of any given 3D room. In particular, the machine learning function offers the possibility to estimate the minimum number of viewpoints for observing the main furniture of the room (post-processing step). The average number of objects per scene is 72.16. The average number of selected renderings is 2.07. The average relative coverage is 0.95. The average total coverage is 0.74. Figures 4 to 8 show 2D renderings obtained from the camera viewpoints estimated by the function (estimation step). These results are obtained for different room types for a model trained on 50,000 3D bedrooms. In particular, Figure 4 shows the first living room used in this first example, and Figure 5 shows three 2D renderings obtained from the three camera viewpoints estimated by the function for this first living room. Figure 6 shows the bedroom used in this second example, and Figure 7 shows three 2D renderings of this bedroom obtained from the three camera viewpoints estimated by the function. Figure 8 shows the second living room used in this second example, and two 2D renderings of this second living room obtained from the two camera viewpoints estimated by the function.

[0112] The first data set and / or the second data set may have the following characteristics: The first data set and / or the second data set may be 10 m 2 and / or 50m 2 Smaller average area (e.g. 16, 7 m 2 The first dataset and / or the second data may include training samples of 3D scenes with an average number of objects per scene greater than 10 and / or less than 50 (e.g., an average number of objects per scene of 32.2). The first dataset and / or the second data may include training samples of 3D scenes with an average total extent greater than 0.25 and / or less than 0.75 (e.g., 0.51). Optionally, the camera positions may be limited in height (e.g., 2 m) and always within the boundaries of the room. For example, the method may include filtering 3D scenes from the library with camera positions that do not adhere to those criteria.

[0113] FIG. 9 shows an example of the architecture of the machine learning function. The machine learning function includes a first neural network (position model) and a second neural network (rotation model). The first neural network has an architecture including a graph neural network (GNN) model 210 and a multi-layer perceptron (MLP) model 212. The MLP model 212 takes as input a concatenation 211 of the output of the GNN model 210 and the coordinates of the starting position. The MLP model 212 outputs an offset vector 215. The final position 213 is calculated by appending the output offset vector 213 to the initially considered starting position 214. The first neural network takes as input the room graph 200 and the initial position and predicts the camera position 213. The first neural network is trained using an L1 loss, f(G,pos init ) and the result is pos pred =posinit +f(G,pos init )

[0114] The second neural network has an architecture that includes a graph neural network (GNN) model 220 and a multi-layer perceptron (MLP) model 222. The MLP model 222 takes as input a concatenation 221 of the output of the GNN model 220 and the coordinates of the camera position output by the first neural network (i.e., the final position 213, which is the result of the addition of the offset vector 213 output by the first neural network and the initial starting position 214). The second neural network 220 takes as input the graph and the position to predict the rotation about the camera. The second neural network is trained using cosine distance (1-cosine similarity) and attempts to learn h(G,pos) and, as a result, the rot pred =h(G,pos).

[0115] Figure 10 shows an alternative example of a functional architecture. In this alternative, an alternative scene representation is used. Instead of a graph, the alternative scene representation is an unordered set of 3D_Elems (i.e., bounding boxes and positions of objects). For example, each of the 3D_Elems may be represented by:

[0116] The 3D bounding box (length, width, and height) of the 3D_Elem, and The transformation matrix of the 3D_Elem (to get the absolute position and orientation of the 3D_Elem in the 3D room) Therefore, the neural network can be based on a transformer architecture instead of a GNN to obtain a scene representation concatenated with the position for both the position estimation model and the orientation estimation model. Figure 10 shows such a transformer.

[0117] In this figure, PE 310 is a positional encoding used to project each of the scalars into a high-dimensional space. With reference to the list of prior art references provided above, this positional encoding can be performed according to the teachings of [6]. x, y, z represent the position of the center of the bounding box, l, w, h represent the size of the bounding box, and θ represents the rotation around the z-axis. The transformer can be implemented as in https: / / pytorch.org / docs / stable / modules / torch / nn / modules / transformer.html#Transformer (which can be performed according to the teachings of [7]).

[0118] FIG. 11 shows an example of a set of starting positions 410, 420. The set of starting positions 410, 420 are regularly distributed on a plan view 400 of a 3D scene taken as input by a first neural network. The starting positions 410, 420 of the set are substantially equidistant from each other. For each starting position, the first neural network is configured to take as input an input representation of the 3D scene and a starting position in the 3D scene, and to output an offset vector between the camera position and the starting position. For example, for the starting position 420, the first neural network is configured to take as input an input representation of the 3D scene and the starting position 420, and to output an offset vector 430 between the camera position 440 and the starting position 420. The method may infer the camera position 440 estimated by the first neural network for the starting position 420 by adding the output offset vector 430 to the starting position 420 taken as input.

[0119] FIG. 12 shows an example of a system, which is a client computer system, for example a user's workstation.

[0120] The client computer of this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass memory devices, such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks. Any of the foregoing may be supplemented by or incorporated into specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, and the like. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. In addition, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes some signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the buttons of the mouse are used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0121] The computer program may include instructions executable by the computer, which instructions include means for causing the system to carry out the method. The program may be recordable on any data storage medium, including the memory of the system. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, for example, an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by the programmable processor executing a program of instructions for performing the functions of the method by operating on input data to generate output. The processor may therefore be programmable and may be coupled to receive data and instructions from, and to transmit data and instructions to, the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. The application of the program on the system, in either case, results in instructions for carrying out the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, the server being in communication with one or more clients via a network, in which case a processing unit executes instructions contained by the program, thereby causing a method to be performed in the cloud computing environment.

Claims

1. 1. A computer-implemented method for determining a machine learning function configured to take an input 3D scene and to output each of one or more camera viewpoints for generating a 2D rendering of each of the 3D scenes, the method comprising: - obtaining (S10) a library comprising 3D scenes and, for each of the 3D scenes, a respective one or more camera viewpoints for generating a 2D rendering of each of said 3D scenes; Based on the library, forming (S20) a first data set for training a first neural network configured to take in a first input representation of a 3D scene and to output information corresponding to a camera position in said 3D scene; forming (S30) a second data set for training a second neural network configured to take a second input representation of a 3D scene and to output information corresponding to a camera orientation in said 3D scene; training the first neural network based on the first data set and training the second neural network based on the second data set (S40), each of the camera viewpoints output by the machine learning function includes a camera position corresponding to the information output by the first neural network and a camera orientation corresponding to the information output by the second neural network; A method for providing the above.

2. The forming (S20) of the first data set may comprise, for each 3D scene: determining (S21) a set of starting positions regularly distributed on a plan view of the 3D scene; determining (S22) for each camera viewpoint of the 3D scene an offset vector between the camera position of said camera viewpoint and the closest starting position, 2. The method of claim 1, wherein the first neural network is configured to further take as input a starting position in the 3D scene, and the information output by the first neural network includes an offset vector between a camera position and the starting position.

3. The method of claim 2 , wherein the starting positions of each of a set are substantially equidistant from each other, and the distance between the starting positions is substantially the same for each of the 3D scenes.

4. 4. The method of claim 2 or 3, wherein the starting positions of each of a set are distributed to form a regular 2D grid on the plan view of the 3D scene, each starting position corresponding to a respective vertex of the regular 2D grid.

5. The method of claim 1 , wherein the first input representation and / or the second input representation comprises a set of geometric properties of 3D objects of the 3D scene.

6. The method of claim 5 , wherein the geometrical characteristics include position, orientation and / or dimension information, and optionally bounding boxes, representing each 3D object in the 3D scene.

7. 7. The method of claim 5 or 6, wherein the first input representation and / or the second input representation comprises a graph comprising respective nodes representing respective objects of the 3D scene and respective arcs linking respective pairs of nodes, the 3D objects represented by the nodes of a pair being positioned according to a mutual placement relationship from a predetermined set of one or more mutual placement relationships.

8. The method of claim 7 , wherein the predetermined set of one or more mutual placement relationships includes an adjacent relationship, a proximate relationship, a distance relationship, and / or a vertical overlap relationship.

9. The method of claim 1 , wherein the second neural network further takes as input the camera position output by the first neural network.

10. The first neural network has an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model, the MLP model taking as input a concatenation of the output of the GNN model and a coordinate of a starting position; and / or 10. The method of claim 1, wherein the second neural network has an architecture including a graph neural network (GNN) model and a multi-layer perceptron (MLP) model, the MLP model taking as input a concatenation of the output of the GNN model and the coordinates of the camera position output by the first neural network.

11. 11. The method according to claim 1, wherein the machine learning functionality comprises a post-processing block configured to select the output camera viewpoint or viewpoints from a set of camera viewpoints determined by applying the first neural network and the second neural network to each of a set of start positions regularly distributed on a plan view of the input 3D scene.

12. The selection of one or more camera viewpoints to be outputted may further comprise: - for each camera viewpoint, determining the 3D objects of the input 3D scene that are visible in a 2D rendering generated from that camera viewpoint; The method of claim 11, further comprising: selecting one or more of the determined camera viewpoints, whereby the 3D objects visible in one or more of the 2D renderings generated from one or more selected camera viewpoints represent at least a predetermined percentage of all the 3D objects present in the 3D scene.

13. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having recorded thereon a machine-learned first neural network as claimed in any one of claims 1 to 12, a machine-learned second neural network as claimed in any one of claims 1 to 12, and / or a computer program as claimed in claim 13.

15. 14. A system including a processor coupled to a memory, the memory having stored thereon the computer program of claim 13.

Images