3D scene understanding

Abstract

To provide an improved method, system and program for 3D scene understanding.SOLUTION: A method comprises: Step S10 of obtaining a dataset of 3D point clouds annotated, for each 3D point cloud, with localized representations each of a respective object and, for each respective object, with a class of the respective object among the predetermined set of classes: and Step S20 of training the function based on the obtained dataset. The predetermined set of classes comprises a plurality of semantic classes and a plurality of geometric classes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for 3D scene understanding. [Background technology]

[0002] Current state-of-the-art methods for 3D scene understanding are based on machine learning functions that are trained in a supervised manner on a dataset of point clouds of 3D scenes. These functions are then able to detect objects in point clouds acquired, for example, by scanning a real scene. To train these functions, the point clouds are typically annotated with localized representations of objects in the 3D scene and with semantic annotations, such as the semantic classes of the objects. However, to ensure that each semantic class contains enough examples in the dataset and that these examples represent the semantic class to which they belong, the number of these semantic classes used to train the machine learning functions is very limited (e.g., 10 to 20 classes). This results in training functions that can process only objects belonging to this limited number of semantic classes while ignoring the vast number of remaining objects. This generally hinders the automation of 3D scene understanding tasks.

[0003] Self-supervised techniques aim to overcome the problem of a lack of annotated examples in a dataset by proposing pre-training of a function for a pre-text task. The pre-training is supervised using data augmentation and masking techniques, and the function learns an intermediate representation of the data and understands its underlying structure. The function is then fine-tuned in a supervised manner on a labeled dataset, i.e., further trained for a specific downstream task such as object detection. While these methods achieve better results for a given semantic class from the labeled dataset used for fine-tuning, they do not address the problem of objects from unconsidered semantic classes still being ignored.

[0004] Open vocabulary methods utilize functions trained on other modalities, such as images and text, which have much denser and richer datasets than those with 3D data. This allows for a broader diversity of classes addressed by these functions. However, they rely heavily on large datasets of image-text pairs, typically gleaned from the Internet and not necessarily copyright-free (a recent lawsuit on this issue is still pending). These methods also typically require significantly more computational power and have been shown to fall short of fully supervised approaches for the same number of classes in labeled 3D datasets. Furthermore, when used to detect loosely defined or ambiguous semantic classes, such as "objects" or even "small objects," they still require a list of categories to detect and exhibit poor performance.

[0005] Within this context, there remains a need for improved solutions for 3D scene understanding. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] European Patent Application No. 23305001.2 [Non-patent literature]

[0007] [Non-Patent Document 1] Hou, Ji, et al. “Exploring data-efficient 3d scene with contrastive scene contexts,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,2021(https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Hou_Exploring_Data-Efficient_3D_Scene_Understanding_With_Contrastive_Scene_Contexts_CVPR_2021_paper.pdf) [Non-patent document 2] Alireza Fathi and Rui Huang about 3D Scene Understanding with TensorFlow3D(https: / / ai.googleblog.com / 2021 / 02 / 3D-scene-understanding-with-tensorflow.html) [Non-patent document 3] Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space”, 2017 [Non-patent document 4] Xian-Feng Han, Shi-Jie Sun, Xiang-Yu Song, Guo-Qiang Xiao, “3D Point Cloud Descriptors in Hand-crafted and Deep Learning Age:State-of-the-Art”, 2020 Summary of the Invention

[0008] Thus, there is provided a computer-implemented method for machine learning a function configured to take as input a 3D point cloud of a real scene and output a respective localized representation of each object in the real scene and, for each object, a class of the respective object from a predetermined set of classes. The method is hereinafter referred to as the machine learning method, or simply the method. The method includes obtaining a dataset of 3D point clouds annotated with, for each 3D point cloud, a respective localized representation of each object and, for each object, a class of the respective object from the predetermined set of classes. The method includes training the function based on the obtained dataset. The predetermined set of classes includes a plurality of semantic classes and a plurality of geometric classes.

[0009] The machine learning methods may include one or more of the following: - Obtaining a dataset is Obtaining a 3D point cloud, which includes: For each 3D point cloud, a localized representation of each object, and For each object in at least some of the objects, the class of each object in the semantic class It is annotated with Calculating geometrical descriptor values ​​for each remaining object in the 3D point cloud; clustering the remaining objects according to the partitioning of the distribution of the calculated values, whereby multiple geometry classes correspond to the resulting clusters; Including, - the geometric descriptor is invariant with respect to orientation at least with respect to the normal axis, - the geometric descriptor of a given object contains metrics of the bounding box of the given object, - The geometric descriptor of a given object is one or more coordinates each representing a dimension of the bounding box of a given object, o A coordinate representing the ratio between the minimum and maximum dimensions of the bounding box of a given object, ○ Coordinates representing the area of ​​the bounding box of a given object, and / or ○ Coordinates that represent the volume of the bounding box of a given object Including, - Obtaining a dataset is Obtaining a 3D point cloud, which includes: ● For each 3D point cloud, a localized representation of each object, For each of at least some of the objects, the class of each object in the semantic class; and For each remaining object, the class of the remaining object among other semantic classes It is annotated with ○ Identifying, among all objects, at least some of the objects of other semantic classes; ○ Assigning a geometry class to each identified object Including, - identifying at least some of the objects of the other semantic class includes filtering all objects of the other semantic class based on at least one geometric criterion; - at least one geometric criterion is ○ Bounding box volume based criteria, ○ Criteria based on the ratio between the minimum and maximum bounding box dimensions, and / or ○ Criterion based on the result of multiplying the bounding box volume by the ratio Including, - The function is configured to perform 3D object detection or 3D object segmentation, - the 3D point cloud of the dataset is a 3D point cloud of an indoor scene and / or is obtained via scanning; and / or The function has an architecture that includes a voxelization layer configured to voxelize a 3D point cloud taken as input, and / or a convolutional neural network that takes as input the 3D point cloud voxelized by the voxelization layer.

[0010] Further provided is a computer-implemented method for using a machine-learned function according to a machine learning method, hereinafter referred to as the "method of using." The method of using optionally includes acquiring a 3D point cloud by scanning a real scene. The method of using includes applying the function to the acquired 3D point cloud.

[0011] Further provided is a computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the machine learning method and / or the method of use.

[0012] Additionally, a computer readable recording medium having a computer program recorded thereon is provided.

[0013] Further provided is a system comprising a processor coupled to a memory and a graphical user interface, the memory having a computer program recorded thereon.

[0014] Additionally, there is provided a device comprising a data storage medium having a computer program recorded thereon.

[0015] The device may form or act as a non-transitory computer-readable medium, such as on a software-as-a-service (SaaS) or other server- or cloud-based platform. Alternatively, the device may comprise a processor coupled to a data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further include a graphical user interface coupled to the processor.

[0016] Non-limiting examples will now be described with reference to the accompanying drawings in which: [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a flowchart of an example method. [Figure 2] FIG. 1 shows the distribution of the number of object occurrences for all semantic classes. [Figure 3] FIG. 1 illustrates an example of an object that belongs to a poorly represented or poorly defined semantic class. [Figure 4] FIG. 1 illustrates an example of a geometric descriptor. [Figure 5] FIG. 1 illustrates an example of ordering objects based on geometric criteria. [Figure 6] FIG. 10 illustrates an example of clustering of remaining objects. [Figure 7] FIG. 10 illustrates an example of clustering of remaining objects. [Figure 8] FIG. 1 shows an example of a method used. [Figure 9] FIG. 1 shows an example of a method used. [Figure 10] FIG. 1 illustrates an example of a metric used to measure the performance of a trained function. [Figure 11]FIG. 1 illustrates an example of a metric used to measure the performance of a trained function. [Figure 12] FIG. 1 illustrates an example of 3D scene understanding. [Figure 13] FIG. 1 illustrates an example of a function architecture. [Figure 14] FIG. 10 illustrates an example of the results of a trained function. [Figure 15] FIG. 1 illustrates an example of a system. DETAILED DESCRIPTION OF THE INVENTION

[0018] With reference to the flowchart of Figure 1, a computer-implemented method for machine learning a function configured to take as input a 3D point cloud of a real scene and output a respective localized representation of each object in the real scene and, for each object, a class of the respective object from a predetermined set of classes. The method is hereinafter referred to as the machine learning method, or simply the method. The method includes obtaining a dataset of 3D point clouds annotated with, for each 3D point cloud, a respective localized representation of each object and, for each object, a class of the respective object from the predetermined set of classes. The method includes training the function based on the obtained dataset. The predetermined set of classes includes a plurality of semantic classes and a plurality of geometric classes.

[0019] Such methods form an improved solution for 3D scene understanding.

[0020] In particular, the method improves the training of machine learning functions in the field of 3D scene understanding. In particular, it makes it possible to overcome both the lack of annotations and the under-representation of some semantic classes in datasets used to train machine learning models in the field of 3D scene understanding. Indeed, adding geometrical classes to supervise the training of neural networks in the field of 3D scene understanding has several advantages.

[0021] First, publicly available datasets provide a limited number of semantic classes as annotations for training machine learning models. This leads to training models that can only understand the classes of objects they are trained on and therefore ignore a huge number of objects in their reasoning on real-life scenes. The method allows these elements to be considered, if not by semantic class, then at least by geometry or shape. Because the model is trained on both semantic and geometric classes, it can detect more objects than those belonging to the semantic classes given in the training dataset.

[0022] This also allows the model to better understand the semantic classes used for training (in addition to geometric classes). Indeed, the use of geometrically defined classes reduces ambiguity regarding semantically defined ones. That is, in addition to being able to handle a greater quantity and variety of objects, a model trained on such classes also reaches better performance in terms of handling semantically defined objects.

[0023] Furthermore, the method enables a more accurate understanding of 3D scenes. Indeed, because the training of the function is more relevant, the machine-learned function can detect more objects with better performance. Thus, the method enables training the function to perform more accurate and relevant 3D object detection and / or 3D object segmentation based on scans of real scenes. The method enables training the function to reconstruct a 3D representation that is closer to the real scene and / or to more accurately register 3D point clouds.

[0024] The machine learning method and / or method of use may be computer-implemented. This means that the steps (or substantially all steps) of the machine learning method and / or method of use are performed by at least one computer or any similar system. Thus, the steps of the machine learning method and / or method of use are performed by a computer, possibly fully automatically or semi-automatically. In an example, triggering at least some of the steps of the machine learning method and / or method of use may be performed through user-computer interaction. The level of user-computer interaction required depends on the level of automation envisioned and can be balanced with the need to implement the user's wishes. In an example, this level may be user-defined and / or predetermined.

[0025] A typical example of a computer implementation of the machine learning method and / or the method of use is to perform the machine learning method and / or the method of use using a system adapted for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), where the memory records a computer program including instructions for performing the generating method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally include several physically distinct parts (e.g., one for the program and possibly one for the database).

[0026] By "database" is meant any collection of data (i.e., information) organized for search and retrieval (e.g., a relational database based on a predetermined structured language, e.g., SQL). When stored in memory, a database allows for rapid search and retrieval by a computer. Databases are actually constructed to facilitate the storage, retrieval, modification, and deletion of data, along with various data processing operations. A database may consist of a file or set of files that may be broken down into records, each consisting of one or more fields. A field is the basic unit of data storage. A user can retrieve data primarily through queries. Using keywords and sorting commands, a user can quickly find, rearrange, group, and select fields within many records, thereby retrieving or creating reports on a particular collection of data according to the rules of the database management system being used.

[0027] The method used may be included in a process that may include applying a function to the acquired 3D point cloud and then performing 3D scene understanding of the real scene based on the localized representations and classes of objects output by the function. 3D scene understanding may be a potentially real-time process that includes, for example, perceiving, analyzing, and / or elaborating an interpretation of the real scene (e.g., the 3D point cloud taken as input by the function) observed via a network of sensors. For example, 3D scene understanding may include performing 3D object detection, 3D object segmentation, 3D scene reconstruction, 3D point cloud registration, and / or 3D mesh transformation based on the localized representations and classes of objects output by the function. As the method improves the output of the localized representations and classes of objects, the method also improves the execution of 3D scene understanding of the real scene.

[0028] 3D scene understanding can be a process performed based on 3D input data. For example, the 3D input data can be a 3D point cloud taken as input by a function (e.g., obtained by scanning a real scene). 3D scene understanding can include, for example, 3D detection, such as in the Fully Convolutional Anchor-Free 3D Object Detection (FCAF3D) method, or the process described in the document (Non-Patent Document 1), which is incorporated herein by reference. As described in this document, 3D scene understanding can include collecting and labeling a real 3D scene.

[0029] Alternatively or additionally, still in the case of processes performed based on 3D input data, 3D scene understanding may include performing any one or any combination of the applications described in the resources from (Non-Patent Document 2), which are incorporated herein by reference.

[0030] Alternatively or additionally, performing 3D scene understanding may include performing 3D object detection. For example, each localized representation output by the function may represent an envelope of a respective real-world object in the real scene (e.g., the localized representation may include a 3D surface representing a real face of the respective real-world object). For example, when a 3D point cloud is acquired by scanning a real scene, each localized representation may surround sampled points on the same real-world object in the real scene (i.e., on the surface of this real-world object). Parameters of each localized representation may include a semantic or geometric class, a center position, dimensions, and / or orientation. Performing 3D scene understanding may include, for each output localized representation, determining the object represented by the localized representation based on, for example, the class and / or shape of the envelope represented by the localized representation. Alternatively, the function can be trained to directly output, for each localized representation, the object represented by the localized representation.

[0031] Then, performing 3D scene understanding may include performing 3D scene reconstruction. The 3D scene reconstruction may include determining a 3D representation of the real scene based on the output of the function. For example, reconstructing the 3D representation may include retrieving 3D representations (e.g., computer-aided design (CAD) models) of objects in the real scene (e.g., from a database storing CAD models of the objects) and assembling these 3D representations (each 3D representation representing a respective real object in the real scene) in the 3D scene according to the localized representations output by the function. The assembled 3D objects may be 3D objects determined for each localized representation and may be assembled according to the respective positions and orientations of the output localized representations. The 3D scene reconstruction may also include reconstructing walls and floors of the real scene based on the 3D point cloud and adding the reconstructed walls and floors to the 3D representation of the real scene.

[0032] Alternatively or additionally, performing 3D scene understanding may include performing 3D object segmentation of the 3D point cloud. Segmentation may include labeling points (e.g., all points) of the 3D point cloud according to the output localized representation and / or output class of the object. Point labeling may include assigning a respective label to each point according to the output localized representation and / or output class of the object to which the point belongs. Points assigned to the same label (i.e., points in the same segment) may correspond to points sampled on the same real-world object in the real scene (i.e., on the surface of this real-world object). In that case, the function may output the label assigned to each point of the 3D point cloud. The function may be trained to output, in addition to the localized representation, groups of points belonging to each localized representation. Each group of points may collect points assigned to the same label, for example, points sampled on the same real-world object in the real scene (i.e., on the surface of this real-world object).

[0033] Alternatively or additionally, performing 3D scene understanding may include performing a 3D mesh transformation. The 3D mesh transformation may include converting the 3D point cloud into a mesh representing the real scene. The mesh of the 3D point cloud may include, for each object, a respective mesh representing the object (e.g., including a surface representing the object). In that case, each output localized representation may include a mesh of the object it represents, and the transformation may include assembling the mesh based on the position and / or orientation of the output localized representation. Alternatively, the transformation may include, for each output localized representation, creating a mesh for each object (e.g., based on parameters and / or semantic or geometric classes of the localized representation) and assembling all the created meshes.

[0034] Alternatively or additionally, performing 3D scene understanding may include registering the 3D point cloud. Registering the 3D point cloud may include aligning one or more portions of the 3D point cloud with a reference point cloud (e.g., corresponding to a scan of an object), respectively. Registering may include, for each portion, determining a reference point cloud for that portion and aligning the portion with the determined reference point cloud. Aligning each portion may include determining rotation and / or scale parameters for aligning the portion with the determined reference point cloud. In an example, the function may also be configured to output one or more portions, a reference point cloud for each portion, and / or an alignment for each portion (i.e., rotation and / or scale parameters for alignment).

[0035] In an example, the process may also include executing a machine learning method before executing the using method. In other words, the process may include, first, a machine learning method for training a function, second, a using method for applying the trained function, and third, performing 3D scene understanding based on an output of the function. Alternatively, training the function may be performed before the process. In that case, the machine learning method may be performed before executing the process, for example, during an offline phase, and the process may be performed later, for example, during an online phase, based on the trained function.

[0036] After training the function, the method includes acquiring a 3D point cloud. Acquiring the 3D point cloud may be performed, for example, by scanning a real scene based on one or more sensors. Alternatively, the 3D point cloud may already be calculated (e.g., by scanning a real or virtual scene) and recorded in a database. In that case, acquiring may include extracting the 3D point cloud from the dataset. After acquiring the 3D point cloud, the method includes applying the trained neural network to the acquired point cloud, thereby performing 3D scene understanding. For example, the function may output a localized representation of each object in the real scene and, for each object, a class of each object from a predetermined set of classes, and the method may include performing 3D scene understanding (e.g., 3D object detection, 3D object segmentation, 3D scene reconstruction, 3D point cloud registration, and / or 3D mesh transformation) based on the localized representations and classes of the objects output by the function. Alternatively, the function may be trained to directly output the results of 3D scene understanding (i.e., detected 3D objects, segments of the 3D point cloud, a reconstructed 3D scene, a registered 3D point cloud, and / or a mesh of the resulting 3D point cloud).

[0037] Next, the acquisition of the data set S10 will be described in more detail.

[0038] The dataset includes a respective 3D point cloud for each real scene containing an object (e.g., acquired by scanning the respective real scene). Each 3D point cloud of the dataset is annotated with a localized representation representing each object of the respective real scene. For example, the localized representations may together form a virtual representation of the real scene (e.g., the virtual representation may be recorded on a file), and the dataset may include a link between the 3D point cloud and the virtual representation (e.g., together with the file containing the localized representation). The virtual representation may also include, for each object of the real scene, the class of the object. Alternatively or additionally, each point of the 3D point cloud may be labeled with the localized representation and / or class of the object to which it belongs. The labeling of each point may be included in the 3D point cloud (e.g., using variables in addition to the position parameters of each point). The 3D point cloud taken as input by the function is not annotated. A function is trained to annotate this 3D point cloud, i.e., to compute localized representations and classes of objects present in the real scene from which this 3D point cloud is acquired.

[0039] The 3D point clouds of the dataset may be obtained from respective real scenes (or virtual representations of these respective real scenes), which may be indoor scenes such as an office, an apartment, and / or a store. The real scenes for which the dataset includes 3D point clouds may all be different. The real scenes may correspond to the same environmental category (e.g., office, apartment, and / or store). Each real scene may include objects that naturally exist in such a category (e.g., an office table and chairs), and these objects may be positioned relative to each other. Each real scene may also include (e.g., flat) floors and / or walls that define the real scene and / or one or more rooms it contains. Each real scene may be an environment that exists in the real world or may be imaginary (but may exist in the real world). For example, the method may include generating one or more (e.g., all) of the real scenes (e.g., randomly and / or based on a real-world environment).

[0040] Each real scene may include one or more (i.e., real) objects. Some (e.g., all) of the real scenes in the dataset may be multi-part rooms, i.e., each may include several objects (e.g., more than 10 or more than 100 objects), and / or each may be non-rectangular in shape (e.g., have five or more walls, and / or include one or more circular walls). Each localized representation annotating the 3D point cloud may represent the geometry of a real object placed (or to be placed) in the real scene. The real object may be manufactured in the real world following completion of its virtual design.

[0041] Each 3D point cloud of the dataset is also annotated with a class of each object in a predetermined set of classes for each object in the real scene. The predetermined set of classes includes multiple semantic classes. For example, the multiple semantic classes may include one or more classes of furniture objects and / or one or more classes of decorative objects, or any other combination of object class types. Any furniture objects herein may have a furnishing function in the real scene in which they are placed. For example, the real scenes of the dataset may each include one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more sideboards, one or more small tables, one or more desks, and / or one or more wardrobes. The predetermined set of classes may include a respective class for each of these furniture objects. Any decorative objects herein may have a decorative function in the room in which they are placed. For example, the real scenes of the dataset may each include one or more accessories, one or more plants, one or more books, one or more frames, one or more kitchen utensils, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more rugs, one or more mirrors, and / or one or more electronic objects (e.g., a refrigerator, a freezer, and / or a washing machine). The predetermined set of classes may include a respective class for each of these decorative objects.

[0042] The predetermined set of classes also includes multiple geometry classes. The multiple geometry classes may complement the multiple semantic classes. Each object may have a class among the semantic classes or a class among the geometry classes. Each geometric class may be defined by geometric criteria applied to a geometry or shape. For example, a geometric class may be defined using only the geometric properties of an object (such as its shape and / or size). A geometric class may be defined using only the geometric properties of its localized representation (e.g., the shape and / or size of its bounding box). A geometry class may be assigned to objects that are under-represented in the dataset. For example, these objects may appear in less than a predetermined percentage of real scenes (e.g., less than 10% of real scenes).

[0043] Acquiring the dataset S10 may include generating a 3D point cloud of the dataset based on a point cloud generation method, such as the method for generating a training dataset disclosed in U.S. Patent Application Publication No. 2009 / 0229994, which is incorporated herein by reference. The point cloud generation method may include acquiring each 3D point cloud in any manner, for example, by scanning a real scene. Each 3D point cloud may include points sampled on the surface of objects present in the real scene. Each point in the 3D point cloud may include spatial coordinates in 3D space. Scanning may be performed by a user using a scanner within the real scene. Alternatively, scanning of the real scene may be performed virtually by scanning a virtual representation of the real scene. This virtual representation may include virtual representations of objects present in the real scene (e.g., reproducing the shapes of these objects). Scanning may include sampling points on the surface of these virtual representations of objects present in the real scene. Each real scene may be an environment present in the real world or may be imaginary (e.g., created and then constructed by a user to visualize a potential real scene). Thus, the dataset may include a 3D point cloud obtained by virtually scanning a virtual representation of a real scene.

[0044] Alternatively, each 3D point cloud of the dataset may have already been generated at the time the method is performed (e.g., using the same point cloud generating method described above). In that case, obtaining the dataset S10 may include retrieving the already generated 3D point clouds, for example, from a database in which they are stored. Further alternatively, some of the 3D point clouds may have already been generated at the time the method is performed, while other 3D point clouds may not. In that case, obtaining S10 may include retrieving the already generated 3D point clouds and generating the other 3D point clouds, for example, by performing the same point cloud generating method described above.

[0045] Acquiring the dataset S10 may also include annotating each 3D point cloud of the dataset, for example, based on an annotation method. The annotation method may include determining a localized representation and a class of an object to annotate the 3D point cloud. The annotation method may be performed in any manner. For example, determining the localized representation and the class of the object may be performed manually by a user. The annotation method may include, for each localized representation, placing the localized representation in a virtual environment representing the real scene and assigning to the placed localized representation a class from a predetermined set of classes for the object represented by the localized representation. Alternatively, the annotation method may be performed automatically. For example, the 3D point cloud may be obtained from a virtual scan of the virtual environment as described above. In that case, the localized representation may correspond to a virtual representation of an object already present in the virtual environment, and the annotation method may include retrieving the localized representation from the virtual environment. The virtual environment may also include a class for each object represented in the virtual environment, and the method of annotating may include retrieving the class of each localized representation from the virtual environment.

[0046] Alternatively, each 3D point cloud of the dataset may already be annotated (e.g., using the annotation method described above) at the time the method is performed. In that case, obtaining the dataset S10 may include, for example, retrieving the already annotated 3D point clouds from a database in which they are stored. Further alternatively, some of the 3D point clouds may already be generated and annotated at the time the method is performed, while other 3D point clouds may not. In that case, obtaining S10 may include retrieving the already annotated 3D point clouds and annotating the other 3D point clouds, for example, by performing the same annotation method described above.

[0047] Training the function S20 may be performed in any manner. For example, training the function S20 may be performed in a supervised manner. Training the function S20 may include training the function to predict annotations (i.e., localized representations and classes) for a given 3D point cloud based on example annotation and 3D point cloud pairs present in the dataset (i.e., for training). The function is trained to output representations of objects present in a real scene from which the 3D point cloud taken as input was acquired, where each representation of the objects is localized, i.e., it includes the object's position coordinates in the real scene. The function is also trained to output the class of each object (e.g., along with a label assigned to the localized representation). Training (S20) may include determining weights for the function such that the function can provide output annotations that closely match the annotations of the 3D point clouds the dataset contains.

[0048] In an example, obtaining S10 may include determining multiple geometry classes based on objects represented in the dataset. In that case, each 3D point cloud of the dataset may first be annotated with a localized representation for each object, but only with a class for the semantic class. Obtaining S10 may include obtaining S11 a 3D point cloud annotated with the localized representation and the semantic class for only the semantic object. Objects that are not annotated with a class among the semantic classes and / or do not belong to any of the semantic classes may not be labeled. Each 3D point cloud may include only a localized representation for each of these remaining objects (i.e., not annotated with a class among the semantic classes). For each remaining object, obtaining S10 may include calculating S13 values ​​of geometric descriptors for the remaining object. Obtaining S10 may then include using these values ​​of the geometric descriptors calculated for all remaining objects to define multiple geometry classes. In particular, the multiple geometry classes may partition the calculated values ​​of the geometric descriptors. For example, obtaining S10 may include clustering S14 the remaining objects according to the partitioning of the distribution of the calculated values. In that case, the multiple geometry classes may correspond to the resulting clusters. Each cluster may define a respective geometry class. Clustering S14 may be performed such that the number of objects in a cluster (i.e., the resulting geometry class) is equivalent to the number of objects in a semantic class, thereby ensuring a balanced dataset with no over- or under-represented classes.

[0049] A geometric descriptor of a given object may be defined by one or more geometric properties of the given object (or its localized representation). The localized representation of the object may be a bounding box encapsulating the object. Each representation is localized within a real scene, i.e., it includes the position coordinates of the object within the real scene. A geometric descriptor of a given object may include metrics of the localized representation of the given object (e.g., a bounding box encapsulating the object). The metrics may measure one or more geometric properties of the given object.

[0050] In an example, the geometric descriptor may be invariant with respect to orientation at least relative to a vertical axis. In other words, the geometric descriptor may be invariant with respect to the orientation of the object relative to a vertical axis. That means that the value of the geometric descriptor may be the same for two identical objects positioned at different orientations relative to the vertical axis (e.g., two tables rotated differently, or two chairs facing each other). For example, one or more geometric properties of a given object considered to calculate the geometric descriptor may be invariant with respect to orientation relative to the vertical axis (e.g., the geometric properties may be calculated based on the dimensions and / or volume of the object). Such a geometric descriptor improves classification of objects by not taking their orientation in a real scene into account when classifying them. In an example, the geometric descriptor may also be invariant with respect to other orientations, such as orientation relative to one or more horizontal axes.

[0051] In an example, a geometric descriptor of a given object may include one or more coordinates each associated with a respective geometric characteristic of the object (or its localized representation). For example, the geometric descriptor may include any one or any combination of the following coordinates: The geometric descriptor may include one or more coordinates (e.g., one for each length, width, and height of the bounding box) each representing a respective dimension of the localized representation (e.g., bounding box) of the given object. For example, the geometric descriptor may include a respective coordinate for each minimum and / or maximum dimension of the bounding box. Alternatively or additionally, the geometric descriptor may include coordinates representing the ratio between the minimum and maximum values ​​of a dimension of the bounding box of the given object. Alternatively or additionally, the geometric descriptor may include coordinates representing the area of ​​the bounding box of the given object. The area may be the area of ​​the base of the bounding box (i.e., the result of multiplying the length and width of the bounding box). Alternatively or additionally, the geometric descriptor may include coordinates that represent the volume of a bounding box of a given object, which may be the result of multiplying the length, width, and height of the bounding box.

[0052] In an example, the dataset may first include a 3D point cloud having objects annotated with classes from a set of other semantic classes. The other semantic classes may include objects that are underrepresented in the dataset compared to objects included in multiple semantic classes. For example, these objects may appear in less than a predetermined percentage of real scenes (e.g., less than 10% of real scenes). These objects belonging to the other semantic classes include the remaining objects to be clustered to determine the geometry class. In that case, obtaining S10 may include identifying S12 at least some (e.g., all) of the objects of the other semantic classes among all the objects. The identified objects may be the remaining objects considered to determine the geometry class. Then, obtaining S10 may include assigning S15 a geometry class to each identified object. Assigning S15 may be performed after determining the geometry class (e.g., based on steps S13 and S14). The assigning step S15 may include, for each identified object, calculating a geometric descriptor of the identified object (e.g., by calculating each coordinate of the geometric descriptor), determining a geometry class to which the identified object belongs based on the geometric descriptor (e.g., by determining which cluster it belongs to), and assigning the determined geometry class to the identified object. Training the function may be performed based on the assigned geometry classes.

[0053] In an example, identifying at least a portion of the objects of the other semantic class S12 may include filtering all objects of the other semantic class based on at least one geometric criterion. This allows maintaining a reasonable number of such objects of the other semantic class to preserve a balanced dataset by filtering the smallest, largest, and / or flattest objects. For example, the at least one geometric criterion includes a criterion based on a bounding box volume, a criterion based on a ratio between a minimum and a maximum bounding box dimension, and / or a criterion based on the result of multiplying the bounding box volume and the ratio.

[0054] With reference to Figures 2 to 15, examples of implementations of machine learning methods and methods of use will now be presented.

[0055] The method resides in the definition of classes for 3D objects defined by geometric criteria: in addition to well-represented and well-defined semantic classes in the annotations of the 3D scene understanding dataset, the method proposes to add geometrically defined classes (geometric classes) to supervise the learning of a function (hereinafter also called a model) to explain objects belonging to under-represented classes, and to do so, the method groups them by geometric descriptors rather than semantic ones.

[0056] Furthermore, only geometrically defined classes can be used for supervision, allowing general object detection without any semantic supervision, meaning that objects can be grouped by shape properties. In this setting, semantic labels can optionally be added to objects that belong to semantic classes that are well represented in the dataset.

[0057] Adding geometrically defined classes to supervise the training of neural networks in the field of 3D scene understanding has several advantages. First, publicly available datasets provide a limited number of semantic classes as annotations for training neural networks. This leads to training models that understand only the class of objects they are trained on and can therefore ignore a huge number of objects in reasoning about real-life scenes. Methods allow for taking these factors into account, if not by semantic class, then at least by geometry or shape. For example, in the case of 3D object detection, a model trained on both semantic and geometrically defined classes can detect more objects than those belonging to the semantic classes provided in the training dataset.

[0058] Furthermore, this allows the model to better understand the semantic classes (in addition to geometry-based ones) used to train it. In effect, the use of geometrically defined classes reduces ambiguity with respect to semantically defined ones. That is, in addition to being able to handle a greater quantity and variety of objects, a model trained with such classes also reaches better accuracy with respect to handling semantically defined objects.

[0059] The improvement in detecting objects labeled with semantic class labels when adding our geometrically defined classes is explained by taking into account more objects in the 3D scene, which are taken into account in the calculation of the loss during training. Geometric classes help the model better distinguish between semantic class objects by allowing it to reduce ambiguity about geometry class objects. Without these geometry classes, many objects present in the 3D scene would not actually be labeled, implicitly marking them as "background" objects that the model should learn not to recognize. Labeling objects with geometry classes aids the model's general understanding by explicitly considering more objects and more classes in the loss, helping it better distinguish each semantic class from others, including geometrically defined ones. Without geometry classes, this is done implicitly, resulting in poorer results.

[0060] A 3D scene understanding dataset consists of samples of 3D data, potentially available in various 3D formats (including RGBD frames, point clouds, meshes, voxel grids, etc.), and their annotations. Depending on the task for which the model is trained, the annotations considered may differ. For example, for semantic segmentation, each point is labeled with a semantic class. For 3D object detection, the annotation is in a list of bounding boxes defined by the semantic class, 3D position (x, y, z), size (w, l, h), and orientation (θ). The method involves creating a new type of annotation, namely, a geometrically defined class for 3D objects. This geometrically defined class may be used instead of the semantic class of an object when the semantic class is underrepresented or ambiguously defined in the dataset. A geometrically defined class is defined using only the geometric properties of a given 3D object (its shape, size, etc. of its bounding box).

[0061] Consider a dataset of 3D point cloud scenes used to train a 3D object detection model with 175 semantic object classes. Only 29 of these categories are deemed sufficiently well-represented in the dataset for training, excluding 146 classes of objects. Figure 2 shows the distribution of object occurrences for the 175 semantic classes. The semantic classes considered by the method are the 29 well-represented semantic classes (with the highest counts (see Figure 9)), while the other 146 remaining semantic classes are not considered (they have been previously referred to as "other semantic classes"). The method uses geometric classes for objects belonging to these 146 remaining classes. These geometric classes are hereinafter also referred to as "meta-classes."

[0062] FIG. 3 shows examples of objects 10 that belong to another semantic class, "Accessories," which is not well-defined. Samples from the "Accessories" class include small kitchen appliances, decorative objects, recycling bins, and bathrobes. A well-defined class is a class that has a clear semantic definition (e.g., the semantic class "Chair"). Conversely, the semantic class "Accessories" includes a variety of objects that are too broad for a clear semantic definition. The selection of a well-defined class can be done manually, for example, using qualitative criteria.

[0063] FIG. 3 also shows examples of objects 20 that belong to another semantic class, “bathtub,” that is not well represented. A well-represented category is also a category that has sufficient occurrences throughout the dataset. While the other semantic class, “bathtub,” is a well-defined semantic class, they do not appear in the dataset enough for the model to learn to identify them when the dataset lacks bathrooms. While the “bathtub” class is highly consistent, these objects appear only 80 times in the dataset of over 2000 examples of rooms. The method may include selecting for the dataset only classes that appear in more than 10% of the real scenes (e.g., rooms) in the dataset.

[0064] In this implementation example, the method uses only the bounding box of the object and ignores the geometry of the points within the bounding box. All bounding boxes, regardless of the initial semantic of the object, can be represented in a simplified 2D space using the following transformation:

[0065]

number

[0066] where (x,y,z) are the coordinates of the 3D position of the object's bounding box, (w,l,h) are its size, and θ is its orientation, as shown in Figure 4. The resulting vector in the simplified 2D space may be a geometric descriptor of the given object and may include a first coordinate that is the ratio between the minimum and maximum of the bounding box dimensions (w,l,h), and V is the volume of the bounding box.

[0067] The method then groups the objects by the shape of their bounding boxes, ranging from small, flat objects to large, cubic objects. The method maintains a reasonable number of such new objects by eliminating objects that are too small, too large, and too flat, thereby preserving a balanced dataset. To do so, the method includes filtering all objects of other semantic classes based on at least one geometric criterion. The at least one geometric criterion may be based on the ratio and volume described above. The at least one geometric criterion may be applied using the following formula:

[0068] min volume <V<max volume

[0069]

number

[0070] min criterion <criterion<max criterion however, ●V denotes the volume of the bounding box in cubic meters.

[0071] ●min volume is the minimum volume to accommodate a bounding box in the metaclass.

[0072]

number

[0073] and the appropriate value is min volume ∈[0,1].

[0074] ●max volume is the maximum volume for accommodating a bounding box in the metaclass.

[0075]

number

[0076] and the appropriate value is max volume ∈[3,5].

[0077] l denotes the length of the bounding box in meters.

[0078] ●w indicates the width of the bounding box in meters.

[0079] ●h indicates the height of the bounding box in meters.

[0080] ●min ratio is the minimum ratio min / max for accepting a bounding box in the metaclass. ratio ∈]0,1] and the appropriate value is min ratio ∈]0,0.3].

[0081] ●max ratio is the maximum ratio min / max for accepting bounding boxes in the metaclass. ratio ∈]0,1] and the appropriate value is min ratio ∈]0.7,1].

[0082] ●Criterion is,

[0083]

number

[0084] is the geometric criterion used for further filtering defined by: Other geometric criteria may be defined and used (as will be detailed later).

[0085] ●min criterion is the minimum criterion for accepting a bounding box in a metaclass.

[0086]

number

[0087] and the appropriate value is min criterion ∈]0,0.4].

[0088] ●max criterion is the maximum criterion for accepting a bounding box in a metaclass.

[0089]

number

[0090] and the appropriate value is max criterion ∈[1,2].

[0091] In this example implementation, the method calculates the following values: ●min volume =0 ●max volume =3 ●min ratio =0.3 ●max ratio =1 ●

[0092]

number

[0093] ●min criterion =0.2m 3 ●max criterion =1.5m 3 can be used.

[0094] The method may then include filtering the objects by sorting them by criterion value and keeping the objects between given limits. The method may include determining these bounds such that the number of objects in each metaclass is equivalent to the number of objects in the semantic class, thereby ensuring a balanced dataset with no over- or under-represented classes.

[0095]

number

[0096] Here is an example of bounding boxes sorted by:

[0097] The method then includes determining geometric classes. The method may include clustering the remaining bounding boxes (after filtering) to create n clusters of objects, each cluster forming a meta-class. The method may use any unsupervised clustering method. Figure 6 shows an example of a clustering method that may be used for clustering using n-1 lines. The clustering method is based on the following formula:

[0098]

number

[0099] This may include defining the slope of the i-th line using x, where i is an integer and i∈[1,n-1]. In the diagram, x represents the slope of the slope. The intercept theorem (Thales) provides the following formula:

[0100]

number

[0101] 7 shows three clusters of objects 101, 102, and 103 resulting from dividing into three clusters (and therefore three meta-classes) using a line as a separator. In other examples, the method may use any one of any combination of other algorithms or heuristics for clustering, such as, for example, K-means, Gaussian mixture models, or other unsupervised learning methods.

[0102] As mentioned above, other methods can be used to determine such a segmentation. For example, using the point clouds of each object, the method can use one of the following two methods:

[0103] A first method that involves computing, for each point cloud, an embedding by a point cloud encoder such as PointNet++ (e.g., as described in the paper by [Non-Patent Document 3], incorporated herein by reference), and using a clustering method on the obtained embeddings, e.g., K-means with k meta-classes, thus obtaining the final partitioning.

[0104] A first method that includes calculating, for each point cloud, an embedding based on geometric features (manufactured or not), such as one or a combination of those detailed in the paper by (Non-Patent Document 4), which is incorporated herein by reference, and then determining a division using any clustering method.

[0105] FIG. 8 illustrates an example of the method used. In this example, the method is included in a process for the exemplary task of 3D object detection in a point cloud using the three metaclasses previously defined. The method includes acquiring a 3D point cloud 110 (S30). Acquiring the 3D point cloud 110 (S30) can be performed by virtually scanning a real scene (e.g., based on a virtual representation of the real scene). The method then includes applying a function to the acquired 3D point cloud (S31). The function outputs localized representations (e.g., bounding boxes 111, 112, 113, 114, 115) for detected objects in the real scene. FIG. 9 illustrates the 29 retained semantic classes (e.g., "plant," "wardrobe," "curtain," etc.) and the three created geometric classes ("meta_0," "meta_1," and "meta_2") of the predetermined set of classes. Objects for which no class (having a semantic class) is represented are detected as metaclasses (geometric classes). In this example, they detect a bathtub 113, a refrigerator 114, and a range hood 115. These objects 113, 114, and 115 do not belong to a semantic class and have been detected based on a defined geometric class. These objects 113, 114, and 115 do not belong to a semantic class and have been detected based on a defined geometric class. This example shows that the method can detect objects that do not belong to a semantic class (i.e., are under-represented, such as a bathtub, a toilet, a household appliance, or an electronic device).

[0106] The introduction of geometric classes also improves the accuracy of the model with respect to semantic classes, as can be seen in Table 1 below: For each of the metrics considered, the function more accurately predicts objects belonging to semantic classes with geometric classes than those without.

[0107] [Table 1]

[0108] Common metrics used in 3D object detection are mean average precision (mAP) and mean average recall (mAR). These metrics are defined using an intersection-over-union (IoU) threshold to evaluate the model's predictions. For 3D bounding boxes, these union and intersection values ​​are calculated as volumes; for clarity, the 2D equivalents are visualized as areas on Figure 10.

[0109] For a given class c, a predicted bounding box may be a correct prediction if it can be paired with a ground truth bounding box such that the IoU between the two bounding boxes is higher than a given threshold. The pairing is unique in the sense that a given ground truth can be assigned to at most one prediction, and vice versa. Pairings are performed in descending IoU order until either all ground truth or predicted bounding boxes are exhausted.

[0110] The following metrics may be used:

[0111] True Positive (TP): Correct model prediction, i.e., a prediction with matching bounding boxes of the same class such that IoU > threshold. False positive (FP): An incorrect model prediction, i.e., the model predicted a bounding box but the corresponding ground truth annotation was not matched, or IoU <= threshold.

[0112] False Negatives (FN): false negatives. An annotation that does not match any prediction (i.e., the object is present but not detected by the model).

[0113] ●Precision provides the percentage of positive predictions that are actually correct.

[0114]

number

[0115] ● Recall provides the proportion of actual positive cases that the model is able to correctly identify.

[0116]

number

[0117] Precision-recall curves: The tradeoff between the quality and quantity of predicted bounding boxes can be controlled after inference, typically in the form of a confidence score that the model associates with each class and bounding box. This score may be thresholded to remove predictions that the model is not confident about, allowing us to plot graphs of precision and recall as the confidence score threshold is varied.

[0118] Average Precision (AP) is the area under the precision-recall curve per class (see the precision-recall curves obtained for the three classes shown in Figure 11).

[0119] The mean average precision (mAP) is the average AP across all classes. We typically denote it as mAP@threshold to specify the IoU threshold (i.e., at what quality level) at which we compute the metric.

[0120] Average Recall (AR) is the recall averaged over a range of IoU thresholds (0.5 to 1.0) and can be approximated as shown below, where o is the IoU and Recall(o) is the Recall-IoU curve with Recall on the y-axis and varying IoU on the x-axis.

[0121] • Mean mean recall (mAR) is the average of AR across all classes.

[0122]

number

[0123] Next, the results of the method are described.

[0124] Table 2 below shows the benefits of filtering objects of other semantic classes.

[0125] [Table 2]

[0126] The metrics are measured only on semantic classes. Table 2 shows that filtering gives more robust results. In fact, the results are typically better than the baseline. For reference, mAP (mean average precision) relates to the number of correct predictions among all model predictions, while mAR (mean average recall) relates to the number of correctly detected objects among all ground truth objects.

[0127] Table 3 below shows the predictive accuracy of the function for different numbers of geometry classes.

[0128] [Table 3]

[0129] The metrics are measured on semantic classes only. Results are obtained with a filtering step. The results show better predictions for any number of geometry classes (1, 3, or 5) than the baseline (i.e., no geometry classes).

[0130] Table 4 below shows the predictive accuracy of the functions for different geometric descriptors.

[0131] [Table 4]

[0132] The metrics are measured only on semantic classes, with filtering and three metaclasses when applicable. Results show that the prediction accuracy is typically better for all geometric descriptors than the no-class baseline (i.e., no geometric classes). The better performance of the geometric descriptor (w*l,h) demonstrates the benefit of invariance at least around the vertical axis. In practice, in indoor scenes, most devices rotate only around the z-axis (vertical axis) rather than lying down.

[0133] Table 5 below shows another example where a smaller number of semantic classes are considered.

[0134] [Table 5]

[0135] In this other example, only six semantic classes are considered: chair, sofa, desk, cabinet, table, and bed. Three geometric classes are defined. The metrics are measured only on the semantic classes. The table shows that even in this case, the function results are more accurate with the three geometric classes than the baseline (i.e., no geometric classes).

[0136] Table 6 below provides the results obtained for semantic and geometric classes. The results show that detection from meta classes is useful since they are better detected compared to semantic classes (e.g., "vase"). The metric "AP@0.25" for meta 0 is particularly high at 0.49.

[0137] [Table 6]

[0138] Table 7 below shows the usefulness of the geometric classes.

[0139] [Table 7]

[0140] The results in Table 7 show that, similar to semantic classes, training a model on a subset of filtered and segmented objects improves overall detection.

[0141] 12 shows an example of 3D scene understanding. In this example, the method used is included in a 3D scene understanding process that consists in reconstructing a 3D scene from a 3D point cloud. This process includes executing the method used, in particular, step S30 of obtaining a 3D point cloud of the real scene and step S31 of applying a function to the obtained 3D point cloud. The function outputs localized representations (in this example, bounding boxes) that represent the envelope of objects in the real scene. For each localized representation, the function also outputs the class of the object that the localized representation represents in terms of semantics and geometry classes.

[0142] The process also includes reconstructing walls and floors of the real scene based on the 3D point clouds (S32). Reconstructing walls and floors (S32) may be performed after or in parallel with applying the function (S31). After applying the function, the process then includes, for each localized representation, retrieving a 3D representation (e.g., a computer-aided design (CAD) model) of the object that the localized representation represents (S33). The retrieving (S33) may be based on the class of the localized representations output for the object. The process then includes assembling the retrieved 3D representations within the 3D scene, thereby reconstructing a 3D scene of the 3D point clouds (S34). The assembling (S34) may be performed according to the positions of the localized representations output by the function (each 3D representation representing a respective real object in the real scene).

[0143] FIG. 13 shows an example of a function architecture. The function has an architecture including a voxelization layer 201 configured to voxelize a 3D point cloud taken as input. The output of the voxelization layer 201 can be a sparse occupancy grid. The function also has a convolutional neural network 202 that takes as input the 3D point cloud voxelized by the voxelization layer. The convolutional neural network outputs a localized representation and class of each object. The convolutional neural network can have a backbone including a sparse 3D ResNet CNN with CBAM blocks, a neck including pruning and decoder layers, and a head including regress bounding box parameters, classification, Möbius strip parameterization (encoding oriented bounding box invariants), and CBAM blocks as refinements.

[0144] For each ground truth bounding box, check if this bounding box is at least N locThe last feature (grid) level that covers output positions is selected. The positions are filtered using center sampling, and only points close to the center of the box are kept as positive matches (using centerness).

[0145] The loss may be defined using the following formula:

[0146]

number

[0147] However, N pos is the number of matched positions, and L cls is the focal loss for classification, and L leg is the IoU between boxes, the regression loss, and L cntr is the binary cross entropy for centerness.

[0148] The method may use a parameterization to be invariant to the front faces of the bounding boxes. With the original parameterization, four different bounding boxes represent the same object. By showing that q=w / l, the following parameterization defines the same bounding box as follows:

[0149]

number

[0150] In fact, these four parameters also represent the same point on a Möbius strip, so the method may use a reparameterization of the bounding box using a classical Möbius strip embedding.

[0151]

number

[0152] In this case, the bounding box (x,y,z,w,l,h,θ) is

[0153]

number

[0154] can be reparameterized (δ1,δ2,δ3,δ4,δ5,δ6,δ7,δ8) using

[0155] Figure 14 shows example results for trained functions. In particular, the figure shows the results obtained for two functions 310, 320 trained using the present method compared to the ground truth 300. The results are presented using the legend of Figure 9. The results show that the use of geometry classes allows for more objects to be detected and for more accurate detection of objects in semantic classes.

[0156] FIG. 15 shows an example of a system where the system is a client computer system, eg, a user's workstation.

[0157] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and a random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass memory devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the foregoing may be supplemented by or incorporated in specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. In addition, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the buttons on the mouse are used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0158] A computer program may include computer-executable instructions, including means for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, for example, an article tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions to perform the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or assembly or machine language, as appropriate. In either case, the language may be a compiled or interpreted language. The program may also be a complete installation program or update program. Application of the program on the system, in either case, results in instructions for performing the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment, the server communicating with one or more clients over a network, in which case the processing unit executes instructions contained in the program, thereby performing the method on the cloud computing environment.

Claims

1. 1. A computer-implemented method for machine learning a function configured to take as input a 3D point cloud of a real scene and to output a localized representation of each respective object of the real scene, and for each respective object, a class of the respective object among a predetermined set of classes, the method comprising: - obtaining a dataset of 3D point clouds annotated with, for each 3D point cloud, a localized representation of each of the respective objects and, for each respective object, a class of said respective object from said predetermined set of classes; training said function on the basis of said acquired data set; Equipped with The predetermined set of classes includes a plurality of semantic classes and a plurality of geometry classes. A method characterized by:

2. The obtaining of the data set includes: － For each 3D point cloud, the localized representation of each of the respective objects, and For each respective object of at least some of said objects, the class of said respective object within said semantic class obtaining the 3D point cloud annotated by - calculating, for each remaining object of said 3D point cloud, a value of a geometric descriptor; clustering the remaining objects according to a partitioning of the distribution of the calculated values, the plurality of geometry classes corresponding to the resulting clusters; 2. The method of claim 1, comprising:

3. 3. The method of claim 2, wherein the geometric descriptor is invariant with respect to orientation at least relative to a vertical axis.

4. 4. The method of claim 3, wherein the geometric descriptor for a given object comprises metrics of a bounding box of the given object.

5. The geometric descriptor for a given object is: - one or more coordinates each representing a respective dimension of the bounding box of the given object; - a coordinate representing the ratio between the minimum and maximum of the dimensions of the bounding box of the given object, coordinates representing the area of ​​the bounding box of the given object, and / or - coordinates representing the volume of the bounding box of the given object 5. The method according to claim 3 or 4, comprising:

6. The obtaining of the data set includes: － For each 3D point cloud, said localized representation of each respective object, for each respective object of said at least some of said objects, the class of said respective object within said semantics class; and For each remaining object, the class of said remaining object in a set of other semantic classes obtaining the 3D point cloud annotated by - identifying at least some of said objects of said other semantic classes from among all said objects; - assigning a geometry class to each identified object; 6. The method according to claim 1, further comprising:

7. 7. The method of claim 6, wherein the identifying of the at least some of the objects of the other semantic class comprises filtering all the objects of the other semantic class based on at least one geometric criterion.

8. The at least one geometric criterion is: - Criteria based on the volume of the bounding box, a criterion based on the ratio between the minimum and maximum bounding box dimensions, and / or a criterion based on the result of multiplying the volume of the bounding box by the ratio 8. The method of claim 7, comprising:

9. 9. The method of any one of claims 1 to 8, wherein the function is configured to perform 3D object detection or 3D object segmentation.

10. 10. The method according to any one of claims 1 to 9, wherein the 3D point cloud of the dataset is a 3D point cloud of an indoor scene and / or is acquired via scanning.

11. 11. The method according to any one of claims 1 to 10, wherein the function has an architecture comprising a voxelization layer configured to voxelize the 3D point cloud taken as input, and / or a convolutional neural network that takes as input the 3D point cloud voxelized by the voxelization layer.

12. 12. A computer-implemented method for using a machine-learned function according to the method of any one of claims 1 to 11, comprising: Optionally, acquiring a 3D point cloud by scanning a real scene; applying said function to said acquired 3D point cloud; A method comprising:

13. A computer program comprising instructions that, when the program is executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 11 and / or the method according to claim 12.

14. A computer-readable recording medium having the computer program according to claim 13 recorded thereon.

15. 14. A system comprising a processor coupled to a memory, the memory having the computer program of claim 13 stored thereon.

Images