3D scene understanding

Abstract

In particular, the present disclosure relates to a computer-implemented method for a machine learning function configured to take a 3D point cloud of a real scene as input and output each localized characterization of a respective object of the real scene, and for each respective object, output a category of the respective object in a predetermined category set. The method includes obtaining a data set of annotated 3D point clouds, where each 3D point cloud is annotated with each localized characterization of a respective object, each respective object is annotated with a category of the respective object in a predetermined set of categories. The method includes training the function based on the obtained data set. The predetermined category set includes a plurality of semantic categories and a plurality of geometric categories. The method provides an optimization scheme for the understanding of the 3D scene.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer programs and systems, and more specifically to a method, system and program for 3D scene understanding. BACKGROUND

[0002] The most advanced current methods for 3D scene understanding are based on machine learning functions trained in a supervised manner on point cloud datasets of 3D scenes. These functions can then detect objects in point clouds, e.g. objects obtained by scanning real scenes. To train these functions, point clouds are annotated with a localized representation of objects in the 3D scene and semantic annotations, such as semantic classes of objects. However, to ensure that each semantic class contains a sufficient amount of samples within the dataset and that these samples represent the semantic class they belong to, the number of semantic classes used to train the machine learning functions is very limited (e.g. between 10 and 20 classes). This results in the training of the functions only handling objects belonging to the limited number of semantic classes mentioned above, while ignoring a large amount of remaining objects. This often hinders the automation of 3D scene understanding tasks.

[0003] Self-supervised techniques aim to overcome the lack of annotated samples in the dataset by proposing a pre-training of the functions in pretext tasks. The pre-training is supervised using data augmentation and masking techniques, the functions learning intermediate representations of the data and understanding its underlying structure. The functions are then fine-tuned in a supervised manner on the annotated dataset, i.e. further trained for a specific downstream task (e.g. object detection). These methods achieve better results on the given semantic classes from the annotated dataset used for fine-tuning, but do not solve the problem of objects from unconsidered semantic classes, which are still ignored.

[0004] Open Vocabulary methods leverage functions trained on other modalities such as images and text, whose datasets are denser and richer than 3D datasets. This allows to extend the variety of classes handled by these functions. However, such methods heavily rely on large datasets of image-text pairs, which are often scraped from the internet and are not always free of copyright restrictions (recent lawsuits about this are still under investigation). Moreover, for the same number of classes of annotated 3D datasets, these methods have been proven to be less efficient than fully supervised methods, while often requiring significantly more computational power. Furthermore, when used to detect semantic classes that are loosely or ambiguously defined (e.g. "objects", or even "small objects"), they still require a range of classes to be detected and show poor performance.

[0005] In this context, there is still a need for an improved solution for 3D scene understanding. SUMMARY

[0006] Thus, there is provided a computer-implemented method for a machine learning function configured to take as input a 3D point cloud of a real scene and to output, for each localized representation of a respective object of the real scene, and for each respective object, a class of the respective object in a predetermined set of classes. The method is hereinafter referred to as the machine learning method, or simply the method. The method comprises obtaining a dataset of annotated 3D point clouds, wherein each 3D point cloud is annotated with each localized representation of a respective object, and each respective object is annotated with a class of the respective object in a predetermined set of classes. The method comprises training the function based on the obtained dataset. The predetermined set of classes comprises a plurality of semantic classes and a plurality of geometric classes.

[0007] The machine learning method can comprise one or more of:

[0008] - obtaining the dataset comprises:

[0009] o obtaining 3D point clouds annotated with:

[0010] ■for each 3D point cloud, each localized representation of a respective object, and

[0011] ■for each respective object in at least a portion of the objects, a class of the respective object in the semantic classes;

[0012] o computing a value of a geometric descriptor for each remaining object of the 3D point cloud; and

[0013] o clustering the remaining objects according to a partitioning of a distribution of the computed values, wherein the plurality of geometric classes corresponds to resulting clusters;

[0014] - the remaining objects are objects other than the at least a portion of the objects;

[0015] - the geometric descriptor has at least directional invariance with respect to a vertical axis;

[0016] - the geometric descriptor of a given object comprises a measure of a bounding box of the given object;

[0017] - the geometric descriptor of a given object comprises:

[0018] o one or more coordinates, each coordinate representing a respective dimension of a bounding box of the given object;

[0019] o a coordinate representing a ratio between a minimum value and a maximum value of a dimension of the bounding box of the given object;

[0020] o a coordinate representing an area of the bounding box of the given object; and / or

[0021] o coordinates representing the volume of the bounding box of the given object;

[0022] - obtaining the dataset comprises:

[0023] o obtaining a 3D point cloud annotated with:

[0024] ■ for each 3D point cloud, each localization representation of a respective object,

[0025] ■ for each respective object among at least a portion of the objects, a class of the respective object among a semantic class, and

[0026] ■ for each remaining object, a class of the remaining object among a set of other semantic classes;

[0027] o identifying at least a portion of objects of other semantic classes among all the objects; and

[0028] o assigning a geometric class to each identified object;

[0029] - identifying at least a portion of objects of other semantic classes comprises: filtering all objects of other semantic classes based on at least one geometric criterion;

[0030] - the at least one geometric criterion comprises:

[0031] o a criterion based on the volume of the bounding box;

[0032] o a criterion based on a ratio between a minimum value and a maximum value of the dimensions of the bounding box; and / or

[0033] o a criterion based on a result of a multiplication of the volume of the bounding box and the ratio;

[0034] - the function is configured to perform 3D object detection or 3D object segmentation;

[0035] - the 3D point clouds of the dataset are 3D point clouds of indoor scenes and / or are obtained by scanning; and / or

[0036] - the function has an architecture comprising a voxelization layer configured to voxelize a 3D point cloud as input and / or a convolutional neural network taking as input the 3D point cloud voxelized by the voxelization layer.

[0037] A method of using the function after machine learning according to the machine learning method described above is also provided. The method is referred to below as the method of use. The method of use comprises obtaining a 3D point cloud, optionally by scanning a real scene. The method of use comprises applying the function to the obtained 3D point cloud.

[0038] A computer program is also provided, comprising instructions which, when the program is executed by a computer, cause the computer to carry out the machine learning method and / or the use method described above.

[0039] A computer-readable storage medium is also provided, having recorded thereon the computer program described above.

[0040] A system is also provided, comprising a processor coupled to a memory and a graphical user interface, the memory having recorded thereon the computer program described above.

[0041] A device is also provided, comprising a data storage medium having recorded thereon the computer program described above.

[0042] The device can form or be used as a non-transitory computer-readable medium, for example on a SaaS (Software as a service) or other server, cloud-based platform, etc. Alternatively, the device can comprise a processor coupled to the data storage medium. The device can thus form, in whole or in part, a computer system (for example, the device is a subsystem of the entire system). The system can further comprise a graphical user interface coupled to the processor. BRIEF DESCRIPTION OF DRAWINGS

[0043] Non-limiting examples will now be described with reference to the accompanying drawings, in which:

[0044] - Figure 1 A flowchart illustrating an example of the method is shown;

[0045] - Figure 2 A distribution of the number of occurrences of objects belonging to all semantic categories is shown;

[0046] - Figure 3 Examples of objects belonging to semantic categories that are not well represented or not well defined are shown;

[0047] - Figure 4 Examples of geometric descriptors are shown;

[0048] - Figure 5 Examples of ranking objects based on geometric criteria are shown;

[0049] - Figure 6 And Figure 7 Examples of clustering the remaining objects are shown;

[0050] - Figure 8 And Figure 9 Examples of the use method are shown;

[0051] - Figure 10 And 11 Metrics for measuring the performance of the trained function are shown;

[0052] - Figure 12 An example of 3D scene understanding is shown;

[0053] - Figure 13 An example of the architecture of the function is shown;

[0054] - Figure 14 An example of the results of the trained function is shown; and

[0055] - Figure 15 An example of the system is shown. DETAILED DESCRIPTION

[0056] With reference to the flowchart of Figure 1 A computer-implemented method for a machine learning function configured to take as input a 3D point cloud of a real scene and to output, for each localized representation of a respective object of the real scene, a class of the respective object among a predetermined set of classes is proposed. The method is hereinafter referred to as machine learning method, or simply the method. The method comprises obtaining a dataset of annotated 3D point clouds, wherein each 3D point cloud is annotated with each localized representation of a respective object, each respective object being annotated with a class of the respective object among a predetermined set of classes. The method comprises training the function based on the obtained dataset. The predetermined set of classes comprises a plurality of semantic classes and a plurality of geometric classes.

[0057] This method forms an improved solution for 3D scene understanding.

[0058] It is worth noting that this method improves the training of machine learning functions in the field of 3D scene understanding. In particular, it makes it possible to overcome the problem of lack of annotations and certain semantic class under-representation in datasets used to train machine learning models in the field of 3D scene understanding. Indeed, the addition of geometric classes to supervise the training of neural networks in the field of 3D scene understanding has several advantages.

[0059] First, publicly available datasets provide a limited number of semantic classes as annotations to train machine learning models. This results in models that, during their training, can only understand the classes of the objects for which they are trained, thus ignoring a large number of objects in the inference of real scenes. The present method allows these elements to be taken into account, if not through semantic classes, at least through geometry or shape. Since the model is trained on both semantic and geometric classes, it will be able to detect more objects than those belonging to the semantic classes given in the training dataset.

[0060] Furthermore, this also enables the model to better understand the semantic categories used for training (in addition to the geometric categories). Indeed, using geometrically defined categories reduces the ambiguity of semantically defined categories. That is, in addition to being able to handle a large variety of objects, the model trained with these categories also achieves better performance in handling semantically defined objects.

[0061] Furthermore, the method allows for a more accurate 3D scene understanding. Indeed, as the training of the function is more relevant, the machine-learned function is able to detect more objects with better performance. Thus, the method allows training a function to perform more accurate and relevant 3D object detection and / or 3D object segmentation based on a scan of a real scene. The method allows training a function to reconstruct a 3D representation closer to the real scene and / or to more accurately register 3D point clouds.

[0062] The machine-learning method and / or the method of use can be computer-implemented. This means that the steps (or substantially all the steps) of the machine-learning method and / or the method of use are performed by at least one computer or any similar system. Thus, the steps of the machine-learning method and / or the method of use are performed by a computer, possibly fully automatically, possibly semi-automatically. In examples, the triggering of at least some steps of the machine-learning method and / or the method of use can be done through a user-computer interaction. The level of user-computer interaction required can depend on the level of automation intended and be balanced with the need to implement the user’s wishes. In examples, this level can be user-defined and / or pre-defined.

[0063] A typical example of computer implementation of the machine-learning method and / or the method of use is the use of a system adapted for this purpose to perform the machine-learning method and / or the method of use. This system can comprise a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program comprising instructions for performing the machine-learning method and / or the method of use. The memory can also store a database. The memory is any hardware adapted for such storage, possibly comprising several physically distinct parts (for example, one for the program, possibly one for the database).

[0064] A "database" refers to any collection of data (i.e., information) organized for search and retrieval (e.g., a relational database, e.g., based on a predetermined structured language, such as SQL). When stored in memory, a database allows a computer to quickly search and retrieve. The structure of a database does facilitate storing, retrieving, modifying, and deleting data, as well as various data processing operations. A database can be composed of a file or collection of files, which can be broken down into records, each record consisting of one or more fields. A field is the basic unit of data storage. Users can retrieve data primarily through queries. Using keywords and ordering commands, users can quickly search, rearrange, group, and select fields in many records to retrieve or create reports of specific data aggregations, according to the rules of the database management system being used.

[0065] The method of use can include in a process that can include, after applying the function to the obtained 3D point cloud, performing 3D scene understanding of the real scene based on the localized representations and categories of objects output by the function. The 3D scene understanding can be a process, possibly in real-time, that includes perceiving, analyzing, and / or articulating an interpretation of a real scene, such as a real scene observed through a network of sensors (e.g., the 3D point cloud input to the function). For example, the 3D scene understanding can include performing 3D object detection, 3D object segmentation, 3D scene reconstruction, 3D point cloud registration, and / or 3D mesh transformation based on the localized representations and categories of objects output by the function. As the method improves the output of localized representations and categories of objects, the method also improves the performance of 3D scene understanding of the real scene.

[0066] 3D scene understanding can be a process that is based on 3D input data. For example, the 3D input data can be a 3D point cloud (e.g., acquired by scanning a real scene) that is input to this functionality. For example, 3D scene understanding can involve 3D detection, such as a Fully Convolutional Anchor-Free 3D Object Detection (FCAF3D) method, or be a process described in Hou, Ji, et al., “Exploring Data-Efficient 3D Scene Understanding With Contrastive Scene Contexts,” in 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition, (https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Hou_Exploring_Data-Efficient_3D_Scene_Understanding_With_Contrastive_Scene_Contexts_CVPR_2021_paper.pdf), which is incorporated by reference herein. As explained in this document, 3D scene understanding can include collecting and annotating real 3D scenes.

[0067] Alternatively or additionally, still in the case of a process that is based on 3D input data, 3D scene understanding can include performing any one or any combination of the applications described in the resource from Google researchers Alireza Fathi and Rui Huang on 3D Scene Understanding with TensorFlow 3D (https: / / ai.googleblog.com / 2021 / 02 / 3d-scene-understanding-with-tensorflow.html), which is incorporated by reference herein.

[0068] Alternatively or additionally, performing 3D scene understanding can comprise performing 3D object detection. For example, each localized representation output by the function can represent an envelope of a respective real-world object within the real scene (e.g., the localized representation can comprise a 3D surface representing a real face of the respective real-world object). For example, when the 3D point cloud is acquired by scanning the real scene, each localized representation can enclose points that are sampled on the same real-world object within the real scene (i.e., on the surface of the real-world object). The parameters of each localized representation can comprise a semantic or geometric class, a center position, a size, and / or an orientation. Performing 3D scene understanding can comprise, for each output localized representation, determining the object represented by the localized representation, e.g., based on the class and / or shape of the envelope represented by the localized representation. Alternatively, the function can also be trained to directly output, for each localized representation, the object represented by the localized representation.

[0069] Performing 3D scene understanding can then comprise performing 3D scene reconstruction. The 3D scene reconstruction can comprise determining a 3D representation of the real scene based on the output of the function. For example, the reconstruction of the 3D representation can comprise retrieving 3D representations (e.g., CAD models) of the objects of the real scene (e.g., from a database storing computer-aided design (CAD) models of objects) and assembling these 3D representations in a 3D scene according to the localized representations output by the function (each 3D representation representing a respective real object of the real scene). The assembled 3D objects can be the 3D objects determined for each localized representation and can be assembled according to the respective positions and orientations of the output localized representations. The 3D scene reconstruction can also comprise reconstructing walls and floors of the real scene based on the 3D point cloud and adding the reconstructed walls and floors to the 3D representation of the real scene.

[0070] Alternatively or additionally, performing 3D scene understanding can comprise performing 3D object segmentation of the 3D point cloud. The segmentation can comprise annotating the points (e.g., all points) of the 3D point cloud according to the output localized representations and / or the output object class. The annotation of the points can comprise assigning to each point a respective label according to the output localized representation and / or the output class of the object to which the point belongs. The points assigned the same label (i.e., the same segment of points) can correspond to points that have been sampled on the same real-world object within the real scene (i.e., on the surface of the real-world object). In this case, the function can output the label to be assigned to each point of the 3D point cloud. In addition to the localized representations, the function can be trained to output a group of points belonging to each localized representation. Each group of points can gather points assigned the same label, e.g., points that have been sampled on the same real-world object within the real scene (i.e., on the surface of the real-world object).

[0071] Alternatively or additionally, performing 3D scene understanding may include performing a 3D mesh transformation. The 3D mesh transformation may include transforming a 3D point cloud into a mesh representing a real scene. For each object, the mesh of the 3D point cloud may include a corresponding mesh representing the object (e.g., including a surface representing the object). In this case, each output localized representation may include a mesh of the object represented by the localized representation, and the transformation may include assembling the mesh based on the position and / or orientation of the output localized representation. Alternatively, the transformation may include: for each output localized representation, creating a mesh for each object (e.g., based on parameters and / or semantic or geometric categories of the localized representation), and assembling all created meshes.

[0072] Alternatively or additionally, performing 3D scene understanding may include registering 3D point clouds. Registration of the 3D point cloud may include aligning one or more parts of the 3D point cloud with a reference point cloud (e.g., corresponding to a scan of the object). For each part, registration may include determining a reference point cloud for the part and aligning the part with the determined reference point cloud. Alignment of each part may include determining rotation and / or scaling parameters for aligning the part with the determined reference point cloud. In an example, the functionality may be configured to also output one or more parts, a reference point cloud for each part, and / or an alignment of each part (i.e., rotation and / or scaling parameters for alignment).

[0073] In an example, the process may also include executing a machine learning method before executing the method of use. In other words, the process may first include a machine learning method for training the function, second, a method of use for applying the trained function, and third, performing 3D scene understanding based on the output of the function. Alternatively, the training of the function may be performed before the process. In this case, the machine learning method may be executed before executing the process, such as in an offline phase, and then the process may be executed based on the trained function, such as in an online phase.

[0074] After the functional training, the use method comprises obtaining a 3D point cloud. The obtaining of the 3D point cloud can be performed by scanning a real scene, e.g. based on one or more sensors. Alternatively, the 3D point cloud can have been computed (e.g. by scanning a real or virtual scene) and recorded in a database. In this case, the obtaining can comprise retrieving the 3D point cloud from the dataset. After the obtaining of the 3D point cloud, the use method comprises applying the trained neural network to the obtained point cloud, thereby performing a 3D scene understanding. For example, the function can output each localization representation of a respective object of the real scene, and for each respective object, a class of the respective object in a predetermined set of classes, and the use method can comprise performing a 3D scene understanding (e.g. 3D object detection, 3D object segmentation, 3D scene reconstruction, 3D point cloud registration and / or 3D mesh transformation) based on the localization representations and classes of the objects output by the function. Alternatively, the function can be trained to directly output the results of the 3D scene understanding (i.e. detected 3D objects, segments of the 3D point cloud, reconstructed 3D scene, registered 3D point cloud and / or resulting 3D point cloud mesh).

[0075] The obtaining S10 of the dataset is now discussed in more details.

[0076] The dataset comprises 3D point clouds, each 3D point cloud being for a respective real scene comprising objects (e.g. acquired by scanning the respective real scene). Each 3D point cloud of the dataset is annotated with localization representations, each localization representation being representative of a respective object of the respective real scene. For example, the localization representations can together form a virtual representation of the real scene (e.g. the virtual representation can be recorded on a file), the dataset can comprise a link between the 3D point clouds and the virtual representation (e.g. with the file comprising the localization representations). For each object of the real scene, the virtual representation can further comprise a class of the object. Alternatively or additionally, each point of the 3D point cloud can be labeled with the localization representation and / or the class of the object it belongs to. The labeling of each point can be included in the 3D point cloud (e.g. in addition to the position parameters of each point, a variable). The 3D point cloud as input of the function is not annotated. The function is trained for annotating the 3D point cloud, i.e. computing the localization representations and the classes of the objects present in the real scene from which the 3D point cloud was acquired.

[0077] The 3D point clouds of the data sets can be acquired from respective real scenes (or virtual representations of these respective real scenes), which can be indoor scenes, e.g., offices, apartments, and / or stores. The real scenes of the data sets comprising the 3D point clouds can all be different. The real scenes can correspond to the same class of environments (e.g., offices, apartments, and / or stores). Each real scene can include objects that naturally occur in this class (e.g., desks and chairs for an office), and these objects can be positioned relative to each other. Each real scene can also include a (e.g., planar) floor and / or walls that define the real scene and / or one or more rooms included by the real scene. Each real scene can be an environment that exists in the real world, or it can be fictional (but can also exist in the real world). For example, the method can include generating one or more (e.g., all) of the real scenes (e.g., randomly and / or based on real-world environments).

[0078] Each real scene can include one or more (i.e., real) objects. Some (e.g., all) of the real scenes of the data sets can be complex rooms, i.e., each room can include several objects (e.g., more than ten or one hundred objects) and / or the shape of each room can not be rectangular (e.g., have more than four walls and / or include one or more circular walls). Each of the localized representations that annotate the 3D point clouds can represent the geometry of a real object that is positioned (or to be positioned) in the real scene. The real objects can be manufactured in the real world after their virtual design is completed.

[0079] Each 3D point cloud of the dataset is also annotated with, for each respective object in the real scene, a class of the respective object from a predetermined set of classes. The predetermined set of classes includes a plurality of semantic classes. For example, the plurality of semantic classes can include one or more furniture object classes and / or one or more decorative object classes, or any other combination of object class types. Any furniture object herein can have a furniture function in the real scene in which they are placed. For example, each real scene of the dataset can include one or more chairs, one or more lamps, one or more cabinets, one or more shelves, one or more sofas, one or more tables, one or more beds, one or more dish cabinets, one or more nightstands, one or more desks, and / or one or more wardrobes. The predetermined set of classes can include a respective class for each of these furniture objects. Any decorative object herein can have a decorative function in the space in which they are placed. For example, each real scene of the dataset can include one or more accessories, one or more plants, one or more books, one or more frames, one or more kitchen accessories, one or more cushions, one or more lamps, one or more curtains, one or more vases, one or more carpets, one or more mirrors, and / or one or more electronic objects (e.g., a refrigerator, a freezer, and / or a washing machine). The predetermined set of classes can include a respective class for each of these decorative objects.

[0080] The predetermined set of classes also includes a plurality of geometric classes. The plurality of geometric classes can supplement the plurality of semantic classes. Each object can have a class in the semantic classes, in addition to a class in the geometric classes. Each geometric class can be defined by a geometric criterion applied to the geometry or shape. For example, a geometric class can be defined using only geometric properties of an object (such as its shape and / or size). A geometric class can be defined using only geometric properties of its localization representation (e.g., the shape and / or size of its bounding box). Geometric classes can be assigned to objects that are underrepresented in the dataset. For example, these objects can appear in less than a predetermined percentage of real scenes (e.g., in less than 10% of real scenes).

[0081] S10 obtaining the dataset can comprise generating a 3D point cloud of the dataset, e.g. based on a point cloud generation method. The point cloud generation method can comprise acquiring each 3D point cloud in any way, e.g. by scanning a real scene. Each 3D point cloud can comprise points sampled on surfaces of objects present in the real scene. Each point of the 3D point cloud can comprise spatial coordinates in a 3D space. The user can perform the scanning within the real scene using a scanner. Alternatively, the scanning of the real scene can be performed virtually by scanning a virtual representation of the real scene. The virtual representation can comprise virtual representations of objects present in the real scene, e.g. reproducing shapes of these objects. The scanning can comprise sampling points on surfaces of these virtual representations of objects present in the real scene. Each real scene can be an environment present in the real world or can be fictional, e.g. created by the user to visualize a potential real scene which can then be built. Thus, the dataset can comprise 3D point clouds obtained by virtually scanning a virtual representation of a real scene.

[0082] Alternatively, each 3D point cloud of the dataset can already be generated when the method is performed, e.g. using the same point cloud generation method described above. In this case, S10 obtaining the dataset can comprise retrieving the already generated 3D point clouds, e.g. from a database storing the already generated 3D point clouds. Further alternatively, some 3D point clouds can already be generated while other 3D point clouds can not yet be generated when the method is performed. In this case, S10 obtaining can comprise retrieving the already generated 3D point clouds and generating the other 3D point clouds, e.g. by performing the same point cloud generation method described above.

[0083] S10 obtaining the dataset can further comprise annotating each 3D point cloud of the dataset, e.g. based on an annotation method. The annotation method can comprise determining a localized representation and a class of an object annotating the 3D point cloud. The annotation method can be performed in any way. For example, the determination of the localized representation and the class of the object can be performed manually by a user. The annotation method can comprise, for each localized representation, placing the localized representation within a virtual environment representing the real scene and assigning a class to the placed localized representation among a predetermined set of classes for the object represented by the localized representation. Alternatively, the annotation method can be performed automatically. For example, the 3D point cloud can be acquired from a virtual scan of the virtual environment, as described above. In this case, the localized representation can correspond to a virtual representation of an object already present in the virtual environment and the annotation method can comprise retrieving the localized representation from the virtual environment. The virtual environment can further comprise a class of each object represented in the virtual environment and the annotation method can comprise retrieving the class of each localized representation from the virtual environment.

[0084] Alternatively, when performing the method, each 3D point cloud of the dataset can already be annotated (e.g. using the annotation method described above). In this case, S10 obtaining the dataset can comprise retrieving the annotated 3D point clouds, e.g. from a database storing the annotated 3D point clouds. Further alternatively, when performing the method, some 3D point clouds can already be annotated, while others can not. In this case, S10 obtaining can comprise retrieving the annotated 3D point clouds and annotating the other 3D point clouds, e.g. by performing the same annotation method described above.

[0085] S20 training the function can be performed in any way. For example, S20 training the function can be performed in a supervised way. S20 training the function can comprise training the function to predict, based on examples of pairs of 3D point clouds and annotations present in the dataset (i.e. the training dataset), an annotation (i.e. a localization representation and a class) for a given 3D point cloud. The function is trained to output a representation of an object present in a real scene from which the 3D point cloud is obtained as input, wherein each representation of an object is localized, i.e. it comprises coordinates of the position of the object in the real scene. The function is also trained to output a class for each object (e.g. with a label assigned to the localization representation). S20 training can comprise determining weights of the function so that the function is able to provide output annotations very close to the annotations of the 3D point clouds comprised in the dataset.

[0086] In an example, S10 obtaining can comprise determining a plurality of geometric classes based on the objects represented in the dataset. In this case, each 3D point cloud of the dataset can initially be annotated with a localization representation for each object, but only annotated with a class for semantic classes. S10 obtaining can comprise S11 obtaining 3D point clouds annotated with a localization representation and a semantic class only for semantic objects. Objects not annotated with a class in the semantic classes and / or objects not belonging to any of the above semantic classes can not be labeled. Each 3D point cloud can only comprise a localization representation for each of these remaining objects (i.e. not annotated with a class in the semantic classes). For each remaining object, S10 obtaining can comprise S13 computing a value of a geometric descriptor for the remaining object. Then, S10 obtaining can comprise defining a plurality of geometric classes with the values of the geometric descriptors computed for all the remaining objects. In particular, the plurality of geometric classes can partition the computed values of the geometric descriptor. For example, S10 obtaining can comprise S14 clustering the remaining objects according to a partitioning of the distribution of the computed values. In this case, the plurality of geometric classes can correspond to the resulting clusters. Each cluster can define a respective geometric class. S14 clustering can be performed so that the number of objects within a cluster (i.e. a resulting geometric class) is comparable to the number of objects within a semantic class, to ensure a balanced dataset without over or under represented classes.

[0087] The geometric descriptor of a given object can be defined by one or more geometric properties of the given object (or of its localized representation). The localized representation of an object can be a bounding box that encloses the object. Each representation is localized in the real scene, i.e., it includes the position coordinates of the object in the real scene. The geometric descriptor of a given object can include a measure of the localized representation of the given object (e.g., the bounding box that encloses the object). The measure can measure one or more geometric properties of the given object.

[0088] In an example, the geometric descriptor can have direction invariance at least with respect to a vertical axis. In other words, the geometric descriptor can be invariant with respect to the direction of the object with respect to the vertical axis. This means that for two identical objects having different directions with respect to the vertical axis (e.g., two tables facing different directions, or two chairs placed opposite to each other), their geometric descriptor values can be the same. For example, the one or more geometric properties of the given object used to compute the geometric descriptor can be invariant with respect to the direction with respect to the vertical axis (e.g., the geometric properties can be computed from the dimensions and / or volume of the object). Such a geometric descriptor improves the classification of objects by not considering their direction in the real scene when classifying the objects. In an example, the geometric descriptor can also have invariance with respect to other direction(s), e.g., with respect to the direction of one or more horizontal axes.

[0089] In an example, the geometric descriptor of a given object can include one or more coordinates, each associated with a respective geometric property of the object (or of its localized representation). For example, the geometric descriptor can include any one or any combination of the following coordinates. The geometric descriptor can include one or more coordinates, each representing a respective dimension of the localized representation of the given object (e.g., one each of the length, width, and height of the bounding box). For example, the geometric descriptor can include a respective coordinate for each of the minimum and / or maximum dimensions of the bounding box. Alternatively or additionally, the geometric descriptor can include a coordinate representing a ratio between the minimum and maximum values of the dimensions of the bounding box of the given object. Alternatively or additionally, the geometric descriptor can include a coordinate representing the area of the bounding box of the given object. The area can be the area of the base of the bounding box (i.e., the result of multiplying the length and the width of the bounding box). Alternatively or additionally, the geometric descriptor can include a coordinate representing the volume of the bounding box of the given object. The volume of the bounding box can be the result of multiplying the length, width, and height of the bounding box.

[0090] In an example, the dataset initially can comprise a 3D point cloud of objects initially annotated with a class among a set of other semantic classes. The other semantic classes can comprise objects in the dataset that are under-represented compared to the objects comprised in the plurality of semantic classes. For example, these objects can appear in less than a predetermined percentage of real scenes (e.g. in less than 10% of real scenes). These objects belonging to the other semantic classes comprise the remaining objects clustered for determining the geometric classes. In this case, S10 obtaining can comprise: S12 identifying at least a portion (e.g. all) of the objects of the other semantic classes among all the objects. The identified objects can be the remaining objects considered for determining the geometric classes. Then, S10 obtaining can comprise: S15 assigning a geometric class to each of the identified objects. S15 assigning can be performed after determining the geometric classes (e.g. based on steps S13 and S14). S15 assigning can comprise, for each of the identified objects, computing a geometric descriptor of the identified object (e.g. by computing each coordinate of the geometric descriptor), determining a geometric class to which the identified object belongs based on the geometric descriptor (e.g. by determining to which cluster it belongs), and assigning the determined geometric class to the identified object. The training of this functionality can be performed based on the assigned geometric classes.

[0091] In an example, S12 identifying at least a portion of the objects of the other semantic classes can comprise filtering all the objects of the other semantic classes based on at least one geometric criterion. This allows to keep a reasonable number of such objects of the other semantic classes to maintain a balanced dataset by filtering out the smallest, largest and / or flattest objects. For example, the at least one geometric criterion comprises a criterion based on a volume of a bounding box, a criterion based on a ratio between a minimum and a maximum of a dimension of the bounding box and / or a criterion based on a result of a multiplication of the volume of the bounding box and the ratio.

[0092] Reference is made to Figures 2 to 15 Implementation examples of the machine learning method and of the use method are now given.

[0093] The method comprises defining classes of 3D objects defined by geometric criteria. In addition to the semantic classes well represented and well defined in the annotation of the 3D scene understanding dataset, the method proposes to add geometrically defined classes (geometric classes) to supervise the learning of the functionality (also called model below) so as to take into account objects belonging to under-represented classes; for this, the method groups them by geometric descriptors instead of semantic descriptors.

[0094] Furthermore, the geometrically defined classes can be used alone to supervise to unlock generic object detection without any semantics, which means that objects can be grouped by shape features. In this setup, optionally, semantic labels can be added for objects belonging to semantic classes well enough represented in the dataset.

[0095] In the field of 3D scene understanding, adding geometry-defined classes to supervise the training of neural networks has several advantages. First, publicly available datasets provide a limited number of semantic classes as annotations to train neural networks. This results in models that, during their training, can only understand the classes of the objects they are trained on, thus ignoring a large number of objects in real-world scene inference. This method allows these elements to be taken into account, if not through semantic classes, at least through geometry or shape. For example, in the case of 3D object detection, a model trained on both semantically and geometry-defined classes is able to detect more objects than those belonging to the semantic classes given in the training dataset.

[0096] Furthermore, this also enables the model to better understand the semantic classes used for training (in addition to the geometry-based classes). Indeed, using geometry-defined classes can reduce the ambiguity of semantically defined classes. That is, in addition to being able to handle a large number of diverse objects, models trained with these classes also achieve better accuracy in handling semantically defined objects.

[0097] The improvement in detecting objects annotated with semantic class labels when adding geometry-defined classes of the present application can be illustrated by considering more objects in a 3D scene that will be taken into account in the loss calculation during the training process. By allowing the model to take advantage of geometry class objects to reduce ambiguity, geometry classes help the model better distinguish objects of semantic classes. Without these geometry classes, many objects present in a 3D scene are actually not annotated, which implicitly labels them as "background" objects that the model should learn not to recognize. By explicitly considering more objects and more classes in the loss, annotating objects with geometry classes helps the model's general understanding, which helps better distinguish each semantic class from other classes (including geometry-defined classes). Without geometry classes, this is done implicitly, but results in poorer results.

[0098] A 3D scene understanding dataset is composed of 3D data samples in various 3D formats (including RGBD (RGB-Depth) frames, point clouds, meshes, voxel grids...) and their annotations. Depending on the task for which the model is trained, the annotations considered can vary. For example, for semantic segmentation, each point is annotated with its semantic class. For 3D object detection, the annotations consist of a list of bounding boxes defined by a semantic class, a 3D position (x, y, z), a size (w, l, h), and an orientation (0). This method includes creating a new type of annotation: geometry-defined classes of 3D objects. This geometry-defined class can be used instead of the semantic class of an object when the semantic class is not well represented within the dataset or is ambiguously defined. The geometry-defined class is defined using only the geometric properties of a given 3D object (such as its shape, the size of its bounding box, etc.).

[0099] Let us consider a 3D point cloud scene dataset used to train a 3D object detection model with 175 semantic object classes. Out of these classes, only 29 classes are considered to be well-represented in the dataset used for training, leaving out 146 object classes. Figure 2 The distribution of the number of occurrences of objects of the 175 semantic classes is shown. The number of semantic classes considered by the method is 29 well-represented semantic classes (with the highest counts, see Figure 9 ), the other 146 remaining semantic classes are not considered (they are referred to as “other semantic classes” before). The method uses geometric classes for objects belonging to these 146 remaining classes. These geometric classes are also referred to as “meta-classes” in the following.

[0100] Figure 3 An example of an object 10 belonging to the other semantic class “accessories” that is not well-defined is shown. The samples of the “accessories” class include small kitchen utensils, ornaments, recycling bins, and bathrobes. A well-defined class is a class with a clear semantic definition (e.g. the semantic class “chair”). In contrast, the semantic class “accessories” contains a wide range of objects that are too broad to have a clear semantic definition. Well-defined classes can be selected manually, e.g. using qualitative criteria.

[0101] Figure 3 An example of an object 20 belonging to the other semantic class “bathtubs” that is not well-represented is also shown. A well-represented class is also a class that occurs often enough in the dataset. The other semantic class “bathtubs” is a well-defined semantic class, but when the dataset lacks bathrooms, their occurrence in the dataset is not enough for the model to learn to recognize them. The “bathtubs” class is very consistent, but these objects only occur 80 times in a dataset of more than 2000 room examples. The method can include selecting only classes that occur in more than 10% of the real scenes (e.g. rooms) in the dataset.

[0102] In this implementation example, the method only uses the bounding boxes of the objects, ignoring the geometry of the point cloud inside the bounding boxes. All bounding boxes can be represented in a simplified 2D space with the following transformation, regardless of the initial semantics of the objects:

[0103]

[0104] where, as Figure 4As illustrated, (x, y, z) are the coordinates of the 3D position of the bounding box of the object, (w, l, h) are its dimensions, and Θ is its orientation. The resulting vector in the simplified 2D space can be a geometric descriptor of the given object and can comprise a first coordinate being the ratio between the minimum and maximum of the dimensions (w, l, h) of the bounding box, and V being the volume of the bounding box.

[0105] The method then groups the objects according to the shape of their bounding boxes, from small planar objects to large cuboid objects. The method preserves a reasonable number of such new objects, keeping a balanced dataset by filtering out too small, too large and too flat objects. To this end, the method comprises filtering all objects of other semantic classes based on at least one geometric criterion. The at least one geometric criterion can be based on the ratio and the volume discussed previously. The at least one geometric criterion can be applied using the following formula:

[0106] min volume <V<max volume

[0107]

[0108] min criterion <criterion<max criterion

[0109] wherein:

[0110] • V denotes the volume of the bounding box in cubic meters.

[0111] • min volume is the minimum volume of the bounding box accepted in the meta-class. A suitable value is min volume ∈ [0, 1].

[0112] • max volume is the maximum volume of the bounding box accepted in the meta-class. A suitable value is max volume ∈ [3, 5].

[0113] • l denotes the length of the bounding box in meters.

[0114] • w denotes the width of the bounding box in meters.

[0115] • h denotes the height of the bounding box in meters.

[0116] • min ratio is the minimum ratio min / max of the bounding box accepted in the meta-class, min ratio ∈]0, 1], a suitable value is min ratio∈]0, 0.3].

[0117] • max ratio is the maximum ratio min / max, max ratio ∈]0, 1], appropriate values are min ratio ∈]0.7, 1].

[0118] • criterion is a geometric criterion used for further filtering, defined by Other geometric criteria can be defined and used (explained in detail later).

[0119] • min criterion is the minimum value of the criterion for bounding boxes accepted in the meta class. Appropriate values are min criterion ∈]0, 0.4].

[0120] • max criterion is the maximum value of the criterion for bounding boxes accepted in the meta class. Appropriate values are max criterion ∈[1, 2].

[0121] In this implementation example, the method can use the following values:

[0122] • min volume = 0

[0123] • max volume = 3

[0124] • min ratio = 0.3

[0125] • max ratio = 1

[0126] •

[0127] • min criterion = 0.2 m 3

[0128] • max criterion = 1.5 m 3

[0129] The method can then include filtering the objects by ordering them by the criterion value and keeping the objects between given bounds. The method can include determining these bounds in such a way that the number of objects within each meta class is comparable to the number of objects within the semantic class, to ensure a balanced dataset without over or under represented classes. Figure 5 An example of bounding boxes ordered by is shown.

[0130] The method then includes determining the geometric classes. The method can include clustering the remaining bounding boxes (after filtering) to create n object clusters, each cluster forming a meta-class. The method can use any unsupervised clustering method. Figure 6 An example of a clustering method that can be used to cluster with n-1 lines is shown. The clustering method can include defining the slope of the i-th line (where i is an integer, i ∈ [1, n-1]) with the following equation: In the diagram, x represents the slope of the line. The intercept theorem (Thales) provides the following equation:

[0131]

[0132] Figure 7 Three object clusters 101, 102 and 103 resulting from the division into 3 clusters (i.e. 3 meta-classes) using lines as separators are shown. In other examples, the method can cluster using any of a number of other algorithms or heuristics, such as K-means, Gaussian Mixture Model or other unsupervised learning methods, in any combination.

[0133] As mentioned previously, other methods can be used to determine such a division. For example, with the point clouds of each object, the method can use any of the following two methods:

[0134] • A first method includes, for each point cloud, computing an embedding through a point cloud encoder (such as PointNet++ (as discussed in Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in Euclidean Spaces”, 2017, which is incorporated herein by reference)) and using a clustering method (such as K-means, k representing the number of meta-classes) on the obtained embedding, thereby obtaining the final division.

[0135] • A second method includes computing an embedding for each point cloud based on geometric features (hand-crafted or non-hand-crafted) (such as one or a combination of those detailed in Xian-Feng Han, Shi-Jie Sun, Xiang-Yu Song, Guo-Qiang Xiao, “3D Point Cloud Descriptors in the Era of Hand-Crafted and Deep Learning: A State-of-the-Art Survey”, 2020, which is incorporated herein by reference) and then using any clustering method to determine the division.

[0136] Figure 8An example of a use of the method is shown. In this example, the use of the method is included in the process of an example task of 3D object detection in a point cloud, with the three metaclasses defined previously. The use of the method comprises: S30 obtaining a 3D point cloud 110. S30 obtaining a 3D point cloud 110 can be achieved by scanning a real scene (for example, a virtual scan based on a virtual representation of the real scene). Then, the use of the method comprises: S31 applying the function to the obtained 3D point cloud. The function outputs a localized representation of the detected objects of the real scene (for example, bounding boxes 111, 112, 113, 114, 115). Figure 9 The 29 semantic classes retained in the predetermined set of classes (“plants”, “wardrobe”, “curtains”, etc.) and the 3 geometric classes created (“meta_0”, “meta_1” and “meta_2”) are shown. The objects not represented by a (semantic) class are detected as meta-classes (geometric classes). In this example, the bathtub 113, the refrigerator 114 and the extractor hood 115 are detected this way. These objects 113, 114, 115 do not belong to a semantic class, but are detected based on the defined geometric classes. This example illustrates that the method is able to detect objects that do not belong to a semantic class (i.e. underrepresented objects such as a bathtub, a toilet, a household appliance or an electronic device).

[0137] In addition, as shown in Table 1 below, the introduction of geometric classes also improves the accuracy of the model on semantic classes. For each considered metric, the function is more accurate in predicting objects belonging to a semantic class with geometric classes than without geometric classes.

[0138] Table 1:

[0139]

[0140] The used metrics will now be presented in more detail.

[0141] The commonly used metrics in 3D object detection are the mean average precision (mAP) and the mean average recall (mAR). These metrics are defined using an intersection over union (IoU) threshold to evaluate the predictions of a model. For 3D bounding boxes, these union and intersection values are computed as volumes; for clarity, 2D equivalents are visualized as areas in Figure 10

[0142] ​For a given class c, a predicted bounding box can be a correct prediction if it can be paired with a ground truth bounding box such that the IoU between the two bounding boxes is above a given threshold. Pairing is unique in that a given ground truth can be assigned to at most one prediction and vice versa. Pairing is done in descending order of IoU until all ground truth or predicted bounding boxes are exhausted.

[0143] The following metrics can be used:

[0144] • True Positive (TP): Correct model prediction, i.e., a prediction that matches a bounding box of the same class with IoU > threshold.

[0145] • False Positive (FP): Incorrect model prediction, i.e., a prediction by the model that does not match a corresponding ground truth annotation or IoU < threshold.

[0146] • False Negative (FN): Missing prediction. An annotation that does not match any prediction (i.e., an object is present but not detected by the model).

[0147] • Precision provides the fraction of actual positive predictions that are correct.

[0148]

[0149] • Recall provides the fraction of actual positives that the model was able to correctly identify.

[0150]

[0151] • Precision-Recall Curve: The trade-off between the quality and quantity of predicted bounding boxes can be controlled after inference, typically in the form of a confidence score associated with each class and bounding box by the model. This score can be thresholded to eliminate predictions that the model is not confident about, and thus a precision-recall plot can be drawn as a function of the confidence score threshold.

[0152] • Average Precision (AP) is the area under the precision-recall curve for each class (see Figure 11 for precision-recall curves obtained for three classes).

[0153] • Mean average precision (mAP) is the average of the APs for all classes. We usually write mAP@threshold to indicate at which IoU threshold (i.e. at which level of quality) the metric is computed.

[0154] • Average Recall (AR) is the average of the recall rates over a range of IoU thresholds (from 0.5 to 1.0), which can be approximated as follows, where o is the IoU, recall(o) is the recall-IoU curve with recall on the y-axis and IoU varying on the x-axis.

[0155] • Mean average recall (mAR) is the average of the ARs for all classes.

[0156]

[0157] The results of this method are now discussed.

[0158] The following table 2 illustrates the benefits of filtering other semantic class objects.

[0159] Table 2:

[0160]

[0161]

[0162] The metric is measured only on the semantic class. Table 2 shows that filtering gives more robust results. In fact, the results are usually better than the baseline. It is reminded here that mAP (mean average precision) is related to the number of correct predictions among all model predictions, while mAR (mean average recall) is related to the number of correctly detected objects among all ground truth objects.

[0163] The following table 3 shows the prediction accuracy of this function for different numbers of geometric classes.

[0164]

[0165] The metric is measured only on the semantic class. The results are with the filtering step. The results show that the prediction is better for each number (1, 3 or 5) of geometric classes compared to the baseline (i.e. without geometric classes).

[0166] The following table 4 shows the prediction accuracy of this function for different geometric descriptors.

[0167] Table 4:

[0168]

[0169] The metric is measured only on the semantic classes, filtered if applicable and with 3 meta-classes. The results show that the prediction accuracy is better for each geometric descriptor compared to the baseline that does not include (i.e. without geometric classes). The better performance of the geometric descriptor (w*l,h) illustrates the advantage of the invariance at least around the vertical axis. Indeed, in indoor scenes, most of the electrical appliances will not lay down but only rotate around the z-axis (vertical axis).

[0170] Table 5 below illustrates another example with a small number of semantic classes considered.

[0171] Table 5

[0172]

[0173] In this other example, only 6 semantic classes are considered: chair, sofa, desk, cupboard, table and bed. Three geometric classes are defined. The metric is measured only on the semantic classes. This table shows that even in this case, the results of this functionality using 3 geometric classes are more accurate than the baseline (i.e. without geometric classes).

[0174] Tables 6 and 7 below provide the results obtained for the semantic classes and the geometric classes. The results show that the detections from the meta-classes are useful as they are well detected compared to the semantic classes (e.g. "vase"). The value of the metric "AP@0.25" for meta 0 is particularly high, at 0.49.

[0175] Table 6

[0176]

[0177] Table 7

[0178]

[0179]

[0180] Table 8 below illustrates the usefulness of the geometric classes.

[0181] Table 8:

[0182]

[0183] The results of Table 8 show that, as for the semantic classes, training the model on a filtered, divided subset of objects improves the overall detection.

[0184] Figure 12An example of 3D scene understanding is shown. In this example, the use method is included in a process of 3D scene understanding, which comprises reconstructing a 3D scene from a 3D point cloud. The process comprises performing the use method, in particular the steps S30 of obtaining a 3D point cloud of a real scene and S31 of applying the function to the obtained 3D point cloud. The function outputs a localized representation (in this example, a bounding box) of an envelope of an object within the real scene. The function also outputs, for each localized representation, a class of the object represented by the localized representation, from among semantic and geometric classes.

[0185] The process also comprises S32 reconstructing walls and floors of the real scene based on the 3D point cloud. S32 reconstructing the walls and floors can be performed after S31 applying the function, or in parallel therewith. After applying the function, the process comprises S33, for each localized representation, retrieving a 3D representation (e.g., a computer-aided design (CAD) model) of the object represented by the localized representation. S33 retrieving can be based on the class of the localized representation output for the object. The process then comprises S34 assembling the retrieved 3D representations in the 3D scene, thereby reconstructing the 3D scene of the 3D point cloud. S34 assembling can be performed according to the positions of the localized representations output by the function (each 3D representation representing a respective real object in the real scene).

[0186] Figure 13 An example of the architecture of the function is shown. The function has an architecture comprising a voxelization layer 201 configured to voxelize the 3D point cloud as input. The output of the voxelization layer 201 can be a sparse occupancy grid. The function also has a convolutional neural network 202, which takes as input the 3D point cloud voxelized by the voxelization layer. The convolutional neural network outputs a localized representation and a class for each object. The convolutional neural can have a backbone (which comprises a sparse 3D ResNet (Residual Neural Network) CNN (Convolutional Neural Network) with CBAM (Convolutional Block Attention Module) blocks), a neck (which comprises pruning and decoder layers), and a head (which comprises regression bounding box parameters, classification, mobius band parameterization (encoding the invariance of oriented bounding boxes), and CBAM blocks as refinements).

[0187] For each ground truth bounding box, the last feature (grid) level of the bounding box covering at least N loc positions is selected. The positions are filtered with center sampling: only the points close to the center of the box (using centrality) are kept as positive matches.

[0188] The loss can be defined with the following formula:

[0189]

[0190] where N pos is the number of matching positions, L cls is the focal loss for classification, L reg is the IoU between boxes, regression loss, L cntr is the binary cross-entropy for center.

[0191] The method can use parameterization to have invariance with respect to the front of the bounding box. Using the original parameterization, 4 different bounding boxes can represent the same object. By representing q = w / l, the following parameterization defines the same bounding box:

[0192]

[0193] In fact, the 4 parameters also represent the same point on the Mobius strip. Therefore, the method can re-parameterize the bounding box using the classical Mobius strip embedding. Given a position A bounding box (x, y, z, w, l, h, Θ) can be re-parameterized into (δ1, δ2, δ3, δ4, δ5, δ6, δ7, δ8) using the following equations:

[0194]

[0195] Figure 14 An example of results of trained functions is shown. In particular, the figure shows results obtained for two functions 310, 320 trained using the method compared to ground truth 300. The results are illustrated using the legend in Figure 9 The results show that using geometric classes more objects can be detected and objects in semantic classes can be detected more accurately.

[0196] Figure 15 An example of a system is shown, where the system is a client computer system, for example, a workstation of a user.

[0197] The client computer of this example includes a central processing unit (CPU) 1010 connected to an internal communication bus (BUS) 1000, a random access memory (RAM) 1070 also connected to BUS. The client computer is also equipped with a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to BUS. Video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device (e.g., a hard disk drive 1030). Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; and magneto-optical disks. Any of the foregoing can be supplemented by, or incorporated in, specially designed ASICs (application-specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer can also include haptic devices 1090, such as a cursor control device, keyboard, etc. A cursor control device is used in the client computer to allow the user to selectively position a cursor on a display 1080 at any desired position. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices to input control signals to the system. Typically, the cursor control device can be a mouse, the buttons of which are used to generate signals. Alternatively or additionally, the client computer system can include a touchpad and / or a touch screen.

[0198] A computer program can include instructions executable by a computer, which instructions include means for causing the system described above to perform the method. The program can be recorded on any data storage medium including the memory of the system. For example, the program can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program can be implemented as a device, for example, a product tangibly embodied in a machine-readable storage device, for execution by a programmable processor. The method steps can be performed by a programmable processor executing a program of instructions to perform functions of the method by operating on input data and generating output. The processor can be programmable and can be coupled to a data storage system, at least one input device, and at least one output device, to receive data and instructions from, and send data and instructions to, the same. The application program can be implemented in a high level procedural or object oriented programming language to be executed by a general purpose computer. As a non-limiting example, assemblies or machine languages can also be employed. In any case, the language can be a compiled or interpreted language. The program can be a complete installation program or an update program. In any case, application of the program on the system results in instructions for performing the method. Alternatively, the computer program can be stored and executed on a server of a cloud computing environment, the server communicating with one or more clients over a network. In this case, the processing unit executes the instructions contained in the program, thus causing the method to be executed on the cloud computing environment.

Claims

1. A computer-implemented method for a machine learning function, the function being configured to take as input a 3D point cloud of a real scene and output each localized representation of a corresponding object of the real scene and, for each corresponding object, output a category of the corresponding object from a predetermined set of categories, the method comprising: - obtaining a dataset of annotated 3D point clouds, wherein each 3D point cloud is annotated with each localized representation of a corresponding object, and each corresponding object is annotated with a class of the corresponding object from the predetermined set of classes; and - training said function based on the obtained dataset, The predetermined category set includes a plurality of semantic categories and a plurality of geometric categories.

2. The method according to claim 1, wherein The obtaining of the data set comprises: - Obtain the 3D point cloud annotated with: o For each 3D point cloud, each localized representation of the corresponding object, and o for each respective object in at least a portion of the objects, a category of the respective object in the semantic category; - calculating the value of the geometric descriptor for each remaining object of the 3D point cloud; and - clustering the remaining objects according to a partitioning of the distribution of the calculated values, wherein the plurality of geometric classes corresponds to the obtained clusters.

3. The method according to claim 2, wherein: The geometric descriptor has orientation invariance at least with respect to a vertical axis.

4. The method according to claim 3, wherein: The geometric descriptor of a given object includes measurements of a bounding box of the given object.

5. The method according to claim 3 or 4, wherein: The geometric descriptor of a given object includes: - one or more coordinates, each coordinate representing a respective dimension of said bounding box of said given object; - coordinates representing the ratio between the minimum and maximum values ​​of the dimensions of the bounding box of the given object; - coordinates representing the area of ​​said bounding box of said given object; and / or - Coordinates representing the volume of said bounding box of said given object.

6. The method according to any one of claims 1 to 5, wherein The obtaining of the data set comprises: - Obtain the 3D point cloud annotated with: o For each 3D point cloud, each localized representation of the corresponding object, o for each respective object in at least a portion of said objects, a category of said respective object in said semantic category, and o For each remaining object, the category of the remaining object in the set of other semantic categories; - identifying at least a portion of the objects of said other semantic category among all said objects; and - Assigning a geometric class to each identified object.

7. The method according to claim 6, wherein: Identifying at least a portion of the objects of the other semantic category includes filtering all objects of the other semantic category based on at least one geometric criterion.

8. The method according to claim 7, wherein: The at least one geometric criterion comprises: - Bounding box volume based criteria; - a criterion based on the ratio between the minimum and maximum values ​​of the bounding box dimensions; and / or - A criterion based on the result of multiplying the bounding box volume and the ratio.

9. The method according to any one of claims 1 to 8, wherein The functionality is configured to perform 3D object detection or 3D object segmentation.

10. The method according to any one of claims 1 to 9, wherein The 3D point cloud of the dataset is a 3D point cloud of an indoor scene and / or is obtained by scanning.

11. The method according to any one of claims 1 to 10, wherein The function has an architecture including a voxelization layer and / or a convolutional neural network, wherein the voxelization layer is configured to voxelize the 3D point cloud as input, and the convolutional neural network takes the 3D point cloud voxelized by the voxelization layer as input.

12. A computer-implemented method for using a function learned by machine learning according to any one of claims 1 to 11, comprising: - Obtain 3D point cloud; as well as - Apply the said function to the obtained 3D point cloud.

13. A computer program product comprising a computer program, the computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 11 and / or the method according to claim 12.

14. A computer-readable storage medium having recorded thereon the computer program according to claim 13.

15. A system comprising a processor coupled to a memory having recorded thereon the computer program of claim 13.

Images