Interactive object selection

Abstract

To provide a computer-implemented method of machine-learning including providing a dataset of 3D point clouds.SOLUTION: In the method of machine-learning, each 3D point cloud including at least one object is equipped with a specification of one or more graphical user interactions each representing a respective selection operation of the same object in the 3D point cloud. The method further includes learning, based on the dataset, a neural network configured for segmenting an input 3D point cloud including an object. The segmenting is based on the input 3D point cloud and on a specification of one or more input graphical user interactions each representing a respective selection operation of the object in the 3D point cloud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of computer programs and systems, and more particularly to machine learning methods, systems, and programs. [Background technology]

[0002] Numerous systems and programs are available on the market for designing, engineering, and manufacturing objects. CAD stands for Computer-Aided Design and refers to software solutions for designing objects. CAE stands for Computer-Aided Engineering and refers to software solutions for simulating the physical behavior of future products. CAM stands for Computer-Aided Manufacturing and refers to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays an important role in the efficiency of the technology. These technologies can be incorporated into Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage corporate knowledge for product development from conception to the end of the product's lifespan across the extended enterprise. Dassault Systèmes' PLM solutions (product names: CATIA, ENOVIA, DELMIA) provide an Engineering Hub to organize product engineering knowledge, a Manufacturing Hub to manage product engineering knowledge, and an Enterprise Hub to enable enterprise integration and connectivity to both the Engineering and Manufacturing Hubs. All systems provide an open object model that connects products, processes, and resources, enabling dynamic knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this and other contexts, point cloud segmentation has become very important. The following papers are relevant to point cloud segmentation and are referenced below: [1]Aleksey Golovinsky, Thomas Funkhouser. “Min-Cut Based Segmentation of Point Clouds”-ICCV Workshops,2009, [2] Kun Liu, Jan Boehm. “A New Framework or Interactive Segmentation of Point Clouds”-Int.Arch.Photogramm.Remote Sens.Spatial Inf.Sci.2014, and [3] Sedlacek D., Zara J. “Graph Cut Based Point-Cloud Segmentation for Polygonal Reconstruction”-ISVC 2009. Summary of the Invention [Problem to be solved by the invention]

[0004] These methods suffer from lack of efficiency, robustness, and generalizability.

[0005] In this context, improved machine learning methods for point cloud segmentation are needed. [Means for solving the problem]

[0006] Thus, a computer-implemented machine learning method is provided. The method includes providing a dataset of 3D point clouds. Each 3D point cloud includes at least one object. Each 3D point cloud includes a specification of one or more graphical user interactions, each of which represents a selection of the same object in the 3D point cloud. The method further includes training a neural network configured to segment input 3D point clouds including the object based on the dataset. The segmentation is based on the input 3D point clouds and one or more input graphical user interaction specifications, each of which represents a selection of the object in the 3D point cloud.

[0007] The method may include one or more of the following.

[0008] The 3D point cloud dataset is provided by providing 3D point clouds and, for each 3D point cloud, information regarding the position of said same object in the 3D point cloud; For each 3D point cloud, determining the specification by simulating each of the graphical user interactions of said one or more graphical user interactions; providing the 3D point cloud with a specification of the one or more graphical user interactions; each of the one or more graphical user interactions corresponds to one or more seed locations defined on a 3D point cloud, and simulating the graphical user interactions includes determining the one or more seed locations; the one or more graphical user interactions include first graphical user interactions for selecting the same object, each of which corresponds to one or more first seed positions defined on the same object; determining the one or more seed locations includes spreading the one or more first seed locations over the same object; for at least one 3D point cloud, the one or more graphical user interactions further include second graphical user interactions for discarding regions outside the same object, the second graphical user interactions each corresponding to one or more second seed locations defined outside the same object; and / or wherein providing the 3D point cloud with the specifications includes adding, to each point of the 3D point cloud, a coordinate that quantifies the intensity of a selection operation at that point based on a respective simulation of a graphical user interaction; each of the one or more graphical user interactions being for selecting the same object or discarding an area outside the same object, and for each point of the 3D point cloud, the adding of coordinates includes: Setting the coordinates to initial values; for each graphical user interaction for selecting the same object, incrementing the coordinate according to the proximity between the graphical user interaction and the point; and for each graphical user interaction to discard areas outside the same object, decreasing the coordinate according to the proximity between the graphical user interaction and the point.

[0009] Additionally, neural networks trainable according to the present method, for example neural networks trained according to the present method, are provided.

[0010] Also provided is a computer-implemented method of using the neural network, the method comprising providing a 3D point cloud, the 3D point cloud including an object, and one or more iterations of: performing a selection operation for the object by performing one or more graphical user interactions; and applying the neural network to segment the 3D point cloud based on the 3D point cloud and a specification of the one or more graphical user interactions.

[0011] The method of use may include one or more of the following:

[0012] The method further comprises, after performing the selection operation and before applying the neural network, determining, for each of the one or more graphical user interactions, one or more seed locations defined by the graphical user interaction, thereby determining the specification; and / or - providing said 3D point cloud with said specification, said specification comprising adding to each point of said 3D point cloud a coordinate quantifying the strength of a selection operation at that point. The one or more graphical user interactions include one or more of the following:

[0013] performing one or more clicks on said object; Performing a stroke on an object; defining a bounding box on and / or around the object; performing one or more clicks outside said object; and / or Performing strokes outside of an object.

[0014] Furthermore, a computer program comprising instructions for carrying out said method and / or said method of use is provided.

[0015] Furthermore, there is provided an apparatus comprising a data storage medium having the computer program and / or the neural network recorded thereon.

[0016] The device may form or function as a non-transitory computer-readable medium, for example, on a software-as-a-service (SaaS) or other server, or cloud-based platform. Alternatively, the device may include a processor connected to the data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of an overall system). The system may further include a graphical user interface connected to the processor. Embodiments of the present invention will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0017] [Figure 1] FIG. [Figure 2] FIG. [Figure 3] FIG. [Figure 4] FIG. [Figure 5] FIG. [Figure 6] FIG. [Figure 7] FIG. [Figure 8] FIG. [Figure 9] FIG. [Figure 10] FIG. [Figure 11] FIG. [Figure 12] FIG. [Figure 13] FIG. [Figure 14] FIG. [Figure 15] FIG. [Figure 16] An example of this system. DETAILED DESCRIPTION OF THE INVENTION

[0018] A computer-implemented machine learning method is proposed. The method includes providing a dataset of 3D point clouds. Each 3D point cloud includes at least one object. Each 3D point cloud includes one or more specifications of graphical user interactions, each representing a selection of the same object in the 3D point cloud. The method further includes training a neural network configured to segment input 3D point clouds containing objects based on the dataset. The segmentation is based on the input 3D point clouds and one or more input specifications of graphical user interactions, each representing a selection of the object in the 3D point cloud. The method is also referred to as a training method.

[0019] This constitutes an improved machine learning method for point cloud segmentation.

[0020] In particular, the present training method trains a neural network configured to segment an input 3D point cloud containing an object. This means that the present training method uses machine learning, e.g., deep learning, in the context of point cloud segmentation, which is not a traditional approach. In particular, this benefits from the well-known advantages of machine learning, e.g., deep learning, in a new technological context. Indeed, as described in this disclosure, this approach has been successfully tested.

[0021] The neural network is trained based on a provided dataset of 3D point clouds, each of which includes one or more specifications of graphical user interactions, each of which represents a selection of the same object in the 3D point cloud. In other words, for each 3D point cloud, the 3D point cloud includes at least one object, referred to as the "same object" or "same object," and one or more graphical user interactions each represent a selection of this same object. The provided dataset is also referred to as a training dataset. The training dataset thus includes 3D point clouds, each of which includes a specification specifying the selection of the same object through one or more graphical user interactions. This means that the training dataset includes a plurality of pairs of 3D point clouds and data specifying object selections by graphical user interactions in each 3D point cloud. By training a neural network on such a training dataset, the neural network can perform segmentation of the input 3D point cloud based solely on the specification of the input 3D point cloud and the input object selection via graphical user interaction in the 3D point cloud. In other words, the training dataset allows the neural network to be trained in a manner that requires only information about the location of objects in the 3D point cloud and the selection of these objects via graphical user interaction to segment the objects. In this way, the training dataset allows the neural network to be trained to perform 3D point cloud segmentation efficiently and easily. Furthermore, the segmentation depends solely on information about the location of objects in the 3D point cloud and the selection of these objects via graphical user interaction; it does not depend on further specifications, such as the specification of the object category / type / class.Indeed, as will be further explained below, each point in each 3D point cloud of the training dataset may include a single annotation as "object" (i.e., the point belongs to an object, e.g., the same object) or "background" (i.e., the point belongs to the background), i.e., may not include any further annotations, e.g., may indicate a category of object (e.g., chair) for that point. In this way, the neural network does not depend on the type of object in the input 3D point cloud when segmenting. In other words, the neural network can extract any type of object from the input 3D point cloud, not just a specific type of object (e.g., chair). This makes the neural network robust.

[0022] As explained further below, any 3D point cloud herein may be obtained synthetically (e.g., via virtual scan simulation or CAD / mesh conversion) or may result from physical measurements (e.g., 3D scans) performed in the real world. In other words, this makes the neural network robust, as it is agnostic to the type of 3D point cloud that needs to be segmented.

[0023] Furthermore, in addition to being configured to segment a 3D point cloud, the neural network is actually configured to interactively segment the 3D point cloud. In fact, the neural network is trained based on, among other things, the specification of object selection through graphical user interaction, thereby enabling the neural network to segment the input 3D point cloud by detecting object selection through graphical user interaction in the input 3D point cloud. In other words, as described in more detail below regarding the application method, the neural network allows a user to graphically select objects in the input 3D point cloud, and then applies the neural network to automatically obtain a segmentation of the input 3D point cloud based on the selection. This enables segmentation with reduced user effort, since the user does not need to perform numerous and / or complex physical actions. That is, simple, ordinary, and / or physically non-fatiguing graphical interactions with the machine are sufficient. In this way, the present learning method provides an ergonomically improved solution for interactively segmenting 3D point clouds.

[0024] Additionally provided is a neural network trainable, e.g., trained, according to the present method. The neural network is configured to segment an input 3D point cloud containing an object. The segmentation is based on the input 3D point cloud and a specification of one or more input graphical user interactions, each representing a selection of an object in the 3D point cloud. Because the neural network is trainable according to the present method, the segmentation is particularly efficient and has particular generalization power. For example, as described above, the segmentation is independent of the type of 3D point cloud and the type of object contained therein.

[0025] Also provided is a computer-implemented method for using the neural network. The method includes providing a 3D point cloud including an object. The method further includes one or more iterations of: performing one or more graphical user interactions to select the object; and applying the neural network to (e.g., automatically) segment the 3D point cloud based on the 3D point cloud and specifications of the one or more graphical user interactions. This method may also be referred to as a "segmentation method."

[0026] The segmentation method enables interactive segmentation in an ergonomically improved manner. Indeed, as described above, the neural network is trained to enable segmentation by simply performing object selection through graphical user interaction on the 3D point cloud. Once the user performs object selection, the 3D point cloud is (e.g., automatically) segmented by applying the neural network. If the user wants to perform further segmentation or improve the level of detail of the segmentation already performed, the user simply performs another object selection through graphical user interaction, and another segmentation is (e.g., automatically) performed by applying the neural network. In other words, the segmentation method enables interactive (possibly iterative) segmentation of the input 3D point cloud. Furthermore, this segmentation is ergonomic because it does not impose a significant burden on the user. In fact, the user only needs to perform simple, common, and / or fatigue-free graphical interactions, such as clicking, drawing strokes, and / or drawing / defining bounding boxes. For example, these graphical interactions may be performed using a mouse and / or a haptic device, such as a touch-based haptic device (e.g., a touchscreen or touchpad), as described further below. Furthermore, because the neural network is agnostic to the type of 3D point cloud and the category / class / type of object, segmentation may be performed on any type of 3D point cloud containing any type of object.

[0027] The training method and the segmentation method can be performed independently, e.g., by different users and / or on different computers, or they can be integrated into the same computer-implemented process. This process will now be described with reference to Figure 1, which shows a flow chart of one example of the process.

[0028] The process includes an offline phase that incorporates the training method. The offline phase includes providing a 3D point cloud dataset (S10) according to the training method. The offline phase further includes training a neural network configured to segment an input 3D point cloud containing an object (S20) according to the training method. The offline phase may further include storing the trained neural network, e.g., in a memory, e.g., for use in the online phase of the process.

[0029] The process further includes an online stage, which integrates the segmentation method. The online stage may include providing the trained neural network, for example by retrieving it from a memory where it was stored during the offline stage. The online stage includes providing a 3D point cloud including the object according to the segmentation method (S30). The online stage further includes performing one or more iterations of selecting the object according to the segmentation method (S40) and segmenting the 3D point cloud by applying the neural network according to the segmentation method (S50).

[0030] Here, the learning method will be further explained.

[0031] The present learning method is for machine learning of point cloud segmentation.

[0032] As it is known, a point cloud is an unordered collection of points with coordinates (usually 3D) that may be accompanied by additional properties (e.g., encapsulated in additional coordinates), such as intensity or color. The unordered aspect of this data makes it difficult to analyze, especially when compared to structured grids such as images. Point clouds come in various formats, but as mentioned above, trained neural networks are agnostic to these various formats.

[0033] Point cloud segmentation, also known as object segmentation, corresponds to the task of extracting all points in a point cloud that belong to an object from the background points of the point cloud. As mentioned above, neural networks trained by this training method can handle all types of objects, i.e., they are independent of the object category. Furthermore, as mentioned above, neural networks can perform interactive segmentation, where a user expresses interest in an object by interacting with the point cloud.

[0034] This learning method is a machine learning method.

[0035] As is known from the field of machine learning, processing an input by a neural network involves applying an operation to the input, the operation being defined by data including weight values. Training a neural network thus involves determining the values ​​of the weights based on a dataset configured for such training; such datasets are sometimes referred to as learning or training datasets. To that end, the datasets contain data that each form a training sample. The training samples represent a diversity of situations in which the neural network will be used after training. The datasets referred to herein may contain 1,000, 10,000, 100,000, or even more than 1,000,000 training samples. In the context of the present disclosure, "training a neural network on a dataset" means that the dataset is a learning / training dataset for the neural network.

[0036] In the context of this learning method, the training dataset for the neural network is a dataset of 3D point clouds provided and will now be described.

[0037] The training data set includes 3D point clouds, each including at least one object, each equipped with one or more graphical user interaction specifications, each representing a selection of the same object in the 3D point cloud. In other words, each 3D point cloud and its associated specifications form a training sample of the training data set. Because these specifications relate to graphical user interactions, they are also referred to as "interaction channels" or "user indicators." This specification is described further below. For simplicity, such training samples are also referred to simply as "a 3D point cloud and its interaction channel." In either case, each training sample includes a pair consisting of a 3D point cloud and its interaction channel. Each training sample may also include a segmentation mask, which, for each point in the 3D point cloud, defines the point as belonging to the same object or as not belonging to the same object. This allows the neural network to be trained (S20) in a fully supervised manner, as described further below. The training data set may, in one example, include approximately 2000 training samples.

[0038] As is known per se, a 3D point cloud consists of a set of 3D points, each of which is described by, for example, three Cartesian coordinates in 3D space. This means that each point cloud can be associated with a set of triples, each of which represents the 3D coordinates of a point of the 3D point cloud. Any 3D point cloud herein may represent a 3D real scene, where the real scene is a part of the real world (e.g., a plant scene or an office scene), and each point of the 3D point cloud represents a point of the scene. "Representing a real scene" does not necessarily mean that the 3D point cloud is derived from physical measurements performed in the real world. For example, any 3D point cloud herein may be a synthetic representation of a real scene, as discussed further below. Any 3D point cloud includes at least one object. Each object of a 3D point cloud herein is a set of 3D point cloud points, each of which represents a point of the same object (e.g., of a real scene), and the sets of points collectively form an object representation.

[0039] Any 3D point cloud herein may be a synthetic point cloud, i.e., a point cloud derived from a synthetic point cloud generation process. In other words, a synthetic point cloud representing a real scene does not originate from physical measurements performed on the real scene to capture a representation of the real scene, but rather forms a virtual, numerically generated representation of the real scene. A synthetic point cloud generation process refers to any computer-implemented process that outputs a synthetic 3D point cloud. Such a process may output 3D point clouds, each with point-level annotations that identify, for each point in the output 3D point cloud, whether the point belongs to an object (in the latter case, the point is said to belong to the background). Alternatively, such a process may output an unannotated 3D point cloud, with each point being annotated at a later stage. Alternatively, any 3D point cloud herein may be determined from physical measurements on a real scene. Determining the 3D point cloud may include providing one or more physical sensors (e.g., 3D scans such as LIDARS), each configured to acquire a respective physical signal and acquire the respective one or more physical signals by operating the one or more physical sensors relative to a real scene (i.e., scanning the real scene with each sensor). The determination then automatically determines the 3D point cloud based on the measurements according to any known technique. Note that today, state-of-the-art sensors can provide point clouds consisting of millions of points, providing very high-quality, very dense 3D scans.

[0040] In one example, the 3D point clouds of the training dataset, or at least a portion thereof, may be synthetic 3D point clouds. In such a case, providing (S10) the training dataset may include synthetically generating these 3D point clouds by any known suitable synthetic point cloud generation process. Providing (S10) may further include annotating each point cloud by annotating each point of the point cloud that belongs to an object as "object" and each point that belongs to the background as "background." Alternatively, providing (S10) these 3D point clouds may include retrieving these 3D point clouds, and for example, their annotations, from a (e.g., remote) memory where they were stored after they were generated.

[0041] In one of these examples, generating the 3D point cloud may include providing a computer-aided design (CAD) model of a 3D object. Generating the 3D point cloud may further include converting each provided CAD model of the 3D object into 3D point clouds, each representing the 3D object, which for simplicity are referred to as "object point clouds." The generating may further include providing 3D point clouds, each representing a background, which for simplicity are referred to as "background point clouds." The generating may further include inserting each object point cloud into a background point cloud. The result is a generated 3D point cloud, each formed by one or more object point clouds inserted into the background point cloud. The generating may additionally include annotating each point of each object point cloud as "object" and each point of each background point cloud as "background." Figures 2 and 3 illustrate these examples. Figure 2 illustrates an example of a CAD object 20 being converted into an object point cloud 20. FIG. 3 shows an example 30 of an object point cloud inserted into a background point cloud.

[0042] It should be noted that during object insertion, the method may take into account possible occlusions (object / object occlusion or object / background occlusion) along with point density normalization for a realistic synthetic point cloud representation. Specifically, the method may take these aspects into account in order to insert a point cloud obtained from a CAD model into a scatter plot representing the background. These correspond to adaptation steps to reproduce effects observable in a real point cloud (from a 3D scanner). For example, if a point cloud corresponding to a CAD model of a table is inserted into a background, the method may adapt its density to correspond to the density of the inserted background. Similarly, to correspond to a real point cloud, the method may take occlusions into account. For example, in the case of a table, the method may remove points belonging to the background under the table legs and keep only points that could have been acquired with a real scanner. It should be noted that 3D scanners only capture points on visible surfaces during acquisition. To avoid this, the method, in one example, can add occlusions to the synthetically generated dataset.

[0043] In an alternative to these examples, generating the 3D point cloud may include applying an automatic point cloud generation method that further provides automatic and accurate point-level annotation. The generation method may be any known generation method, such as that described in "A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving" by Xiangyu Yue, Bichen Wu, Sanjit A. Seshia, Kurt Keutzer, and Alberto L. Sangiovanni-Vincentelli (2018), which is incorporated herein by reference. This method proposes a virtual scanner simulation to extract point clouds from 3D CAD scenes (e.g., video game environments).

[0044] In yet another example, the 3D point clouds of the training dataset, or at least a portion thereof, may originate from physical measurements performed on a real scene, as described above. In such a case, providing (S10) the training dataset may include performing physical measurements and determining a 3D point cloud based on the physical measurements, as described above. The providing (S10) may further include annotating each point cloud by annotating each point of the point cloud that belongs to an object as "object" and each point that belongs to the background as "background." Alternatively, providing (S10) these 3D point clouds may include retrieving these 3D point clouds and, for example, their annotations, from a (e.g., remote) memory where they were stored after they were generated. For example, the 3D point clouds of the training dataset may be derived from publicly available point cloud datasets (e.g., Timo Hackel, N. Savinov, L. Ladicky, Jan D. Wegner, K. Schi ndler, and M. Pollefeys, “SEMANTIC3D.NET:A n ew large-scale point cloud classification n benchmark”.-ISPRS2017, and Andreas Geige, P hilip Lenz, Raquel Urtasun’s “Are we ready for Autonomous Driving? The KITTI Vision Be nchmark Suite”-CVPR 2012, Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis. Brilakis, Martin Fischer, Silvio Savarese 3D Semantic Parsing of Large-Scale Indo or Spaces" - Datasets provided to CVPR 2016 (these are all The above-mentioned provision (S10) The binary annotation of point clouds obtained from such public datasets, "Object vs. It may also include a "background."

[0045] In either case, providing a training dataset for 3D point clouds (S10) may include providing the 3D point clouds. As described above, each 3D point cloud is provided with its specifications. The specifications form data about one or more graphical user interactions, each representing a selection of the same object in the 3D point cloud. This means that the specifications describe how the same object is selected by one or more graphical user interactions, for example, based on data about the relative positions of the objects and the one or more graphical user interactions. This allows the neural network to be trained to segment the 3D point cloud based solely on detecting object selections by graphical user interactions. Providing a training dataset (S10) may include determining specifications for each point cloud and providing the point cloud with them. Determining the specifications may include calculating / simulating the specifications for each 3D point cloud of the dataset. Alternatively, determining the specification may involve providing the specification by hand, e.g., by a user manually performing one or more graphical user interactions, i.e., for each 3D point cloud of the dataset.

[0046] As mentioned above, this specification is also referred to as an "interaction channel" or "user indicator." Specifically, each point in a 3D point cloud may be represented by three Cartesian coordinates in 3D space. The point may further include a fourth coordinate that represents how the point is affected by one or more graphical user interactions. In other words, the fourth coordinate forms data encapsulating the degree of influence of the point by selection of the same object via a graphical user interaction. The data may, for example, represent the proximity of the point to at least one of the one or more graphical user interactions.

[0047] Here, an example of providing a training data set (S10) will be described.

[0048] In one example, providing (S10) a training dataset for 3D point clouds includes providing the 3D point clouds. Providing (S10) a training dataset further includes, for each 3D point cloud, providing information regarding the location of the same object in the 3D point cloud. Providing (S10) a training dataset further includes, for each 3D point cloud: determining specifications by simulating each of one or more graphical user interactions; and providing the 3D point cloud with the specifications of the one or more graphical user interactions.

[0049] The information about the location of the same object may comprise any information that allows distinguishing the same object from other parts of the 3D point cloud (e.g., other objects, if any, or the background). In one example, the information includes, for each point of the object, annotating the point as "object." The information may also include the coordinates of the point. Providing the information may include annotating each point of the same object as "object," e.g., as described above. Providing the information may further include annotating each point of the background as "background," e.g., as described above. Providing the information may include annotating each point of other objects, if any, as "other objects." This may be used by the method, for example, to add negative seeds to objects adjacent to the target object, as further described below. However, this is not required. In fact, simply adding seeds to the area around the target object will also result in seeds being found in adjacent objects (if any).

[0050] Simulating each of the graphical user interactions provides data on the location of the graphical user interactions. Note that "simulating" means that the graphical user interactions are virtual, i.e., generated numerically. In other words, the learning method provides a 3D point cloud with specifications of object selections resulting from simulated user behavior. In other words, the location data is generated numerically, e.g., relatively randomly and / or according to a predetermined program / strategy. The specification is determined based on this data and information about the location of the same objects. For example, the specification may be generated by taking these data and this information into account. By simulating each of the one or more graphical user interactions, the specification can be obtained without manual user effort, which can be time-consuming and / or prone to overfitting. Furthermore, the graphical user interactions are virtualized, e.g., relatively randomly and / or sufficiently diverse, resulting in a sufficiently diverse training dataset. This improves the quality of learning, as known per se in the field of machine learning.

[0051] In one example, each of the one or more graphical user interactions corresponds to one or more seed locations defined on the 3D point cloud. In one of these examples, simulating the graphical user interactions includes determining the one or more seed locations.

[0052] Each seed location is data representing the location of at least a portion of a graphical user interaction. The one or more seed locations collectively form data representing the location of the graphical user interaction on the 3D point cloud. Each seed location may, for example, consist of a set of three Cartesian coordinates in 3D space, indicating the location of a point of the graphical user interaction. In other words, a seed location may consist of a point in 3D space, which may or may not coincide with a point in the 3D point cloud. For example, a seed location may be a point in the 3D point cloud that is closest to the location represented by the seed of at least a portion of the graphical user interaction. This allows the simulated graphical user interaction to be a virtual representation of various types of different and typical graphical user interactions for object selection. In fact, many types of well-known and typical graphical user interactions can be represented by a set of one or more seed locations. For example, the one or more seed locations may represent one or more clicks by a user on the 3D point cloud, with each seed location representing (e.g., being) a point in 3D space corresponding to the location of one of the clicks. A seed location may be, for example, a point in the 3D point cloud that is closest to the location of the click. Alternatively, the one or more seed locations may represent a stroke by a user on the 3D point cloud, with each seed location representing (e.g., being) a point in 3D space corresponding to a sampled location in the line of the stroke. Alternatively, the one or more seed locations may represent a bounding box drawn / defined by the user on the 3D point cloud, with each seed location representing (e.g., being) a point in 3D space corresponding to a sampled location inside or outside the bounding box.Since all this is true for each graphical user interaction, this means that one or more graphical user interactions can be a virtual representation of one or more of the following: one or more clicks on a 3D point cloud, a stroke on a 3D point cloud, and / or a bounding box drawn / defined on a 3D point cloud. In the following, for brevity, we will also refer to each seed location simply as a "seed".

[0053] Determining the one or more seeds may include calculating the one or more seeds, e.g., relatively randomly and / or according to a predetermined program / strategy, e.g., to ensure a predetermined number of seeds and / or reasonable distances between them. If the one or more seeds are one or more points in 3D space, determining the one or more seeds may include determining the coordinates of the one or more points.

[0054] In one example, the one or more graphical user interactions include a first graphical user interaction for selecting the same object, each of the first graphical user interactions corresponding to one or more first seed locations defined on the same object.

[0055] The first graphical user interaction is also referred to as a "positive interaction" because it involves the selection of the same object. In other words, a positive interaction represents a graphical user selection of the same object, for example, by actually selecting it rather than discarding other parts of the 3D point cloud. Providing data on positive interactions in the training dataset allows the neural network to be trained to detect such interactions to perform segmentation. Examples of positive interactions, for example, a first graphical user interaction, may represent performing one or more clicks on the same object, performing a stroke on the same object, or drawing / defining a bounding box on and / or around the same object.

[0056] Since the first graphical user interaction is a positive user interaction, it is a virtual representation of a graphical user interaction performed on the same object, i.e., on a point of the same object. One or more first seed locations are then defined on the same object. For example, if the one or more first seed locations are one or more points, the one or more points are substantially located within the points of the same object. The simulation of the first graphical user interaction includes determining one or more first seeds. The determination of the one or more first seeds may be performed randomly, for example, by randomly or partially randomly selecting one or more points of the same object as the one or more first seeds. This facilitates the creation of a training dataset.

[0057] In one example, determining one or more seed locations includes spreading one or more first seed locations over the same object. Spreading the one or more first seeds may include determining each seed by selecting, for example, randomly, a point of the point cloud from a limited set of points in the point cloud as a seed. Spreading the one or more first seeds may include determining a number of first seeds and selecting the number of first seeds with a predetermined distance between them. The number and / or the predetermined distance may depend, for example, on the size of the object. This allows the first seeds to sufficiently cover the same object.

[0058] Next, the implementation of this diffusion will be described.

[0059] This implementation can simulate user behavior for positive interactions. Compared to a completely random approach, the simulated first graphical user interaction here corresponds to a likely positive user interaction, where the user's interaction generates a certain number of seeds with a reasonable distance between them relative to the size of the same object. This ensures sufficient coverage of the object. For example, when selecting an object by clicking, the user typically covers the object with the click. When selecting an object by performing a stroke on it, the user typically draws a stroke that covers the object. When selecting an object with a bounding box, the user typically may draw the bounding box so that the inside of the bounding box covers the object, for example, by framing the object within the bounding box. This implementation includes determining the number of one or more first seeds according to a type of normal probability distribution.

number

[0060] A specific implementation of the currently described diffusion implementation will now be described. In the specific implementation, the objects are called "target objects" and each seed is called a "positive seed." The specific implementation is performed according to an algorithm having the following specifications:

[0061] The input variables are as follows:

[0062] objPoints: A variable containing all the points of the target object.

[0063] K: A variable that is the number of neighboring objects to consider in this point cloud. The number K is explained further below.

[0064] proportion: A parameter that controls the proportion of the object affected by the first graphical interaction.

[0065] standardDeviation: A parameter that controls the variance of the distribution used to describe the number of positive seeds.

[0066] The output variables are:

[0067] seeds: A variable that contains a list of all positive clicks (coordinates) on the target object.

[0068] The intermediate variables that appear are:

[0069] numberOfObjPoints: A variable corresponding to the number of points belonging to the target object.

[0070] numberOfSeeds: A variable corresponding to the number of positive clicks performed on this target object.

[0071] seed: A variable corresponding to the coordinates of one positive click.

[0072] Functions that appear without an explicit declaration are:

[0073] RandPoint(): A function that returns a random point from a set of points.

[0074] RegionAround(): A function that returns the region around the coordinates of a point. Here, it is used to suppress all points of an object that are in the region around the previous click, minimizing the distance between clicks.

[0075] NormalDistribution(mean,standardDeviation): A function that returns a floating-point value according to the normal distribution defined by the input parameters.

[0076] The algorithm is described in the pseudocode below.

number

[0077] The second type of graphical user interaction is also called a "negative interaction" because it involves selecting the same object by discarding areas outside the object. In other words, a negative interaction represents a graphical user selection of the same object by discarding parts of the point cloud that are not the same object. By providing data on negative interactions in addition to positive interactions in the training dataset, we can train a neural network to detect both positive and negative interactions for segmentation. This improves the neural network's ability to perform segmentation because a user's graphical selection of an object in a point cloud often involves a mixture of positive interactions (e.g., a rough selection of the object) and negative interactions (e.g., a subsequent refinement of the selection by discarding areas outside the object, such as the background or other objects). Examples of such negative interactions, e.g., a second graphical user interaction, can represent one or more clicks outside the same object (e.g., on the background or another object) or a stroke outside the same object (e.g., on the background or another object). Furthermore, the ability of the learning method to mix positive and negative interactions allows the neural network to be trained to detect graphical selection of an object consisting of a user drawing / defining a bounding box around the object: points (or at least a sample of them) outside the bounding box are detected by the neural network as second seeds, and points (or at least a sample of them) inside the bounding box are detected as first seeds.

[0078] The mixture of positive and negative seeds allows each graphical user interaction to represent a typical interaction, such as a click, a stroke, or a bounding box. All of these graphical interactions share the property of being represented by a seed, as previously described. Specifically, because clicks correspond directly to seeds, they represent the smallest possible interaction. Strokes can be sampled into a set of seeds (e.g., a set of individual points) by sampling the seed locations from the stroke line, resulting in clicks that are very close to the desired generation. Assuming a bounding box is used to roughly represent the target object, the bounding box can also be converted into a set of seeds by sampling individual point indicators inside and outside the box for positive and negative clicks, respectively. However, in some cases, our learning method may deviate from these paradigms to determine seeds that best correspond to a particular type of user interaction, i.e., for applications where the input graphical user interactions are always of the same type, such as bounding boxes. For example, the learning method may perform optimization to generate seeds from bounding boxes by sampling seeds from inside and outside training boxes with different heights, lengths, and widths for each interaction on the same object. Furthermore, some training bounding boxes may partially contain the same object. In this way, the neural network can identify an object in a point cloud even if the user provides a bounding box that only partially surrounds the object.

[0079] Since the second graphical user interaction is a negative user interaction, this is a virtual representation of a graphical user interaction performed outside the same object, i.e., on a point outside the same object. One or more second seed locations are thereby defined, each outside the same object. For example, if the one or more second seed locations are one or more points, the one or more points are substantially positioned among the points outside the same object. The simulation of the second graphical user interaction includes determining one or more second seeds. The determination of the one or more second seeds may be performed randomly, for example, by randomly or partially randomly selecting one or more points outside the same object (i.e., one or more points of the background or another object) as the one or more second seeds.

[0080] An implementation of determining one or more second seeds will now be described.

[0081] In this implementation, the determination of one or more second seeds may be performed according to different strategies, each of which can simulate typical user behavior and match real-world interaction types.

[0082] According to a first strategy, determining one or more second seeds includes determining one or more second seeds around the same object (e.g., within a predetermined distance). This strategy simulates the method a user is most likely to use to identify the boundary of an object. Determining one or more second seeds according to this strategy may include extracting a nearby region of background points around the target object (e.g., a region less than 30 centimeters in size, i.e., a narrow region around the object with a width of 30 centimeters), and selecting one or more random points within the region as one or more second seeds.

[0083] According to a second strategy, determining one or more second seeds around the same object (e.g., within a predetermined distance) includes determining one or more second seeds on objects adjacent to the same object (e.g., objects within a predetermined distance from the same object). This strategy simulates the classification of an object as a background object and is consistent with real-world user behavior. Determining one or more seeds according to the second strategy requires annotating all objects in the point cloud scene. Determining one or more second seeds according to this strategy may include randomly selecting points on each background object from those closest to the same object, where the selected points are the one or more second seeds.

[0084] According to a third strategy, the determination of the one or more second seeds may involve randomly determining the one or more second seeds (e.g., even if they are far away from the object of interest) to suppress random false first seeds and to be consistent with the behavior of other users. This strategy may include randomly selecting one or more background points (i.e., all points in the 3D point cloud minus points of the same object) as the one or more second seeds.

[0085] For each 3D point cloud in the training dataset, there is at least one positive interaction (i.e., a first graphical user interaction) among one or more graphical user interactions, but not necessarily one negative interaction (i.e., a second graphical user interaction). In one example, there are one or more 3D point clouds in the dataset, and one or more graphical user interactions include the second graphical user interaction. The training method may determine one or more second seeds for each of these second graphical user interactions, each having a given strategy, selected relatively randomly from among the first, second, and third strategies. "Relatively randomly" here means that some strategies may be more likely to be selected than others. For example, the first strategy may be most likely to be selected because it simulates user behavior that might occur in the real world. The second strategy may be optional. That is, in the present example training method, if a background object is not so annotated, it will not be selected.

[0086] Regardless of the strategy, the number of second seeds determined for the first graphical user interaction may be derived from a probability distribution centered on a target number of second seeds. This target number of second seeds typically controls not only the amount of simulated interaction but also the ease of the task (i.e., segmentation) that the neural network is trained to perform. For example, a target number of 0 ensures that positive interactions provide enough information to extract the object, but may destabilize negative clicks and limit the quality of the extraction. On the other hand, a large target number (e.g., 15) ensures an accurate segmentation mask but requires a large number of user interactions during the neural network's use. A corresponding implementation involves determining the number of one or more second seeds according to a type of normal probability distribution.

number

[0087] A specific implementation of the presently described determination of one or more second seeds will now be described. In the specific implementation, the objects are referred to as "target objects" and each seed is referred to as a "negative seed." The specific implementation is performed according to an algorithm having the following specifications:

[0088] The input variables are as follows:

[0089] objPoints: A variable containing all the points of the target object.

[0090] scenePoints: A variable containing all points in the scene (including the target object).

[0091] mean: A parameter that controls the mean of the distribution used to describe several negative seeds.

[0092] standardDeviation: A parameter that controls the variance of the distribution used to describe some negative seeds.

[0093] pStrategy: A parameter that controls the probability of choosing one strategy over another.

[0094] The output variables are:

[0095] seeds: A variable that contains a list of all negative clicks (coordinates) on the target object.

[0096] The intermediate variables that appear are:

[0097] backgroundPoints: A variable containing all points from the point cloud scene minus points belonging to the object of interest.

[0098] strategy: the variable corresponding to the chosen strategy. This can be:

[0099] random (a random point from the background points).

[0100] neighborObjs (clicks the object closest to the target object).

[0101] surroundings (click a small area around the target object).

[0102] numberOfSeeds: A variable corresponding to the number of negative clicks performed on this target object.

[0103] seed: A variable corresponding to the coordinates of one negative click.

[0104] nearestObjs: A variable that is a list of points of all neighboring objects, of length numberOfSeeds.

[0105] Functions that appear without an explicit declaration are:

[0106] RandPoint(): A function that returns a random point from a set of points.

[0107] RegionAround(): A function that returns the region around the coordinates of a point. In our case, this is used to suppress all points in the background that lie in the region around the previous click, minimizing the distance between clicks.

[0108] NearestObjs(): A function that returns a list of points of all nearby objects, of length numberOfSeeds.

[0109] SmallerRegionAroundObj(): A function that returns the proximity region of background points around the target object.

[0110] NormalDistribution(mean,standardDeviation): A function that returns a floating-point value according to the normal distribution defined by the input parameters.

[0111] The algorithm is described in the pseudocode below.

number

[0112] if the graphical user interaction is a positive interaction (i.e., at least one of the one or more graphical user interactions is a positive interaction), the one or more first seeds have been determined as described above; If the graphical user interaction is a negative interaction, one or more second seeds are determined as described above.

[0113] Determining the specification may then include processing data regarding the positions of all simulated graphical user interactions and converting them into a piece of data that forms the specification. Equipping the 3D point cloud with the specification may then include associating the specification with the 3D point cloud to form pairs, which form the training samples.

[0114] Here, an example of the preparation of this learning method will be described.

[0115] In one example, providing the 3D point cloud with the specifications includes adding a coordinate to each point of the 3D point cloud based on each simulation of a graphical user interaction, the coordinate quantifying the intensity of a selection operation at that point.

[0116] "Based on each simulation of a graphical user interaction" means that furnishing the 3D point cloud with the specification takes into account data relating to the location of each simulated graphical user interaction. In fact, furnishing the 3D point cloud with the specification now involves processing this data and converting it into a set of coordinates, one for each point in the 3D point cloud. Furnishing the 3D point cloud with the specification then involves adding this coordinate to each point, thereby obtaining the specification. In other words, the specification is the set of all coordinates added to each point in the 3D point cloud.

[0117] The transformation is performed such that for each point, the coordinates quantify the strength of the selection operation at that point. Specifically, the coordinates quantify the proximity of the point to each graphical user interaction, thereby quantifying whether the point belongs to the object to be selected (i.e., the same object). This ultimately allows the neural network to learn whether to select the point as part of an object when it performs segmentation in the learning (S20) step. For example, the neural network is trained to detect that the larger the coordinate, the higher the probability that the point belongs to the object to be selected.

[0118] In one example, each of the one or more graphical user interactions is for selecting the same object (e.g., the first graphical user interaction described above) or discarding an area outside the same object (e.g., the second graphical user interaction described above). In one of these examples, for each point in the 3D point cloud, adding coordinates includes: Setting the coordinates to initial values; for each graphical user interaction for selecting the same object, incrementing the coordinate according to the proximity between the graphical user interaction and the point; and for each graphical user interaction to discard areas outside the same object, decreasing the coordinate according to the proximity between the graphical user interaction and the point.

[0119] Increasing a coordinate according to the proximity between the graphical user interaction and the point may include quantifying a distance between a location of the point and a location of the graphical user interaction. For example, the graphical user interaction may be the first graphical user interaction defined by one or more first seeds. Quantifying the distance may include quantifying each distance between the point and each first seed. Increasing the coordinate may then include adding a value representing each quantified distance to the coordinate.

[0120] Reducing the coordinates according to the proximity between the graphical user interaction and the point may include quantifying the distance between the location of the point and the location of the graphical user interaction. For example, the graphical user interaction may be the second graphical user interaction defined by one or more second seeds. Quantifying the distance may include quantifying each distance between the point and each second seed. Increasing the coordinates may then include adding a value representing each quantified distance to the coordinates.

[0121] Note that for any point in the 3D point cloud, if that point is too far from a graphical user interaction, whether it is a graphical user interaction for selecting the same object or for discarding the area outside it, the coordinates of the point are not incremented or decremented. In other words, for each point and each of one or more graphical user interactions, the increment or decrement of coordinates occurs only if the point is relatively close to the graphical user interaction (e.g., within a certain distance from the seed defining that graphical user interaction). Therefore, it is possible for the coordinates of a point to remain equal to their initial values ​​after specification.

[0122] Adding coordinates in this manner has the following effect: the closer a point is to one or more graphical user interactions to select the same object, the more likely the point belongs to the same object, and the larger the added coordinate value. Conversely, the closer a point is to one or more graphical user interactions to discard an area outside the same object, the more likely the point belongs to that area, and the smaller the added coordinate value. Ultimately, this helps train the neural network to detect when graphical user interactions are to select an object and when they are to discard an area outside the object.

[0123] Here, we explain the implementation of this learning method by providing specifications to 3D point clouds.

[0124] In this implementation, for each 3D point cloud, the one or more graphical user interactions include one or more first graphical user interactions, and for each one of them, the determination of one or more seed locations includes spreading one or more first seeds according to the implementation of spreading described above. In the currently described implementation, the one or more graphical user interactions may also include one or more second graphical user interactions, and in that case, for each one of them, the determination of one or more second seeds is performed according to the implementation of determining one or more second seeds described above. Thus, in the currently described implementation, simulating one or more graphical user interactions results in the following:

[0125] For each first graphical user interaction (i.e., a positive interaction), one or more first seed locations.

[0126] For each second graphical user interaction (i.e., a negative interaction), one or more second seed locations.

[0127] Furthermore, in this implementation, as mentioned above, each seed is a point. This implementation involves converting the seeds into specifications, which we refer to here as interaction channels. The conversion uses a K-nearest neighbor search to obtain the K points closest to each seed and assigns them a specific strength based on their distance from each seed.

[0128] Specifically, the coordinate addition includes initializing the coordinates for all points in the 3D point cloud with the same value, which is equal to 128. The coordinate addition then includes, for each seed, obtaining the K points closest to the seed, for example according to a nearest neighbor search algorithm, where the number K is the number K described above in connection with equation (1). The obtaining may include adjusting the number of neighbors used in the K-nearest neighbor search according to the number of points in the 3D point cloud, thereby making the search invariant to the point cloud density. The adjustment may be performed according to the following equation:

number

[0129] Experiments showed that this formula for K was very stable for a variety of point cloud sizes (we tested a range of approximately 10k to 500k points for a 3x3x5 meter area with D=100,000, k=1024, and m=128). These numbers were chosen experimentally to correspond to a graphical interaction size that was globally relevant for the smallest object sizes, while being large enough to reduce the number of interactions for large objects. Then, for simplicity's sake, we chose the number of object points observed per object in the training dataset and the density of the point cloud. This allowed us to experiment and find a way to determine the optimal K that would fit all objects and point cloud densities.

[0130] Adding coordinates then involves modifying, for each seed and for each point in the 3D point cloud that belongs to the seed's K neighborhood, the coordinates by:

[0131] If the seed is the first seed, add the Gaussian intensity from the seed to the coordinate, or If the seed is a second seed, the Gaussian intensity from the seed is subtracted from the coordinate.

[0132] In practice, the modification involves calculating, for each seed, a Gaussian intensity and dynamically adjusting the Gaussian intensity based on the median distance of the K distances of the K closest points to the seed. The calculation of the Gaussian intensity of the first seed, also called the positive seed, may be performed according to the following formula:

number

number

number

[0133] The calculation of the Gaussian intensity of the second seed, also called the negative seed, may be performed according to the following formula:

number

number

number

[0134] Note that the added coordinates are initialized to 128 for all points. Multiplying the Gaussian intensity by 127 yields a maximum value of 255 for points matching a positive seed. For points matching a negative seed, the intensity of the corresponding final point is 1. Thus, the intensity covers the same amplitude as the color channels, allowing for a simpler normalization step (normalizing each channel's 0-1 by 255) if the point cloud has separate color channels. The dynamic s parameter as the median of the array of neighboring distances provides flexibility for handling various point cloud densities. Contributions from all points matching a seed are added, and the above fitting involves clipping coordinates in the [0, 255] range, concatenating the point to other coordinates, and so on for each point. The interaction channel / specification is the set of all concatenated coordinates. For example, for a 3D point cloud consisting of N points with coordinates x, y, and z as the only features, the fitting results in a final point cloud dimension of (N, 3 + 1). In the presence of color, the above arrangement results in, for example, (N, 6+1) dimensions. Experiments performed using only point coordinates, i.e., no color, showed excellent results.

[0135] As a result, in this implementation, the training data set consists of training samples, each consisting of a pair of point clouds and their interaction channels. Each training sample may further include, as known per se from the field of point cloud segmentation, a segmentation mask indicating for each point of the 3D point cloud whether the point belongs to said same object. The segmentation mask may be created in the step of annotating the 3D point cloud, as described above.

[0136] Figure 4 shows a point cloud 40 and its interaction channels, particularly showing a positive seed 42 and two negative seeds 44 and 46. Figure 5 shows the corresponding segmentation mask 50, indicating the portion of the point cloud 40 that corresponds to the same object 52 and the portion that does not correspond to the same object 54.

[0137] Here, the neural network training (S20) will be explained.

[0138] As described above, a neural network has weight formation data that defines the operations that the neural network applies to process inputs. Training the neural network (S20) includes determining values ​​for the weights based on a training data set. Training the neural network (S20) can be performed according to any known technique. In an example, training the neural network (S20) is performed according to any known deep neural network (DNN) training method.

[0139] DNNs, as they are known in the field of machine learning, are a collection of techniques for learning neural networks, a biologically inspired programming paradigm that enables computers to learn from observed data. See, for example, E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning internal representations by error propagation," in Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986, which is incorporated herein by reference. In object recognition, the success of DNNs owes their ability to learn rich mid-level media representations, as opposed to the hand-crafted low-level features (e.g., Zernike moments, HOG, Bag-of-Words, SIFT) used in other methods (e.g., min-cut, SVM, Boosting, Random Forest). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move as far away from feature engineering as possible by achieving end-to-end optimization starting from raw features and ending with labels.

[0140] The neural network herein may be any known deep learning model capable of performing point cloud segmentation. For example, the neural network may be PointCNN (see [4] Li, Yangyan, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. “PointCNN: Convolution on X-Transformed Points.” - NeurIPS 2018, incorporated herein by reference). A neural network may contain millions of parameters whose values ​​cannot be set manually. Learning (S20) involves setting / updating the values ​​of these parameters. When the learning algorithm is updating the model parameters, the model is said to be in “training mode.” This involves continuously “correcting” the model according to the model’s output for each input, with annotations associated with each input. Annotations are a collection of data associated with specific inputs that allow the model’s output to be evaluated as true or false. The method of supervising model training with annotated datasets is called “supervised learning.” Once the model is trained, its parameters are stopped from being updated. The model is then only used to process new inputs (i.e., inputs not seen during training mode) and return detection results; such a model is said to be in "test mode." Training a neural network (S20) may integrate all these deep learning concepts that are well known in the field of machine learning.

[0141] The neural network is configured to segment the input 3D point cloud containing objects based on the input 3D point cloud and specifications of one or more input graphical user interactions, each representing a selection operation. In other words, the neural network receives as input the input 3D point cloud with the specifications. In yet another way, the neural network receives as input data of the same type as any sample in the training dataset. The neural network is configured to segment the input 3D point cloud. In other words, the neural network performs object extraction based on the input. The neural network is able to do this because the input 3D point cloud is provided with specifications, and the neural network is trained (S20) to detect such specifications. In other words, the specifications indicate the locations of one or more input graphical user interactions for selecting the object, and the neural network detects that an object is selected from the specifications. Note that, as mentioned above, the neural network is independent of point cloud category and object category. The neural network outputs data, e.g., a segmentation mask, that indicates the difference between the object and the rest of the background.

[0142] For example, the neural network may output a classification result for each point in the input point cloud, along with a corresponding confidence score. Specifically, in such examples, for each point, the neural network outputs a classification of whether the point is part of an object, along with a confidence score. The confidence score is a real number between 0 and 1. The closer the score is to 1, the more confident the neural network is in the label associated with that point. In other words, the neural network outputs a segmentation mask for the object. In these examples, each training sample also includes a segmentation mask, which classifies each point in the point cloud as either an "object" or "background," as described above. In these examples, the learning (S20) is fully supervised.

[0143] The segmentation method is now described.

[0144] The segmentation method is a method for interactive point cloud segmentation.

[0145] Thus, the segmentation method includes providing (S30) a 3D point cloud including the object. Like other 3D point clouds in a training dataset, the 3D point cloud here may represent a real scene or may result from physical measurements or a synthetic point cloud generation process. Providing (S30) may include displaying the 3D point cloud on a display (e.g., a graphical user interface) of a computer system executing the segmentation method. The 3D point cloud may be displayed to a user, allowing the user to graphically interact with the 3D point cloud, e.g., to indicate interest in an object.

[0146] The segmentation method also includes one or more iterations corresponding to interactive segmentation of the provided 3D point cloud. Specifically, each iteration includes performing (S40) an object selection operation by performing one or more graphical user interactions. In other words, a user can indicate interest in an object by graphically interacting with the provided (e.g., displayed) 3D point cloud one or more times. Each graphical user interaction relates to the selection of an object, and the one or more graphical user interactions collectively form an object selection operation.

[0147] With respect to selecting an object, a "graphical user interaction" means a user interaction in which a user indicates interest in an object, for example by selecting one or more locations on a display unit using a tactile system (e.g., a mouse or a touch device such as a sensitive / touchscreen or sensitive / touchpad). The one or more locations may form distinct locations, such as one or more clicks. Alternatively, they may collectively form a continuous shape, such as a stroke or a bounding box. The one or more locations may be on the object, in which case the graphical user interaction is to select the object. Alternatively, the one or more locations may be on an area outside the object (e.g., another object or the background), in which case the graphical user interaction is to discard said area. After activation, the selection may be visually indicated substantially in real time.

[0148] In one example, the one or more graphical user interactions may include a graphical user interaction consisting of performing one or more clicks on the object. Performing the one or more clicks may be performed using a haptic device, for example, a user clicking or touching the displayed 3D point cloud one or more times at one or more locations on the object. The graphical user interaction involves selecting one or more locations on the object (i.e., the locations of the one or more clicks), and therefore is intended to select the object.

[0149] Additionally or alternatively, the one or more graphical user interactions may include a graphical user interaction consisting of performing a stroke on the object. The stroke may consist of the user drawing a stroke on the displayed 3D point cloud by successively moving a cursor along a line on the object, for example with a haptic device, or by successively performing touches on the object. This graphical user interaction involves selecting a position on the object (i.e., a position defined by a stroke), and therefore is for selecting the object.

[0150] Additionally or alternatively, the one or more graphical user interactions may include a graphical user interaction consisting of defining a bounding box on or around the object. The bounding box definition may consist of a user drawing a bounding box on the displayed 3D point cloud, for example, by touching a haptic device or by continuously moving a cursor to form the bounding box. This graphical user interaction is for selecting the object, as it involves selecting a location on the object (i.e., a location inside the bounding box).

[0151] Additionally or alternatively, the one or more graphical user interactions may include a graphical user interaction consisting of performing one or more clicks outside the object. The one or more clicks may be performed using a haptic device, for example, a user clicking or touching the displayed 3D point cloud one or more times at one or more locations outside the object. This graphical user interaction involves selecting one or more locations on the area outside the object (i.e., the locations of the one or more clicks) to discard the area outside the object. The one or more locations may, for example, be on an adjacent area outside the object to identify the object's boundary, or on an adjacent object to identify it as a background object.

[0152] Additionally or alternatively, the one or more graphical user interactions may include a graphical user interaction consisting of performing a stroke outside the object. The stroke may consist of the user drawing a stroke on the displayed 3D point cloud, for example, by using a haptic device to continuously move a cursor along a line on the 3D point cloud or by continuously performing touches on the 3D point cloud. This graphical user interaction involves selecting a position on the area outside the object (i.e., a position defined by a stroke) to discard the area outside the object. The position may, for example, be on an adjacent area outside the object, identifying the object's boundary, or on an adjacent object, identifying it as a background object.

[0153] The one or more graphical user interactions complement each other: for example, one to roughly select an object (e.g., a single click on the object) and another to refine the object selection by discarding neighboring regions (e.g., clicks on neighboring regions of the object). The one or more graphical user interactions collectively form an object selection operation. The selection operation is specified by data encapsulating the locations defined by the one or more graphical user interactions. In other words, these data form a specification of one or more graphical user interactions. The specification is the same type of data as the specification provided in the training dataset, as described above. The main difference is that the specification in the context of the segmentation method comes from real graphical user interactions, whereas in the context of the learning method, the specification may come from simulated virtual graphical user interactions, as described above.

[0154] The segmentation method then includes segmenting the 3D point cloud based on the specifications and the input 3D point cloud by applying (S50) a neural network. As described above, the neural network takes the specifications and the 3D point cloud as input and performs operations on these inputs, resulting in the neural network extracting objects from the 3D point cloud. For example, the neural network may output classifications of points in the 3D point cloud as "objects" or "non-objects," along with confidence scores, as described above.

[0155] At each iteration, the segmentation method may include, after performing the selection operations and before applying the neural network, converting the selection operations into interaction channels, which in this example of the training method are sets of coordinates respectively added to each point of the 3D point cloud. Converting the selection operations into interaction channels involves: determining, for each of the one or more graphical user interactions, one or more seed locations defined by the graphical user interaction, thereby determining the specification; and providing the 3D point cloud with the specification, the provision including adding to each point of the 3D point cloud a coordinate that quantifies the strength of the selection operation at that point.

[0156] Each graphical user interaction defines a geometry on the 3D point cloud (e.g., one or more clicks, strokes, or a bounding box) and generates data about the location of this geometry. These data may be detected by any known method for detecting the location of graphical user interactions. From these data, segmentation determines one or more seed locations that represent the geometry as a whole. For simplicity, each seed location is referred to herein as a "seed." Determining the one or more seeds may include calculating one or more points characteristic of the geometry as the one or more seeds, meaning that each seed is a point in this case. Seeds on an object are called positive seeds, and seeds on areas outside the object are called negative seeds. A graphical user interaction may define both positive and negative seeds.

[0157] In one example, if the graphical user interaction consists of performing one or more clicks, determining one or more seeds may include determining each click as a seed, where a click on the object is determined as a positive seed, while a click on an area outside the object is determined as a negative seed.

[0158] Additionally or alternatively, if the graphical user interaction consists of performing a stroke, determining one or more seeds may include sampling the stroke to several points by taking points below the stroke line. The seeds are then the sampled points. If a sampled point is on the object, it is a positive seed; if it is on an area outside the object, it is a negative seed. The number of sampled points may be proportional to the length of the stroke. Note that this is consistent with the graphical user interaction simulated in the example of providing a training dataset (S10), where, for positive seeds, the larger the object, the larger the strokes performed on the object and the more seeds are determined.

[0159] Additionally or alternatively, if the graphical user interaction consists of defining a bounding box, determining one or more seeds may include sampling the bounding box to some points inside the bounding box and some points outside the bounding box. The inside points are determined as positive seeds, and the outside points are determined as negative seeds. The number of positive seeds may be proportional to the number of points within the bounding box and may be selected randomly among them. Negative seeds may be selected randomly from points within a small region (e.g., less than 30 cm) around the bounding box. Note that this is consistent with the graphical user interaction simulated according to the first strategy described in this learning method.

[0160] Continuing with the segmentation method, providing the 3D point cloud with the specifications includes adding to each point of the 3D point cloud a coordinate that quantifies the strength of the selection operation at that point.

[0161] Here, the segmentation method will be described to have the above specifications.

[0162] Equipping the 3D point cloud with a specification takes the determined seeds and converts them into a set of coordinates. Each point in the 3D point cloud is converted into one coordinate. Equipping the 3D point cloud then involves adding this coordinate to each point, resulting in the specification. In other words, the specification is the set of all coordinates added to each point in the 3D point cloud.

[0163] The transformation is performed such that for each point, the coordinates quantify the strength of the selection operation at that point. Specifically, the coordinates quantify the proximity of the point to each graphical user interaction, thereby quantifying whether the point belongs to the object to be selected (i.e., the same object). This allows the neural network to extract / select the point as part of the object when it performs segmentation in application (S50). For example, the neural network may detect that the larger the coordinate, the higher the probability that the point belongs to the object to be selected.

[0164] In one example, the adding includes setting coordinates for all points to the same initial value. The adding may then include, for each determined positive seed, increasing the coordinate for each point in the point cloud according to the proximity between the seed and the point. The adding may also include, for each determined negative seed, decreasing the coordinate for each point in the point cloud according to the proximity between the seed and the point.

[0165] Increasing (or decreasing) a coordinate in response to the proximity between a positive (or negative) seed and a point may include quantifying the distance between the location of the point and the location of the seed, and then adding (or subtracting) a value representing the quantified distance to the coordinate.

[0166] Note that for any point in the 3D point cloud, if that point is too far from the seed, whether it is a positive or negative seed, the coordinates of the point will not be incremented or decremented. In other words, for each point and determined seed, incrementing or decrementing of coordinates will only occur if the point is relatively close to the graphical user interaction (e.g., within a predetermined distance from the seed). Therefore, it is possible for the coordinates of a point to remain equal to their initial values ​​even after specification.

[0167] Adding coordinates in this way has the following effect: the closer a point is to a positive seed, the more likely it is that it belongs to the same object, and the larger the value of its added coordinate will be. Conversely, the closer a point is to a negative seed, the more likely it is that it does not belong to the same object, and the smaller the value of its added coordinate will be. Ultimately, this allows the neural network to detect when graphical user interactions are intended to select an object and discard areas outside of the object.

[0168] Here we describe the implementation of having a 3D point cloud with specifications.

[0169] In this implementation, determining a specification results in one or more positive seeds and one or more negative seeds. Furthermore, in this implementation, as previously described, each seed is a point. This implementation involves converting the seeds to specifications, which we refer to as interaction channels. The conversion uses a K-nearest neighbor search to obtain the K points closest to each seed and assigns them a specific strength based on their distance from each seed.

[0170] Specifically, the coordinate addition includes initializing the coordinates for all points in the 3D point cloud with the same value, which is equal to 128. The coordinate addition then includes, for each seed, obtaining the K points closest to the seed, for example according to a nearest neighbor search algorithm, where the number K is the number K described above in connection with equation (1). The obtaining may include adjusting the number of neighbors used in the K nearest neighbor search according to the number of points in the 3D point cloud, thereby making the search invariant to point cloud density. The adjustment may be performed according to the following equation:

number

[0171] Experiments have shown that this formula for K is very stable for a variety of point cloud sizes (we tested a range of approximately 10k to 500k points for a 3*3*5 meter area with D=100000, k=1024, m=128).

[0172] Adding coordinates then involves modifying, for each seed and for each point in the 3D point cloud that belongs to the seed's K neighborhood, the coordinates by:

[0173] If the seed is the first seed, add the Gaussian intensity from the seed to the coordinate, or If the seed is a second seed, the Gaussian intensity from the seed is subtracted from the coordinate.

[0174] In practice, the modification involves calculating, for each seed, a Gaussian intensity and dynamically adjusting the Gaussian intensity based on the median distance of the K distances of the K closest points to the seed. The calculation of the Gaussian intensity of the first seed, also called the positive seed, may be performed according to the following formula:

number

number

number

[0175] The calculation of the Gaussian intensity of the second seed, also called the negative seed, may be performed according to the following formula:

number

number

number

[0176] Note that coordinates are initialized to 128 for all points. Multiplying the Gaussian intensity by 127 yields a maximum value of 255 for points matching a positive seed. For points matching a negative seed, the intensity of the corresponding final point is 1. Thus, the intensity covers the same amplitude as the color channels, and if the point cloud has separate color channels, an additional simple normalization step is possible (normalizing each channel's 0-1 by 255). The dynamic s parameter as the median of the array of neighboring distances provides flexibility for handling various point cloud densities. Contributions from all points matching a seed are added, and the above fitting involves clipping coordinates that fall within [0, 255], concatenating the point to other coordinates, and so on for each point. The interaction channel / specification is the set of all concatenated coordinates. For example, for a 3D point cloud consisting of N points with coordinates x, y, and z as their only features, the fitting results in a final point cloud dimension of (N, 3 + 1). In the presence of color, the above arrangement results in, for example, (N, 6+1) dimensions. Experiments performed using only point coordinates, i.e., no color, showed excellent results.

[0177] As a result, in this implementation, the 3D point cloud is equipped with an interaction channel. The pair consisting of the 3D point cloud and its interaction channel is then fed as input to a neural network to perform segmentation.

[0178] Prior to this, in certain implementations of this implementation, this may include cropping the point cloud around the centroid of the positive seeds. Using this region instead of the entire point cloud reduces execution time. This particular implementation may also include uniform sampling to replicate points within sparse regions of the point cloud. This particular implementation may also include partitioning the points of the 3D point cloud into sets of 2048 points to match the input size of the neural network, PointCNN, in this particular implementation. In this particular implementation, the neural network then outputs predictions for these points, as described above. The segmentation method may also include merging the predictions based on the network's confidence score. The segmentation method may also optionally include post-processing to smooth predictions that may appear relatively sparse. Smoothing may use a voxel-based approach that relies on the confidence score output by the neural network. At each voxel, smoothing compares the average object category confidence across all contained points to a fixed threshold. This allows the voxel category to be described and all contained points to be labeled accordingly. One advantage of such post-processing methods is time efficiency.

[0179] An experimental example of the segmentation method will now be described.

[0180] In this experiment, a neural network is trained according to the example of the training method. The training data set used for training consists of synthetic point clouds, as described above. In the example of the training method in which a neural network is trained, the training method includes an implementation of the above-described provision of specifications for 3D point clouds for the training method. The experiment includes several implementations of segmentation methods.

[0181] A first implementation is shown in Figures 6 and 7. As shown in Figure 6, a 3D point cloud 60 containing an object 62 is displayed to the user. The user performs several clicks 620, 622, 624, and 626 on the object 62, with each click resulting in a positive seed. As shown in Figure 7, the object 62 is extracted by applying a neural network (S50).

[0182] A second implementation is shown in Figures 8 and 9. As shown in Figure 8, a 3D point cloud 80 containing an object 82 is displayed to the user. The user performs several clicks 824 and 822 on the point cloud 80. Click 822 results in a negative seed and click 824 results in a positive seed. As shown in Figure 9, the object 82 is extracted by applying a neural network (S50).

[0183] A third implementation is shown in Figures 10 and 11. As shown in Figure 10, a 3D point cloud 100 including an object 102 is displayed to a user. The user performs several clicks 1022 and 1024 on the object 102, with each click resulting in a positive seed. As shown in Figure 11, the object 102 is extracted by applying a neural network (S50).

[0184] A fourth implementation is shown in Figures 12 and 13. As shown in Figure 12, a 3D point cloud 120 including an object 122 is displayed to the user. The user performs several clicks 1222 and 1224 on the object 122, with each click resulting in a positive seed. As shown in Figure 13, the object 122 is extracted by applying a neural network (S50).

[0185] A flowchart of an example process for integrating the training method with the segmentation method is shown in Figure 14. As shown in Figure 14, the example process performs the following steps:

[0186] Offline Phase: This phase aims to train the model using point clouds with real-world data and user indicator / segmentation mask pairs that are likely to resemble user behavior. It involves two main steps. Note that this phase is transparent to the user.

[0187] Generating a training dataset.

[0188] Given a point cloud, collecting consistent indicators from real users can be very difficult and time-consuming, and there is no guarantee that they will cover a wide range of user behaviors. In this example, training indicators are generated automatically by defining user-like behaviors. Training user indicators include:

[0189] A positive training indicator consisting of at least one point in the training point cloud that is part of an identified object of interest in the same point cloud.

[0190] A negative training indicator consisting of at least one background point in the training point cloud that is not part of an identified target object in the same point cloud.

[0191] Note that at least one positive indicator must be generated for each target object.

[0192] The generated indicators are converted into additional channels (i.e., interaction channels) that are added to the original point cloud as input to the neural network, and then converted into a format that can be input to the neural network. The resulting training dataset then contains both an enriched point cloud with training indicators and a segmentation mask of the target object.

[0193] The neural network model is trained based on a training dataset, which is constructed on a point cloud DNN-based model.

[0194] Online Phase: Given the point cloud and user indicators, the data is first preprocessed to transform the indicators, then the trained model is applied, and finally a segmentation mask is obtained.

[0195] FIG. 15 shows a flow chart for the implementation of the process.

[0196] The training method and the segmentation method are computer-implemented methods.

[0197] This means that the steps (or substantially all steps) are performed by at least one computer or any similar system. Thus, the method steps may be performed fully automatically or semi-automatically by a computer. In one example, at least some of the method steps may be triggered via user-computer interaction. The level of user-computer interaction required may depend on the level of automation envisioned, balancing the need to implement user wishes. In one example, this level may be user-defined and / or predefined.

[0198] A typical example of a computer implementation of the method is to carry out the method using a system suitable for this purpose. The system may include a processor connected to a memory having a computer program recorded thereon, the computer program including instructions for carrying out the method, and a graphical user interface (GUI). The memory may store a database. The memory is any hardware suitable for such storage, and may optionally include several physically distinct parts (e.g., one for the program and optionally one for the database).

[0199] FIG. 16 shows an example of this system, which is a client computer system, such as a user's workstation.

[0200] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices, such as a hard drive 1030. Mass memory devices suitable for embodied computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented by, or incorporated into, specially designed ASICs (application-specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a keyboard, or other tactile device 1090. A cursor control device is used within the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. Furthermore, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse, with the buttons on the mouse being used to generate the signals. Alternatively, or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0201] A computer program herein may include computer-executable instructions, including means for causing the system to perform one or more of the methods. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, such as an article of manufacture tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the method may be performed by a programmable processor executing a program of instructions and performing the functions of the process by manipulating input data and generating output. Thus, the processor may be programmable to receive data and instructions from, or be coupled to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may also be a full installation program or an update program. In either case, instructions are provided for carrying out one or more of the present methods of applying the program to a system.

Claims

1. 1. A computer-implemented method of machine learning, comprising: providing (S10) a dataset of 3D point clouds, each 3D point cloud comprising at least one object, each 3D point cloud comprising one or more graphical user interaction specifications each representing a respective selection operation of the same object in the 3D point cloud; training (S20) a neural network configured to segment an input 3D point cloud containing an object based on the dataset, wherein the segmentation is based on the input 3D point cloud and a specification of one or more input graphical user interactions representing respective selection operations of the object in the 3D point cloud; Including, The provision of the 3D point cloud data set (S10) includes: providing the 3D point clouds and, for each 3D point cloud, information regarding the position of the same object in the 3D point cloud; For each 3D point cloud, determining a specification by simulating each of the one or more graphical user interactions; providing the 3D point cloud with a specification of the one or more graphical user interactions; Including, Each of the one or more graphical user interactions comprises: each corresponding to one or more seed locations defined on the 3D point cloud, and simulating the graphical user interaction includes determining the one or more seed locations; the one or more graphical user interactions include first graphical user interactions for selecting the same object and each corresponding to one or more first seed locations defined on the same object; For at least one 3D point cloud, the one or more graphical user interactions further include second graphical user interactions for discarding regions outside the same object, each corresponding to one or more second seed locations defined outside the same object. A method characterized by:

2. determining the one or more seed locations includes spreading the one or more first seed locations over the same object; 2. The method of claim 1 .

3. Equipping the 3D point cloud includes adding, to each point of the 3D point cloud, a coordinate that quantifies the strength of a selection operation at that point based on a simulation of each graphical user interaction.

3. The method according to claim 1 or 2.

4. wherein each of the one or more graphical user interactions is for selecting the same object or for discarding an area outside the same object, and for each point of the 3D point cloud, adding coordinates comprises: setting the coordinates to initial values; for each graphical user interaction for selecting the same object, incrementing the coordinate according to the proximity between the graphical user interaction and the point; and for each graphical user interaction to discard an area outside the same object, decreasing the coordinates according to the proximity between the graphical user interaction and the point.

4. The method of claim 3.

5. 5. A computer-implemented method for utilizing a neural network trained according to the method of any one of claims 1 to 4, comprising: providing a 3D point cloud including the object (S30); It is to repeat, performing a selection operation of said object by performing one or more graphical user interactions (S40); applying (S50) the neural network to segment the 3D point cloud based on the 3D point cloud and a specification of the one or more graphical user interactions; one or more times. A method characterized by:

6. After the selection operation is performed (S40) and before the neural network is applied (S50), determining, for each of the one or more graphical user interactions, one or more seed locations defined by the graphical user interaction to determine the specification; and providing the specification to the 3D point cloud, the specification including adding to each point of the 3D point cloud a coordinate quantifying the strength of a selection operation at that point.

6. The method of claim 5.

7. The one or more graphical user interactions include: performing one or more clicks on the object; performing a stroke on the object; defining a bounding box on and / or around the object; performing one or more clicks outside the object; and / or performing strokes outside the object 7. The method of claim 5, further comprising one or more of the following:

8. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 4 and / or the utilization method according to any one of claims 5 to 7.

9. 9. An apparatus comprising a data storage medium having recorded thereon a computer program according to claim 8.

10. and a processor connected to the data storage medium.

10. The device of claim 9.

Images