Interaction object selection

Abstract

The invention relates in particular to a computer-implemented machine learning method. The method comprises providing a dataset of 3D point clouds. Each 3D point cloud comprises at least one object. Each 3D point cloud is equipped with a specification of one or more graphical user interactions, each graphical user interaction representing a respective selection operation on the same object in the 3D point cloud. The method further comprises causing a neural network to learn based on the dataset, the neural network being configured for segmenting an input 3D point cloud comprising an object. The segmentation is based on the input 3D point cloud and on a specification of one or more input graphical user interactions, each of the one or more input graphical user interactions representing a respective selection operation on the object in the 3D point cloud. The method can be referred to as a learning method.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly to a method, system, and program for machine learning. Background Technology

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD, an abbreviation for Computer-Aided Design, refers, for example, to software solutions used for designing objects. CAE, an abbreviation for Computer-Aided Engineering, refers, for example, to software solutions used for simulating the physical behavior of future products. CAM, an abbreviation for Computer-Aided Manufacturing, refers, for example, to software solutions used for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play a crucial role in technical efficiency. These technologies can be embedded in Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes, and leverage enterprise knowledge to develop products across an extended company concept, from product conception to the end of the product's lifespan. Dassault Systèmes (traded as CATIA, ENOVIA, and DELMIA) offers PLM solutions that provide an engineering hub that organizes product engineering knowledge; a manufacturing hub that manages manufacturing engineering knowledge; and an enterprise hub that enables enterprise integration and connectivity between both the engineering and manufacturing hubs. The system collectively provides an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, which drives optimized product definition, manufacturing preparation, production, and service.

[0003] Against this and other backdrops, point cloud segmentation is becoming increasingly important. The following papers relate to point cloud segmentation and are cited below:

[0004] [1] Aleksey Golovinsk, Thomas Funkhouser, “Min-Cut Based Segmentation of Point Clouds”, ICCV Workshop, 2009.

[0005] [2] Kun Liu, Jan Boehm, "A New Framework or Interactive Segmentation ofPoint Clouds", - Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 2014, and

[0006] [3] Sedlacek D., Zara J., “Graph Cut Based Point-Cloud Segmentation for Polygonal Reconstruction” - ISVC 2009.

[0007] These methods lack efficiency, robustness, and generalization ability.

[0008] In this context, an improved machine learning approach is needed for point cloud segmentation. Summary of the Invention

[0009] Therefore, a computer-implemented machine learning method is provided. The method includes providing a dataset of 3D point clouds. Each 3D point cloud includes at least one object. Each 3D point cloud is equipped with a specification for one or more graphical user interactions. Each graphical user interaction represents a corresponding selection operation on the same object in the 3D point cloud. The method also includes learning a neural network based on the dataset, the neural network being configured to segment the input 3D point cloud including the object. The segmentation is based on the input 3D point cloud and on the specification for one or more input graphical user interactions. Each input graphical user interaction represents a corresponding selection operation on an object in the 3D point cloud.

[0010] The method may include one or more of the following:

[0011] - Datasets providing 3D point clouds include:

[0012] o provides 3D point clouds, and for each 3D point cloud, provides information about the position of the same object in the 3D point cloud; and

[0013] For each 3D point cloud:

[0014] The specifications are determined by simulating each graphical user interaction in one or more graphical user interactions; and

[0015] To enable 3D point clouds to be equipped with specifications for one or more graphical user interactions;

[0016] - Each graphical user interaction in one or more graphical user interactions corresponds to one or more seed locations customized on a 3D point cloud, and the simulation of the graphical user interaction includes determining one or more seed locations;

[0017] - One or more graphical user interactions include a first graphical user interaction for selecting the same object and corresponding to one or more first seed positions customized on the same object;

[0018] - Determining one or more seed locations includes distributing one or more first seed locations over the same object;

[0019] - For at least one 3D point cloud, one or more graphical user interactions further include a second graphical user interaction for discarding the area outside the same object and corresponding to one or more second seed locations customized outside the same object;

[0020] - Equipping the 3D point cloud includes adding the coordinates of the selected operation intensity at the quantized point to each point of the 3D point cloud based on a simulation of each graphical user interaction; and / or

[0021] - Each of one or more graphical user interactions is used to select the same object or discard the outer region of the same object, and for each point in the 3D point cloud, adding coordinates includes:

[0022] o sets the coordinates to initial values;

[0023] For each graphical user interaction used to select the same object, the coordinates are increased based on the proximity of the graphical user interaction to the point; and

[0024] For each graphical user interaction used to discard the outer region of the same object, the coordinates are reduced based on the proximity between the graphical user interaction and the point.

[0025] It also provides neural networks that can be learned using this method, such as neural networks that have already been learned using this method.

[0026] A method for using a computer implementation of a neural network is also provided. The method includes providing a 3D point cloud comprising objects. The method also includes one or more iterations of: performing a selection operation on objects by executing one or more graphical user interactions; and segmenting the 3D point cloud based on the 3D point cloud and based on the specifications of the one or more graphical user interactions by applying a neural network.

[0027] The usage method may include one or more of the following:

[0028] - The method further includes, after performing the selection operation and before applying the neural network:

[0029] o Determines the specification by determining the location of one or more seed locations defined by the graphical user interaction for each of one or more graphical user interactions; and

[0030] o Configure the 3D point cloud with specifications, which include adding coordinates of the intensity of the selection operation at each point in the 3D point cloud to the quantized point; and / or

[0031] - One or more graphical user interactions include one or more of the following:

[0032] o Perform one or more clicks on the object;

[0033] o performs a line drawing on the object;

[0034] o Defines a bounding box on and / or around an object;

[0035] o Perform one or more clicks outside the object; and / or

[0036] o Performs a line drawing outside the object.

[0037] A computer program is also provided, which includes instructions for performing the method and / or using the method.

[0038] An apparatus is also provided, which includes a data storage medium on which computer programs and / or neural networks have been recorded.

[0039] This device can be formed or used as a non-transitory computer-readable medium, such as in a SaaS (Software as a Service) or other server or cloud-based platform. The device may alternatively include a processor coupled to the data storage medium. Therefore, the device can form a computer system wholly or partially (e.g., the device is a subsystem of an overall system). The system may further include a graphical user interface coupled to the processor. Attached Figure Description

[0040] Embodiments of the invention will now be described by way of non-limiting examples and with reference to the accompanying drawings, wherein:

[0041] - Figures 1 to 15 The method is shown; and

[0042] - Figure 16 An example of the system is shown. Detailed Implementation

[0043] A computer-implemented machine learning method is proposed. The method includes providing a dataset of 3D point clouds. Each 3D point cloud includes at least one object. Each 3D point cloud is equipped with a specification for one or more graphical user interactions, each representing a selection operation on the same object in the 3D point cloud. The method also includes enabling a neural network configured to segment the input 3D point clouds containing objects to learn based on the dataset. Segmentation is based on the input 3D point clouds and on the specification for one or more input graphical user interactions, each representing a selection operation on an object in the 3D point cloud. This method may be referred to as a learning method.

[0044] This constitutes an improved machine learning approach for point cloud segmentation.

[0045] It is worth noting that this learning method enables a neural network configured to segment an input 3D point cloud containing objects to learn. This equates to the learning method using machine learning, such as deep learning, which is an unconventional approach in the context of point cloud segmentation. It is noteworthy that, in the context of new technologies, this allows for benefiting from the known advantages of machine learning, such as deep learning. Indeed, as explained in this disclosure, the method has been successfully tested.

[0046] The neural network is learned based on a provided dataset of 3D point clouds, each of which is equipped with a specification for one or more user interactions, each representing a selection operation on the same object within the 3D point cloud. In other words, for each 3D point cloud, the 3D point cloud includes at least one object, referred to as the "same object" or "the same object," and each of the one or more graphical user interactions represents a selection operation on that same object. The provided dataset can also be called a training dataset. Therefore, the training dataset consists of 3D point clouds, each equipped with a specification that defines the selection operation on the same object through one or more graphical user interactions. This is equivalent to saying that the training dataset includes pairs of 3D point clouds and data specifying object selection through graphical user interactions within each 3D point cloud. Enabling the neural network on such a training dataset allows the neural network to perform segmentation of the input 3D point cloud based solely on the input 3D point cloud and on the specification of selecting the input object through graphical user interactions on the 3D point cloud. In other words, the training dataset allows the neural network to learn in such a way that it only needs information about the object's location on the 3D point cloud and about the selection of that object through graphical user interaction to segment the object. Therefore, the training dataset allows the neural network to learn to perform 3D point cloud segmentation in an efficient and simple manner. Furthermore, the segmentation depends only on information about the object's location on the 3D point cloud and about the selection of that object through graphical user interaction. For example, the neural network does not rely on other specifications, such as specifications for object category / type / class. In fact, as discussed further below, each point in each 3D point cloud of the training dataset can include a single annotation as either "object" (i.e., the point belongs to an object, such as the same object mentioned) or "background" (i.e., the point belongs to the background), without further annotations, such as indicating the object category of the point (e.g., a chair). Therefore, when segmenting the input 3D point cloud, the neural network is independent of the type of object in the input 3D point cloud. In other words, the neural network can extract any type of object from the input 3D point cloud, not just a specific type of object (e.g., a chair). This makes the neural network robust.

[0047] As discussed further below, any 3D point cloud in this paper may be synthetically obtained (e.g., simulated via virtual scan or CAD / mesh conversion) or may originate from physical measurements performed in the real world (e.g., 3D scans). In other words, the neural network is independent of the type of 3D point cloud it must segment, which makes the neural network robust.

[0048] Furthermore, besides being configured to segment 3D point clouds, the neural network is actually configured for interactive segmentation of 3D point clouds. In fact, the neural network is specifically learned based on the rules governing object selection via graphical user interaction, allowing the neural network to segment the input 3D point cloud by detecting object selections made through graphical user interaction on the input 3D point cloud. In other words, and as discussed further in detail in the usage method below, the neural network allows the user to graphically select objects on the input 3D point cloud and then automatically obtain the segmentation of the input 3D point cloud based on that selection by applying the neural network. Since the user does not need to perform a large number and / or complex physical actions, this allows segmentation to be performed with reduced user burden: simple, ordinary, and / or physically fatigue-free graphical interaction with the machine is sufficient. Therefore, this learning method provides an ergonomically improved solution for interactive segmentation of 3D point clouds.

[0049] A neural network that can be learned according to this method is also provided, for example, a neural network that has already been learned according to this method. Therefore, the neural network is configured to segment an input 3D point cloud including objects. The segmentation is based on the input 3D point cloud and a specification of one or more input graphical user interactions. Each graphical user interaction represents a corresponding selection operation on an object in the 3D point cloud. Because the neural network can be learned according to this method, the segmentation of this neural network is particularly efficient and has a certain generalization ability. For example, as discussed earlier, the segmentation is independent of the type of the 3D point cloud and the type of objects contained in the 3D point cloud.

[0050] It also provides a method for using a computer implementation of a neural network. The method includes providing a 3D point cloud containing objects. The method also includes one or more iterations of: performing a selection operation on objects by executing one or more graphical user interactions; and segmenting (e.g., automatically) the 3D point cloud based on the 3D point cloud and the prescribed segmentation of the 3D point cloud by applying a neural network based on one or more graphical user interactions. The method may be referred to as a "segmentation method".

[0051] Segmentation methods allow for interactive segmentation in an ergonomically improved manner. In fact, as discussed earlier, neural networks learn in such a way that users only need to segment a 3D point cloud by making object selections through graphical interaction: the user performs object selection, and then the 3D point cloud is segmented (e.g., automatically) by applying a neural network. If the user wishes to perform further segmentation and / or improve the level of detail of the already performed segmentation, the user only needs to perform another object selection through graphical user interaction, and then another segmentation is achieved (e.g., automatically) by applying a neural network, and so on. In other words, segmentation methods allow for interactive (potentially iterative) segmentation of an input 3D point cloud. Furthermore, the segmentation is ergonomic because it does not place a significant burden on the user. In practice, only graphical interactions are required, such as simple, common, and / or effortless graphical interactions, such as clicking, drawing lines, and / or drawing / defining bounding boxes. For example, as discussed further below, these graphical interactions can be performed using haptic devices, such as a mouse and / or touch-based haptic devices (e.g., touchscreens or touchpads). Furthermore, since neural networks are independent of the type of 3D point cloud and the object category / classification / type, segmentation can be performed on any type of 3D point cloud, including any type of object.

[0052] The learning method and the segmentation method can be executed independently, for example, by different users and / or on different computers. Alternatively, the learning method and the segmentation method can be integrated into a process implemented on the same computer. (See now for reference.) Figure 1 Discuss the process, the Figure 1 A flowchart illustrating an example of the process is shown.

[0053] The process includes an offline phase that integrates the learning method. The offline phase includes providing a dataset of 3D point clouds (S10) according to the learning method. The offline phase also includes enabling a neural network (S20) to learn, the neural network being configured to segment the input 3D point cloud, including objects, according to the learning method. The offline phase may also include, for example, storing the learned neural network in memory, as in the online phase now discussed for this process.

[0054] The process also includes an online phase that integrates the segmentation method. The online phase may include providing a learned neural network, for example, by retrieving the neural network from memory that was stored in the offline phase. The online phase includes providing a 3D point cloud of the object in S30 according to the segmentation method. The online phase also includes performing an object selection operation in S40 according to the segmentation method and one or more iterations of segmenting the 3D point cloud according to the segmentation method by applying the neural network in S50.

[0055] Now let's discuss learning methods further.

[0056] This learning method is a machine learning approach for point cloud segmentation.

[0057] As is well known, point clouds are unordered sets of points with coordinates (usually 3D coordinates), which may be accompanied by other features (e.g., encapsulated in additional coordinates), such as intensity or color. This unordered aspect of the data makes it difficult to analyze, especially compared to structured meshes such as images. Different formats of point clouds exist, but as discussed earlier, learned neural networks are independent of these different formats.

[0058] Point cloud segmentation, also known as object segmentation, corresponds to the task of extracting all points in a point cloud that belong to an object from the background points of that point cloud. As mentioned earlier, neural networks learned through learning methods can handle all types of objects, i.e., regardless of object category. Furthermore, as previously mentioned, neural networks allow for interactive segmentation. Interactive segmentation refers to segmentation in which a user expresses their interest in an object by interacting with the point cloud.

[0059] The learning method is the machine learning method.

[0060] As is well known in the field of machine learning, the processing of input by a neural network involves applying operations to the input, which are defined by data including weight values. Making a neural network learn therefore involves determining the values ​​of the weights based on a dataset configured for such learning, which may be called a learning dataset or training dataset. For this purpose, a dataset includes data segments, each forming a corresponding training sample. Training samples represent the diversity of situations in which the neural network will be used after learning. Any dataset referred to herein may include a number of training samples greater than 1,000, 10,000, 100,000, or 1,000,000. In the context of this disclosure, "making a neural network learn based on a dataset" means that the dataset is a learning / training dataset for the neural network.

[0061] In the context of learning methods, the training dataset for neural networks is a provided 3D point cloud dataset, which will now be discussed.

[0062] The training dataset comprises 3D point clouds, each containing at least one object and equipped with a specification for one or more graphical user interactions, each representing a selection operation on the same object within the 3D point cloud. In other words, each 3D point cloud and its associated specification forms a training sample of the training dataset. Since this specification relates to graphical user interaction, it can be referred to as an "interaction channel" or "user indicator." This specification is discussed further below. For simplicity, such training samples can be simply referred to as "3D point clouds and their interaction channels." In any case, each training sample comprises a pair consisting of a 3D point cloud and its interaction channel. Each training sample may also include a segmentation mask that, for each point in the 3D point cloud, annotates points that belong to or do not belong to the same object. This allows the neural network to learn in a fully supervised manner, as discussed further below. In the example, the training dataset may include approximately 2000 training samples.

[0063] As is well known, a 3D point cloud consists of a set of 3D points, each described by, for example, three Cartesian coordinates in 3D space. This means that each point cloud can be associated with a set of triples, each representing the 3D coordinates of a point in the 3D point cloud. Any 3D point cloud in this paper can represent a real-world 3D scene, which is a part of the real world (e.g., a factory scene or an office scene), and each point in the 3D point cloud represents a point within that scene. "Representing a real-world scene" does not necessarily mean that the 3D point cloud originates from physical measurements taken in the real world. For example, any 3D point cloud in this paper can be a synthetic representation of a real-world scene, as discussed further below. Any 3D point cloud includes at least one object. Each object in the 3D point cloud in this paper is a set of points in the 3D point cloud, each representing a point of the same object (e.g., the same object in a real-world scene), which together form a representation of the object.

[0064] Any 3D point cloud in this document can be a synthetic point cloud, i.e., a point cloud derived from a synthetic point cloud generation process. In other words, a synthetic point cloud representing a real scene is not derived from physical measurements of the real scene to capture a representation of the real scene, but rather forms a virtual, digitally generated representation of the real scene. The synthetic point cloud generation process specifies any computer-implemented process that outputs a synthetic 3D point cloud. Such a process can output 3D point clouds, each with point-level annotations specifying whether each point in the output 3D point cloud belongs to an object (or, if not, to the background). Alternatively, such a process can output unannotated 3D point clouds, with each point annotated at a later stage. Alternatively, any 3D point cloud in this document can be determined from physical measurements of the real scene. Determining a 3D point cloud can include: providing one or more physical sensors (e.g., 3D scanning, such as LIDARS), each configured to acquire a corresponding physical signal, and acquiring one or more corresponding physical signals by operating the one or more physical sensors in the real scene (i.e., scanning the real scene using each sensor). Then, based on any known technology, the 3D point cloud is automatically determined based on measurements. Note that today, existing sensors are capable of providing clouds of millions of points, thus providing very high-quality, very dense 3D scans.

[0065] In the example, the 3D point cloud of the training dataset, or at least a portion thereof, can be a synthetic 3D point cloud. In this case, providing the training dataset in S10 may include synthesizing these 3D point clouds using any known suitable synthetic point cloud generation process. Providing S10 may further include annotating each point cloud by labeling each point belonging to an object as "object" and each point belonging to the background as "background". Alternatively, providing these 3D point clouds in S10 may include retrieving these 3D point clouds and their annotations from memory (e.g., remote memory) where, for example, the 3D point clouds have been stored after their generation.

[0066] In these examples, generating 3D point clouds may include providing CAD (Computer-Aided Design) models of 3D objects. Generating 3D point clouds may also include converting each CAD model of the provided 3D objects into a 3D point cloud representing the 3D object, referred to for simplicity as an "object point cloud." For simplicity, generation may also include providing 3D point clouds representing backgrounds, each referred to as a "background point cloud." Generation may also include inserting each object point cloud into the background point cloud. This results in generated 3D point clouds, each freely inserted into one or more object point clouds within the background point cloud. Generation may further include annotating each point in each object point cloud as "object" and each point in each background point cloud as "background." Figure 2 and 3 These examples are shown. Figure 2 An example of a CAD object 20 converted into an object point cloud 22 is shown. Figure 3 Example 30 shows an object point cloud inserted into a background point cloud.

[0067] Note that when inserting objects, the method can consider possible occlusion (object / object occlusion or object / background occlusion) and point density normalization for realistic synthetic point cloud representation. Specifically, the method can consider these aspects to insert a point cloud obtained from a CAD model into a scatter plot representing the background. These aspects correspond to adjustment steps to reproduce the effect that can be observed in the real point cloud (from a 3D scanner). For example, if a point cloud corresponding to a CAD model of a table is inserted into the background, the method can adjust the density of that point cloud so that it corresponds to the density of the background into which it will be inserted. In the same way, the method can consider occlusion to approximate the real point cloud. Taking the table as an example, the method can remove points belonging to the background located under the table legs to retain only the points that can be obtained by a real scanner. It should also be noted that the 3D scanner only captures visible surface points during acquisition. To avoid this, the method allows occlusion to be added to the synthetically generated dataset in the example.

[0068] In alternative examples to these examples, generating 3D point clouds may include applying an automated point cloud generation method that further provides automatic and accurate point-level annotations. The generation method can be any known method, such as the one described in the 2018 paper “A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving” by Xiangyu Yue, Bichen Wu, Sanjit A. Seshia, Kurt Keutzer, and Alberto L. Sangiovanni-Vincentelli, which is incorporated herein by reference. This method proposes virtual scanner simulation to extract point clouds from 3D CAD scenes, such as video game environments.

[0069] In other examples, the 3D point clouds of the training dataset, or at least a portion thereof, may originate from physical measurements performed on a real-world scene, as discussed previously. In this case, providing the training dataset via S10 may include performing physical measurements and determining the 3D point clouds based on those measurements, as discussed previously. Providing S10 may further include annotating each point cloud by labeling each point belonging to an object as "object" and each point belonging to the background as "background". Alternatively, providing S10 of these 3D point clouds may include retrieving the 3D point clouds and their annotations from memory (e.g., remote memory), for example, where the 3D point clouds have been stored after their generation. For example, 3D point clouds of training datasets can be retrieved from public point cloud datasets such as “SEMANTIC3D.NET: A new large-scale point cloud classification benchmark” by Timo Hackel, N. Savinov, L. Ladicky, Jan D. Wegner, K. Schindler, and M. Pollefeys in ISPRS in 2017; “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite” by Andreas Geiger, Philip Lenz, and Raquel Urtasun in CVPR in 2012; and “3D Semantic Parsing of Large-Scale Indoor Spaces” by Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese in CVPR in 2016, all of which are incorporated herein by reference. Providing S10 can include binary annotation of point clouds retrieved from such public datasets, labeled "object vs. background".

[0070] In any case, providing the S10 3D point cloud training dataset may include providing 3D point clouds. As previously described, each 3D point cloud is equipped with its own specification. This specification forms data about one or more graphical user interactions, each representing a corresponding selection operation on the same object in the point cloud. That is, the specification describes how the same object is selected via one or more graphical user interactions, for example, based on the relative positioning of the object and the data from one or more graphical user interactions. This allows a neural network to learn that the neural network can segment the 3D point cloud based solely on detecting object selection via graphical user interactions. Providing the S10 training dataset may include determining the specification for each point cloud and equipping the point cloud with the specification. Determining the specification may include computational / simulation of the specification, i.e., for each 3D point cloud in the dataset. Alternatively, determining the specification may include, for example, manually providing the specification, i.e., for each 3D point cloud in the dataset, by a user performing one or more graphical user interactions.

[0071] As previously mentioned, this specification can be referred to as an "interaction channel" or a "user indicator." Specifically, each point in the 3D point cloud can be represented by a set of three Cartesian coordinates in 3D space. This point can further include a fourth coordinate indicating how the point is affected by one or more graphical user interactions. In other words, the fourth coordinate forms data encapsulating the degree to which the selection of the same object through graphical user interaction affects the point. This data can, for example, represent the proximity of the point to at least one of the one or more graphical user interactions.

[0072] Now let's discuss an example of providing the S10 training dataset.

[0073] In the example, providing a training dataset for S10 3D point clouds includes providing 3D point clouds. Providing the S10 training dataset also includes providing information about the location of the same object within the 3D point cloud for each 3D point cloud. Providing the S10 training dataset further includes determining specifications for each 3D point cloud by simulating each graphical user interaction in one or more graphical user interactions, and equipping the 3D point cloud with specifications for one or more graphical user interactions.

[0074] Information about the location of the same object can be any information that allows the same object to be distinguished from other parts of the 3D point cloud (e.g., other objects, if any, or the background). In the example, the information includes a note for each point of the object as the point “object”. The information may also include the coordinates of the point. Providing information may include annotating each point of the same object as “object”, as described above. Providing information may further include annotating each point of the background as “background”, as described above. Providing information may include annotating each point of each other object (if any) as “other object”. This can be used by the method to, for example, add negative seeds to neighboring objects of the target object, as discussed further below. However, this is not mandatory. In practice, it is sufficient to add seeds in the region around the object of interest so that these seeds can also be found on neighboring objects (if any).

[0075] Simulating each graphical user interaction generates data about the location of the graphical user interaction. It should be understood that "simulation" means the graphical user interaction is virtual, i.e., digitally created. In other words, the learning method equips a 3D point cloud with rules for object selection through simulated user behavior. In other words, the data about location is digitally created, for example, relatively randomly and / or according to a predefined procedure / strategy. Rules are determined based on this data and information about the location of the same object. For example, rules can be determined by considering this data and that information to generate rules. Simulating each graphical user interaction in one or more graphical user interactions allows rules to be obtained without manual user action, which would be time-consuming and / or prone to overfitting. Furthermore, this makes the graphical user interactions virtual, which, for example, allows for relatively random and / or good diversity of graphical user interactions, resulting in a well-diversified training dataset. As is known from the field of machine learning, this improves the quality of learning.

[0076] In the examples, each of the one or more graphical user interactions corresponds to one or more seed locations defined on a 3D point cloud. In these examples, the simulation of the graphical user interaction involves determining one or more seed locations.

[0077] Each seed location is data representing the location of at least a portion of a graphical user interaction. One or more seed locations together form data representing the location of the graphical user interaction on a 3D point cloud. Each seed location can, for example, be a set of three Cartesian coordinates in 3D space, indicating the location of a point in the graphical user interaction. In other words, a seed location can be a point in 3D space that may or may not correspond to a point in the 3D point cloud. For example, a seed location can be a point in the 3D point cloud that is closest to the location represented by the seed for at least a portion of the graphical user interaction. This allows the simulated graphical user interaction to be a virtual representation of many different and common graphical user interactions for object selection. In fact, many types of well-known and common graphical user interactions can be represented by a set of one or more seed locations. For example, one or more seed locations can represent one or more clicks by a user on a 3D point cloud, each seed location representing (e.g., is) a point in 3D space corresponding to the location of one of those clicks. A seed location can, for example, be the point in the 3D point cloud closest to the click location. Alternatively, one or more seed locations may represent lines drawn by the user on a 3D point cloud, each seed location representing (e.g., is) a point in 3D space corresponding to a sampled location of the drawn line. Alternatively, one or more seed locations may represent bounding boxes drawn / defined by the user on a 3D point cloud, each seed location representing (e.g., is) a point in 3D space corresponding to a sampled location outside or inside the bounding box. Since all of these hold true for every graphical user interaction, this means that one or more graphical user interactions can be a virtual representation of one or more of the following: one or more clicks on a 3D point cloud, drawing lines on a 3D point cloud, and / or drawing / defining bounding boxes on a 3D point cloud. For simplicity, each seed location will be simply referred to as a "seed" below.

[0078] Determining one or more seeds may include calculating one or more seeds, for example, relatively randomly and / or according to a predetermined strategy, to ensure, for example, a certain number of seeds and / or reasonable distances between seeds. In the case where one or more seeds are one or more points in 3D space, determining one or more seeds may include determining the coordinates of one or more points.

[0079] In the example, one or more graphical user interactions include a first graphical user interaction for selecting the same object. The first graphical user interaction corresponds to one or more first seed positions defined on the same object.

[0080] Since the first graphical interaction involves selecting the same object, it can be referred to as a "positive interaction." In other words, a positive interaction represents a graphical user selection of the same object by actually selecting it, rather than discarding other parts of the 3D point cloud. Providing a training dataset with data on positive interactions allows a neural network to learn to detect such interactions for segmentation. Examples of such positive interactions that the first graphical user interaction can represent include: clicking once or multiple times on the same object, drawing a line on the same object, or drawing / defining a bounding box on and / or around the same object.

[0081] Since the first graphical user interaction is a positive user interaction, it is a virtual representation of a graphical user interaction performed on the same object, located at a point on that same object. Therefore, one or more first seed locations are each defined on the same object. For example, in the case where one or more first seed locations are one or more points, the one or more points are substantially located between points on the same object. Simulating the first graphical user interaction involves determining one or more first seeds. Determining one or more first seeds can be done randomly, for example, by randomly selecting one or more points on the same object as one or more first seeds; or partially randomly. This simplifies the formation of the training dataset.

[0082] In the example, determining one or more seed locations includes spreading one or more first seed locations over the same object. Spreading one or more first seeds may include determining each seed by randomly selecting points in a point cloud from a limited set of points. Spreading one or more first seeds may include determining the number of first seeds and selecting that number of first seeds spaced apart by a predetermined distance. The number and / or the predetermined distance may depend, for example, on the object size. This allows the first seeds to cover the same object well.

[0083] Now let's discuss the implementation of distribution.

[0084] This implementation allows for the simulation of positive user interactions. Compared to completely random methods, the first graphical user interaction simulated here corresponds to a positive interaction of user likelihood, where the interaction results in a certain number of seeds regarding the size of the same objects and reasonable distances between them. This allows for good object coverage. For example, when selecting an object by clicking on it, the user typically uses the click to cover the object. When selecting an object by drawing a line on it, the user typically draws a line that covers the object. When selecting an object using a bounding box, the user can draw the bounding box in such a way that the interior of the bounding box covers the object, for example, by framing the object within the bounding box. This implementation includes determining the number of one or more first seeds based on a normal probability distribution of the type:

[0085]

[0086] in, X The number of the first seed. μ Corresponding to (i.e. equal to) M*p / K ,in M The number of points for the same object. p The proportion of the object to be affected by the first graphical interaction (e.g., 20%) and K This refers to the number of neighbors. K This will be discussed further below. σ can be equal to 1. This allows for flexibility regarding the number of first seeds while maintaining consistency with the pooling of the graphical interactions under consideration. This embodiment may further include fixing the minimum distance between seeds, thereby allowing for better coverage of the same object. This fixing may include iteratively selecting the first seed and, in each iteration, excluding from the selection all points contained in the bounding box region surrounding the previously selected first seed. The size of the bounding box may be fixed relative to the size of the object (e.g., a proportionate length, width, and height of the object). The number X of first seeds can be selected according to formula (1), but may be rounded to the nearest integer greater than or equal to 1.

[0087] We will now discuss a specific implementation of the scattering implementation currently under discussion. In this specific implementation, the objects are referred to as "interest objects," and each seed is referred to as a "positive seed." The specific implementation is executed according to an algorithm, which has the following stipulations:

[0088] The input variables are:

[0089] objPoints: A variable containing all points of the object of interest.

[0090] K: This is a variable representing the number of neighbors considered for this point cloud. The number K will be discussed further below.

[0091] proportion: A parameter that controls the scale of objects affected by the first graphical interaction.

[0092] standardDeviation: A parameter that controls the variance of the distribution, which is used to indicate the number of positive seeds.

[0093] The output variable is:

[0094] seeds: A variable containing a list of all positive clicks (coordinates) for an object of interest.

[0095] The intermediate variables that appear are:

[0096] numberOfObjPoints: A variable corresponding to the number of points belonging to the object of interest.

[0097] numberOfSeeds: A variable corresponding to the number of positive clicks that will be performed on this object of interest.

[0098] seed: A variable corresponding to the coordinates of a single positive click.

[0099] The function that is not explicitly declared but appears is:

[0100] RandPoint(): A function that returns a random point from a set of points.

[0101] RegionAround(): This function returns the region surrounding a point's coordinates. In this context, it's used to suppress all points within the region surrounding the previously clicked object, thus ensuring a minimum distance between clicks.

[0102] NormalDistribution(mean, standardDeviation): A function that returns a floating-point value based on a normal distribution defined by the input parameters.

[0103] The algorithm is described by the following pseudocode:

[0104]

[0105] In an example where one or more graphical user interactions include a first graphical user interaction, for at least one 3D point cloud (i.e., at least one 3D point cloud in the training dataset), one of the graphical user interactions further includes a second graphical user interaction for discarding regions outside the same object and corresponding to one or more second seed locations, each defined outside the same object.

[0106] Because the second graphical user interaction involves selecting the same object by discarding regions outside the object, it can be referred to as a "negative interaction." In other words, a negative interaction represents a graphical user selection of the same object by discarding regions in the point cloud that are not part of the same object. Providing data on negative interactions, in addition to positive interactions, to the training dataset allows the neural network to learn to detect both positive and negative interactions to perform segmentation. Since graphical selection of objects in a point cloud can often involve a mixture of positive interactions (e.g., for performing a coarse selection of objects) and negative interactions (e.g., for refining the selection by discarding external regions (e.g., background or other objects)), this enhances the neural network's ability to perform segmentation. Examples of such negative interactions that the second graphical user interaction can represent include: performing one or more clicks outside the same object (e.g., on the background or another object) or drawing a line outside the same object (e.g., on the background or another object). Furthermore, the ability of the learning method to mix positive and negative interactions allows the neural network to learn in such a way that it can detect object graphical selections containing bounding boxes drawn / defined by the user around the object: points outside the bounding box (or at least one sample of these points) are detected by the neural network as a second seed, while points inside the bounding box (or at least one sample of these points) are detected as a first seed.

[0107] The mixing of positive and negative seeds allows each graphical user interaction to represent a typical graphical interaction, such as a click, a dash, and a bounding box, all of which share the characteristics represented by the seeds as described above. Specifically, since a click directly corresponds to a seed, a click represents a minimal interaction. By sampling the location of the seed from the line of a dash, the dash can be sampled as a set of seeds (e.g., a set of single points), making this set of seeds very close to the generation performed for multiple clicks in the desired generation. Assuming a bounding box is used to coarsely indicate an object of interest, the bounding box can be transformed into a set of seeds by sampling the single point indicators inside and outside the box for positive and negative clicks, respectively. However, in the example, the learning method can deviate from these paradigms to determine the seeds that best correspond to a particular type of user interaction, i.e., for applications where the input graphical user interaction is always the same type (e.g., a bounding box). For example, the learning method can perform optimization to generate seeds from bounding boxes by sampling seeds from the inside and outside of a training box, which has varying height, length, and width for each interaction on the same object. Furthermore, some training bounding boxes may partially include the same object. In this way, even if the user provides a bounding box that only partially surrounds the object, the neural network can identify the object in the point cloud.

[0108] Since the second graphical user interaction is a negative user interaction, it is a virtual representation of a graphical user interaction performed outside the same object, i.e., at a point outside the same object. Therefore, one or more second seed locations are each defined outside the same object. For example, in the case where one or more second seed locations are one or more points, the one or more points are substantially located between points outside the same object. Simulating the second graphical user interaction involves determining one or more second seeds. The determination of one or more second seeds can be randomized, for example, by randomly selecting one or more points outside the same object as one or more second seeds (i.e., one or more points of the background or another object) or partially randomized.

[0109] We will now discuss determining the implementation of one or more second seeds.

[0110] In this implementation, one or more second seeds can be determined based on different strategies. Each strategy allows for the simulation of typical user behavior and matches the actual interaction type.

[0111] According to the first strategy, one or more second seeds are determined around the same object (e.g., within a predetermined distance of the same object). This strategy simulates the most likely way for the user to specify the object boundary. Determining one or more second seeds according to this strategy may include extracting a closed region of background points around the object of interest (e.g., a region less than 30 cm, i.e., a strip around the object with a width of 30 cm) and selecting one or more random points in that region as one or more second seeds.

[0112] According to the second strategy, determining one or more second seeds around the same object (e.g., within a predetermined distance of the same object) determines one or more second seeds on neighboring objects of the same object (e.g., objects within a predetermined distance of the same object). This strategy simulates classifying objects as background objects and also matches actual user behavior. Determining one or more second seeds according to the second strategy requires annotating all objects in the point cloud scene. Determining one or more second seeds according to this strategy may include randomly selecting points on each background object among the background objects closest to the same object, and the selected one or more points are one or more second seeds.

[0113] According to the third strategy, one or more second seeds are determined randomly (e.g., even far from the object of interest) to suppress random erroneous first seeds and match other user behaviors. The determination according to this strategy may include randomly selecting one or more background points (i.e., all points in the 3D point cloud minus the points of the same object) as one or more second seeds.

[0114] It should be understood that while for each 3D point cloud in the training dataset, at least one positive interaction (i.e., the first graphical user interaction) exists within one or more graphical user interactions, a negative interaction (i.e., the second graphical user interaction) does not necessarily exist. In the example, there are one or more 3D point clouds in the dataset, and for this dataset, the one or more graphical user interactions include the second graphical user interactions discussed earlier. The learning method can determine one or more second seeds corresponding to each of these second graphical user interactions, each of which utilizes a given strategy, for example, a strategy relatively randomly selected from the first, second, and third strategies discussed earlier. "Relatively random" means that some strategies may be more likely to be selected than others. For example, the first strategy is most likely to be selected when simulating user behavior that may occur in the real world. The second strategy can be optional, i.e., in the example of the learning method, for example, the second strategy is never selected when the background object itself is not annotated.

[0115] For the first graphical user interaction, the determined number of the second seed can be derived from a probability distribution centered on the number of targets for the second seed, independent of the policy. The number of targets for the second seed can typically be a parameter that controls both the amount of simulated interaction and the difficulty of the task (i.e., segmentation) the neural network learns. For example, a target number of 0 aims to ensure that positive interactions provide only enough information to extract the object, but may lead to unstable negative clicks and limit the quality of extraction. On the other hand, a high target number (e.g., 15) should ensure accurate mask segmentation, but will require many user interactions during neural network operation. Corresponding implementations involve determining one or more second seed numbers based on a normal probability distribution of the following types:

[0116]

[0117] in, X This represents the number of the second seed. An example of a normal probability distribution has parameters μ = 1 and σ = 1. In determining... X At that time, the number of second seeds X It can have 0 as the lower limit.

[0118] We now discuss a specific implementation of the current discussion that determines one or more second seeds. In this specific implementation, the object is referred to as the "interest object," and each seed is referred to as a "negative seed." The specific implementation is executed according to an algorithm, which has the following stipulations:

[0119] The input variables are:

[0120] objPoints: A variable containing all points of the object of interest.

[0121] scenePoints: A variable containing all points in the scene (including objects of interest).

[0122] mean: A parameter that controls the mean of the distribution used to indicate the number of negative seeds.

[0123] standardDeviation: A parameter that controls the variance of the distribution used to indicate the number of negative seeds.

[0124] pStrategy: A parameter that controls the probability of choosing one strategy over another.

[0125] The output variable is:

[0126] seeds: A variable containing a list of all negative clicks (coordinates) for an object of interest.

[0127] The intermediate variables that appear are:

[0128] backgroundPoints: A variable containing all points in the point cloud scene minus the points belonging to the object of interest.

[0129] strategy: The variable corresponding to the selected strategy. This variable may be:

[0130] orandom (random point from background points)

[0131] oneighborObjs (click on the object closest to the object of interest).

[0132] osurroundings (click in a small area around the object of interest).

[0133] numberOfSeeds: A variable corresponding to the number of negative clicks performed on this interest object.

[0134] seed: A variable corresponding to the coordinates of a negative click.

[0135] nearestObjs: This is a variable that is a list of point clouds of all neighboring objects. Its length is numberOfSeeds.

[0136] Functions that appear without being explicitly declared are:

[0137] RandPoint(): A function that returns a random point from a set of points.

[0138] RegionAround(): This function returns the region surrounding a point's coordinates. In our case, this function is used to suppress all points in the background surrounding the previous click, thus ensuring a minimum distance between clicks.

[0139] NearestObjs(): A function that returns a list of point clouds containing all neighboring objects. The length of the list is numberOfSeeds.

[0140] SmallerRegionAroundObj(): A function that returns the enclosed region of background points surrounding the object of interest.

[0141] NormalDistribution(mean, standardDeviation): A function that returns a floating-point value based on a normal distribution defined by the input parameters.

[0142] The algorithm is described by the following pseudocode:

[0143]

[0144] In any case, following the simulation, data regarding the location of each graphical user interaction has been determined through the simulation of the graphical user interaction. In the example, at that point, for each 3D point cloud and for each image user interaction in one or more graphical user interactions targeting the 3D point cloud, the simulation result of the graphical user interaction is:

[0145] - If the graphical user interaction is a positive interaction (at least one of one or more graphical user interactions is a positive interaction), then one or more first seeds have been determined as described above, and

[0146] - If the graphical user interaction is a negative interaction, then one or more second seeds have been identified as described above.

[0147] Next, defining the specification may include processing data about the location of all simulated graphical user interactions and transforming this data into a segment of data forming the specification. Then, equipping the 3D point cloud with the specification may include associating the specification with the 3D point cloud to form pairs, which form training samples.

[0148] Now let's discuss examples of configurations for learning methods.

[0149] In the example, equipping a 3D point cloud involves adding coordinates to each point in the 3D point cloud based on a simulation of each graphical user interaction. The coordinates quantify the intensity of the selection operation at that point.

[0150] "Based on simulation of each graphical user interaction" means configuring a 3D point cloud taking into account the data regarding the location of each simulated graphical user interaction. In practice, configuring a 3D point cloud here involves processing this data and converting it into a set of coordinates, one coordinate for each point in the 3D point cloud. Configuring a 3D point cloud includes then adding these coordinates to each point, and this results in a specification. In other words, the specification is all the coordinates in that set added to each point in the 3D point cloud.

[0151] This transformation allows, for each point, the coordinates to quantify the strength of the selection operation at that point. Specifically, the coordinates quantify the proximity of the point to each graphical user interaction. This quantifies whether the point belongs to the object to be selected (i.e., the same object). When the neural network performs segmentation, this ultimately allows the neural network to learn to select or not select points as part of an object during training S20. For example, the neural network can be trained to detect that the higher the coordinates, the greater the probability that the point belongs to the object to be selected.

[0152] In the examples, each of one or more graphical user interactions is used to select the same object (e.g., the first graphical user interaction previously discussed) or to discard regions outside the same object (e.g., the second graphical user interaction previously discussed). In these examples, adding coordinates for each point in the 3D point cloud includes:

[0153] - Set the coordinates to initial values;

[0154] - For each graphical user interaction used to select the same object, the coordinates are increased based on the graphical user interaction and the proximity between the points; and

[0155] - For each graphical user interaction used to discard the region outside the same object, the coordinates are reduced based on the proximity between the graphical user interaction and the point.

[0156] Adding coordinates based on the proximity between the graphical user interaction and the point can include quantizing the distance between the location of the point and the location of the graphical user interaction. For example, the graphical user interaction could be a previously discussed first graphical user interaction defined by one or more first seeds. Quantization distances could include each distance between the quantized point and the corresponding first seed. Adding coordinates could then include adding a value representing each quantized distance to the coordinate system.

[0157] Reducing coordinates based on the proximity between graphical user interactions and points can include quantizing the distance between the location of the point and the location of the graphical user interaction. For example, the graphical user interaction could be a previously discussed second graphical user interaction defined by one or more second seeds. The quantization distance could include each distance between the quantized point and the corresponding second seed. Increasing the coordinates could then include adding a value representing each quantization distance to the coordinate system.

[0158] It should be understood that for any point in the 3D point cloud, if the point is too far from a graphical user interaction (GUI), whether for selecting the same object or for discarding a region outside the same object, the point's coordinates neither increase nor decrease. In other words, for each point and for each of one or more GUIs, the increase or decrease in coordinates only occurs when the point is relatively close to the GUI (e.g., within a certain distance from the seed defining the GUI). Therefore, after configuration, the point's coordinates may still be equal to their initial values.

[0159] Adding coordinates in this way has the following effect: the closer a point is to one or more graphical user interactions used to select the same object, the more likely that point belongs to that same object, and its added coordinates have high values. Conversely, the closer a point is to one or more graphical user interactions used to discard regions outside the same object, the more likely that point belongs to those regions, and its added coordinates have low values. Ultimately, this helps the neural network learn in this way that it can detect when graphical user interactions are used to select objects and when they are used to discard regions outside the objects.

[0160] We will now discuss the implementation of a learning method that equips 3D point clouds with a specified implementation.

[0161] In this implementation, for each 3D point cloud, one or more graphical user interactions include one or more previously discussed first graphical user interactions, and for each of the first graphical user interactions, determining one or more seed locations includes distributing one or more first seeds according to the previously discussed distribution implementation. In the implementation currently discussed, one or more graphical user interactions may include one or more previously discussed second graphical user interactions, and if this is the case, for each of the second graphical user interactions, determining one or more second seeds is performed according to the previously discussed implementation of determining one or more second seeds. Therefore, in the implementation currently discussed, one or more graphical user interactions are simulated to generate:

[0162] - For each first graphical user interaction (i.e., each positive interaction), generate one or more first seed positions.

[0163] - For each second graphical user interaction (i.e. each negative interaction), generate one or more second seed positions.

[0164] Furthermore, in this implementation, each seed is a point, as previously described. This implementation includes converting the seeds into a specification, which will now be referred to as an interaction channel. The conversion uses a K nearest neighbor search to retrieve the K nearest points to each seed and assigns specific intensities to these points based on their distance from each seed.

[0165] Specifically, for all points in the 3D point cloud, adding coordinates involves initializing the coordinates with the same value, which is equal to 128. Then, for each seed, adding coordinates involves retrieving the K nearest points to the seed, for example, according to a nearest neighbor search algorithm. Here, the number K is the same as previously discussed with respect to formula (1). The retrieval may include modulating the number of neighbors used in the K nearest neighbor search based on the number of points in the 3D point cloud. This allows the search to remain constant for the point cloud density. Modulation can be performed according to the following formula:

[0166]

[0167] in D This is the default number of points being considered. m It is the selected modulation level. k This is the number of default neighbors considered.

[0168] Experiments show that, for K, this formula is very stable for various point cloud sizes (for a region of 3 * 3 * 5 meters). D = 100000, k = 1024 and m = 128 (tested for points ranging from approximately 10k to 500k). These numbers were chosen in the experiments to correspond to the graph interaction size, which generally corresponds to the size of the smallest object, while being large enough to reduce the number of interactions for large objects. Then, for clarity, the size was chosen based on the number of observed object points and the density of the point cloud for each object in the training dataset. In the experiments, this allowed for finding a way to determine the optimal K that is suitable for all objects and point cloud densities.

[0169] Then, for each seed and each point in the 3D point cloud that belongs to the seed's K neighbors, the coordinates are modified by adding the following steps:

[0170] - If the seed is the first seed, add the Gaussian intensity of the seed to the coordinates, or

[0171] - If the seed is the second seed, subtract the Gaussian intensity of that seed from the coordinate.

[0172] For each seed, the modification can actually include calculating the Gaussian intensity and dynamically modulating it based on the median distance of the K nearest points to the seed. For the first seed (also known as the positive seed), the Gaussian intensity can be calculated according to the following formula:

[0173]

[0174] in ,in It contains the distance of each neighbor from the positive seed and has a size of (K, 1), where K is the number of nearest neighbors.

[0175] For the second seed (also known as the negative seed), the Gaussian intensity can be calculated using the following formula:

[0176]

[0177] in ,in It contains the distance of each neighbor to the negative seed and has a size of (K, 1), where K is the number of nearest neighbors.

[0178] It should be noted that for all points, the added coordinates are initialized to 128. Multiplying the Gaussian intensity by 127 allows for a maximum value of 255 for points matching a positive seed. For those matching a negative seed, the corresponding final point intensity is 1. Therefore, if the point cloud is individually equipped with color channels, this intensity covers the same amplitude as the color channels and allows for a simpler normalization step (normalized between 0 and 1 for each channel by dividing by 255). The dynamic S-parameters, as the median of the neighbor distance array, provide some flexibility for working with different point cloud densities. Contributions from all points matching the seed are added, and the equipment includes clipping coordinates belonging to [0, 255] and linking the clipped coordinates to the other coordinates of that point, and this is true for each point. The interactive channels / specifications are a set of all linked coordinates. For example, for a 3D point cloud with N points having coordinates x, y, and z as unique features, the equipment will produce (N, 3+1) final point cloud dimensions. If color is present, the equipment produces (N, 6+1) dimensions, and so on. In the experiments conducted, using only point coordinates, i.e. without using color, showed very good results.

[0179] As a result, in this implementation, the training dataset consists of training samples, each of which comprises a point cloud and a pair of its interaction channels. As is known from the field of point cloud segmentation itself, each training sample may further include a segmentation mask indicating whether, for each point in the 3D point cloud, that point belongs to the same object. As previously discussed, the segmentation mask can be created during the annotation step of the 3D point cloud.

[0180] Figure 4 Point cloud 40 and its interactive channels are shown. Figure 4 Positive seed 42 and two negative seeds 44 and 46 are shown in particular. Figure 5 The corresponding segmentation mask 50 is shown, which indicates the portion of the point cloud 40 that corresponds to the same object 52 and the portion that does not correspond to the same object 54.

[0181] Now let's discuss how to make the neural network learn S20.

[0182] As previously mentioned, neural networks have weights that form data, defining the operations the neural network applies to an input to process that input. Making the neural network learn S20 involves determining the values ​​of the weights based on the training dataset. Making the neural network learn S20 can be performed according to any known technique. In the example, making the neural network learn S20 is performed according to any known deep neural network (DNN) learning method.

[0183] As is known from the field of machine learning itself, DNNs are a powerful set of techniques for learning within neural networks, a biologically inspired programming paradigm that enables computers to learn from observed data. See, for example, E. Rumelhart, GE Hinton, RJ Williams, “Learning internal representations by error propagation, Parallel distributed processing”: published in The Microstructure of Cognition, vol. 1: Foundations, MIT Press, Cambridge, MA, 1986, which is incorporated herein by reference. In object recognition, the success of DNNs is attributed to their ability to learn rich, intermediate-level media representations, in contrast to the hand-designed low-level features (Zernike moments, HOG, bag-of-words, SIFT, etc.) used in other methods (min-cut, SVM, Boosting, Random Forest, etc.). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, by performing end-to-end optimization starting from raw features and ending with labels, DNNs minimize the need for feature engineering.

[0184] The neural network in this paper can be any known deep learning model capable of performing point cloud segmentation. For example, the neural network could be PointCNN (see [4] Li, Yangyan, Rui Bu, Sun Mingchao, Wei Wu, XinhanDi and Baobao Chen “PointCNN: Convolution On X-Transformed Points.” - NeurIPS2018, which is incorporated herein by reference). The neural network could include millions of parameters whose values ​​cannot be manually set. Learning S20 involves setting / updating the values ​​of these parameters. The model is said to be in “training mode” while the learning algorithm is updating the model parameters. The model continuously “corrects” the model for each input based on its output, with the help of annotations associated with each input. Annotations are a set of data associated with a specific input that allows evaluation of whether the model’s output is right or wrong. Supervising the training of the model with an annotated dataset is called “supervised learning”. After the model is trained, updating the parameters stops. The model is then used only to process new inputs (i.e., inputs not seen during training mode) and return results, and the model is said to be in “test mode”. By enabling neural networks to learn, S20 can integrate all these deep learning concepts, which are well-known in the field of machine learning.

[0185] The neural network is configured to segment an input 3D point cloud, including objects, based on an input 3D point cloud and on a specification of one or more input graphical user interactions, each representing a corresponding selection operation. In other words, the neural network takes the input 3D point cloud equipped with a specification as input. In other words, the neural network takes a segment of data of the same type as any sample in the training dataset as input. The neural network is configured to segment the input 3D point cloud. In other words, based on the input to the neural network, the neural network performs the extraction of objects contained in the input 3D point cloud. The neural network does this because the input 3D point cloud is equipped with a specification and the neural network learns (S20) to detect such a specification. In other words, the specification indicates the location of one or more input graphical user interactions used to select an object, and the neural network detects from this specification that the object should be selected. As mentioned earlier, it should be noted that the neural network is independent of the point cloud category and the object category. The neural network outputs data indicating the difference between the object and the rest of the background (e.g., a segmentation mask).

[0186] For example, a neural network can output a classification result with a corresponding confidence score for each point in an input point cloud. Specifically, in such an example, for each point, the neural network outputs a classification of whether the point is part of an object or not, along with a confidence score. The confidence score is a real number between 0 and 1. The closer the value is to 1, the higher the confidence of the neural network in the label associated with that point. In other words, the neural network output serves as a segmentation mask for objects. In these examples, each training sample also includes a segmentation mask, thus classifying each point in the point cloud as either "object" or "background," as described earlier. In these examples, learning S20 is fully supervised learning.

[0187] The segmentation method is currently being discussed.

[0188] The segmentation method is used for interactive point cloud segmentation.

[0189] Thus, the segmentation method includes providing S30 a 3D point cloud comprising the object. Here, any 3D point cloud used as a training dataset can represent a real-world scene and can originate from physical measurements or a synthetic point cloud generation process. Providing S30 may include displaying the 3D point cloud on a display (e.g., a graphical user interface) of the computer system performing the segmentation method. The 3D point cloud can be displayed to a user, allowing the user to interact with the 3D point cloud graphically, for example, to indicate their interest in the object.

[0190] The segmentation method also includes one or more iterations corresponding to the interactive segmentation of the provided 3D point cloud. Specifically, each iteration includes performing an object selection operation in S40 by executing one or more graphical user interactions. In other words, the user interacts graphically with the provided (e.g., displayed) 3D point cloud one or more times to indicate their interest in objects. Each graphical user interaction relates to the selection of an object, and one or more graphical user interactions together form the object selection operation.

[0191] "Graphical user interaction" as used herein to refer to object selection means any user interaction in which a user uses a tactile system (e.g., a mouse or a touch device such as a sensitive / touchscreen or a sensitive / touchpad) to indicate their interest in an object, such as by selecting one or more locations on a display unit. The one or more locations can form distinct positions, such as one or more clicks. Alternatively, the one or more locations can collectively form a continuous shape, such as a line or bounding box. The one or more locations can be on an object, in which case the graphical user interaction is used to select the object. Alternatively, the one or more locations can be on an area outside the object (e.g., another object or background), in which case the graphical user interaction is used to discard said area. Visual indications of the selection can be displayed substantially in real-time after activation.

[0192] In the example, one or more graphical user interactions may include a graphical user interaction that performs one or more clicks on an object. Performing one or more clicks can be done using a haptic device, for example, a user clicking or touching a displayed 3D point cloud once or multiple times at one or more locations on the object. Because this graphical user interaction involves selecting one or more locations on the object (i.e., the locations of the one or more clicks), it is used for selecting objects.

[0193] Alternatively or concurrently, one or more graphical user interactions may include a graphical user interaction that performs drawing on an object. Performing drawing may include a user drawing a line on a displayed 3D point cloud, for example, by continuously moving a cursor on the object according to a line using a haptic device or by continuously performing touches on the object according to a line. Since this graphical user interaction involves selecting one or more locations on the object (i.e., locations clicked once or multiple times), this graphical user interaction is used for selecting objects.

[0194] Alternatively or concurrently, one or more graphical user interactions may include graphical user interactions that define bounding boxes on and / or around objects. Defining the bounding box may include the user drawing the bounding box on a displayed 3D point cloud, for example, by continuously moving the cursor of a touch or haptic device to form the bounding box. Since this graphical user interaction involves selecting one or more locations on the object (i.e., the location of one or more clicks), this graphical user interaction is used for selecting objects.

[0195] Alternatively or concurrently, one or more graphical user interactions may include graphical user interactions that involve performing one or more clicks outside an object. Performing one or more clicks may be performed using a haptic device, such as a user clicking or touching a displayed 3D point cloud one or more times at one or more locations outside the object. Because this graphical user interaction involves selecting one or more locations on that area (i.e., the locations of the one or more clicks), it serves to discard areas outside the object. One or more locations may be located, for example, on adjacent areas outside the object to specify object boundaries. Alternatively, one or more locations may be located on adjacent objects to designate those adjacent objects as background objects.

[0196] Alternatively or concurrently, one or more graphical user interactions may include a graphical user interaction that performs drawing on the exterior of an object. Performing drawing may include a user drawing a line on a displayed 3D point cloud, for example, by continuously moving a cursor on the 3D point cloud according to a line using a haptic device or by continuously performing touches on the 3D point cloud according to a line. Since this graphical user interaction involves selecting a location on the area (i.e., the location defined by the line), it is used to discard areas outside the object. These locations may, for example, be on adjacent areas outside the object to specify object boundaries. Alternatively, these locations may be on adjacent objects to specify those adjacent objects as background objects.

[0197] One or more graphical user interactions can perform actions on each other: for example, one of these interactions might be used for coarse selection of an object (e.g., a single click on the object), while another interaction might be used to refine the selection by discarding adjacent regions (e.g., a single click on an adjacent region of the object). Collectively, one or more interactions form a selection operation on the object. This selection operation is specified by encapsulating data at one or more locations defined by the one or more interactions. In other words, this data forms a specification for the one or more interactions. This specification is the same type of data as the specification provided in the training dataset, as discussed previously. The main difference is that in the context of segmentation methods, this specification originates from real graphical user interactions, while in the context of learning methods, it may originate from simulated virtual graphical user interactions, as previously discussed.

[0198] Based on this specification and the input 3D point cloud, the segmentation method then includes segmenting the 3D point cloud by applying an S50 neural network. As previously described, the neural network takes the specification and the 3D point cloud as input and performs operations on these inputs, resulting in the neural network extracting objects from the 3D point cloud. For example, as previously discussed, the neural network can output a classification of the points in the 3D point cloud as "objects" or "non-objects" along with a confidence level.

[0199] In each iteration, the segmentation method may include: after performing the selection operation and before applying the neural network, converting the selection operation into an interactive channel. Here, as an example of a learning method, this interactive channel is a set of coordinates, each of which is added to a corresponding point in the 3D point cloud. Converting the selection operation into an interactive channel includes:

[0200] - The specification is determined by determining the location of one or more seed locations defined by each graphical user interaction for one or more graphical user interactions; and

[0201] - Configure the 3D point cloud with a specification that includes adding coordinates that quantify the intensity of the selection operation at that point to each point in the 3D point cloud.

[0202] Each graphical user interaction defines geometry on a 3D point cloud (e.g., one or more clicks, lines, or bounding boxes) and generates data about the location of this geometry. This data can be detected using any known method for detecting the location of a graphical user interaction. The segmentation method determines one or more seed locations from this data, which together represent the geometry. For simplicity, each seed location will be referred to as a "seed." Determining one or more seeds can include calculating one or more points of the geometry as one or more seeds, meaning that in this case, each seed is a point. Seeds on an object are called positive seeds, while seeds on regions outside the object are called negative seeds. A graphical user interaction can define both positive and negative seeds.

[0203] In the example, if the graphical user interaction involves performing one or more clicks, determining one or more seeds can include identifying each click as a seed. Clicks on an object are identified as positive seeds, while clicks on areas outside the object are identified as negative seeds.

[0204] Alternatively, if the graphical user interaction involves performing line drawing, determining one or more seeds may include sampling the line into multiple points by acquiring points below the line. The seed is then the sampled points. If the sampled points are on an object, it is a positive seed; if the sampled points are on a region outside the object, it is a negative seed. The number of sampled points may be proportional to the length of the line drawing. Note that this matches the simulated graphical user interaction in the example providing the S10 training dataset, where, for positive seeds, the larger the object, the larger the line drawing performed on the object may be, and the more seeds are determined.

[0205] Alternatively or concurrently, if the graphical user interaction involves defining a bounding box, determining one or more seeds may include sampling the bounding box into points inside and outside the bounding box. Points inside the bounding box are determined as positive seeds, while points outside are determined as negative seeds. The number of positive seeds may be proportional to the number of points inside the bounding box and may be randomly selected from among the points inside the bounding box. Negative seeds may be randomly selected from points contained in a small region around the bounding box (e.g., less than 30 cm). Note that this corresponds to the graphical user interaction simulated according to the first strategy discussed for the learning method.

[0206] The segmentation method is still discussed. The specification for equipping 3D point clouds includes adding coordinates to each point of the 3D point cloud, which quantifies the intensity of the selection operation at that point.

[0207] The configuration of segmentation methods has been discussed.

[0208] Fitting a 3D point cloud involves considering one or more deterministic seeds and converting those seeds into a set of coordinates, one coordinate for each point in the 3D point cloud. Fitting the 3D point cloud then involves adding these coordinates to each point, and this results in a specification. In other words, a specification is a set of all coordinates added to each point in the 3D point cloud.

[0209] This transformation enables the coordinates to quantify the strength of the selection operation at each point. Specifically, the coordinates quantify the proximity of the point to each graphical user interaction. This quantifies whether the point belongs to the object to be selected (i.e., the same object). When the neural network performs segmentation, this ultimately allows the neural network to extract / select or not extract / select points as part of an object during the application of S50. For example, the neural network detects that the higher the coordinates, the greater the probability that the point belongs to the object to be selected.

[0210] In the example, the addition includes setting the coordinates to the same initial value for all points. Then, for each determined positive seed, the addition may include increasing the coordinates for each point in the point cloud based on the proximity between the seed and the point. For each determined negative seed, the addition may also include decreasing the coordinates for each point in the point cloud based on the proximity between the seed and the point.

[0211] Increasing (decreasing) coordinates based on the proximity of a positive (and negative) seed to a point can include quantizing the distance between the point's location and the seed. Then, increasing (decreasing) coordinates can include adding (and subtracting) a value representing the quantized distance to the coordinates.

[0212] It should be understood that for any point in a 3D point cloud, if the point is too far from the seed, whether it is a positive or negative seed, the point's coordinates will neither increase nor decrease. In other words, for each point and each given seed, the increase or decrease in coordinates only occurs when the point is relatively close to the graphical user interaction (e.g., within a certain distance from the seed). Therefore, after configuration, the coordinates of a point may still be equal to the initial values.

[0213] Adding coordinates in this way has the following effect: the closer a point is to a positive seed, the more likely that point belongs to the same object, and the larger the value of its added coordinates. Conversely, the closer a point is to a negative seed, the more likely that point does not belong to that object, and the lower the value of its added coordinates. Ultimately, this allows neural networks to detect when graphical user interactions are used to select objects and when they are used to discard regions outside of objects.

[0214] We will now discuss the implementation of equipping 3D point clouds with specified features.

[0215] In this implementation, the determination of a specification results in one or more positive seeds and one or more negative seeds. Furthermore, in this implementation, each seed is a point, as discussed previously. This implementation includes converting the seeds into specifications, now referred to as interaction channels. The conversion uses a K nearest neighbor search to retrieve the K nearest points to each seed and assigns specific intensities to these points based on the distances of those K nearest points to each seed.

[0216] Specifically, for all points in the 3D point cloud, adding coordinates involves initializing the coordinates with the same value, which is equal to 128. Then, for each seed, adding coordinates involves obtaining K points closest to the seed, for example, according to a nearest neighbor search algorithm. Here, the number K is the same as previously discussed with respect to formula (1). The retrieval may include modulating the number of neighbors used in the K nearest neighbor search based on the number of points in the 3D point cloud. This allows the search to remain constant for the point cloud density. Modulation can be performed according to the following formula:

[0217]

[0218] in, D This is the default number of points being considered. m It is the selected modulation level. k This is the number of default neighbors considered.

[0219] Experiments show that the formula is proven to be very stable for various point cloud sizes for K (points ranging from about 10k to 500k were tested for a 3 * 3 * 5 meter area with D = 100000, k = 1024 and m = 128).

[0220] Then, for each seed and each point in the 3D point cloud that belongs to the seed's K neighbors, the coordinates are added, including modifying the coordinates in the following ways:

[0221] - If the seed is the first seed, add the Gaussian intensity from the seed to the coordinates, or

[0222] - If the seed is the second seed, subtract the Gaussian intensity from the seed from the coordinates.

[0223] For each seed, the modification can actually include calculating the Gaussian intensity and dynamically modulating it based on the median distance of the K nearest points from the seed. The Gaussian intensity can be calculated for the first seed (also known as the positive seed) according to the following formula:

[0224]

[0225] in, ,in It contains the distance of each neighbor from the positive seed and has a size of (K, 1), where K is the number of nearest neighbors.

[0226] The Gaussian intensity can be calculated for the second seed (also known as the negative seed) using the following formula:

[0227]

[0228] in, ,in It contains the distance of each neighbor to the negative seed and has a size of (K,1), where K is the number of nearest neighbors.

[0229] It's important to note that for all points, the coordinates are initialized to 128. For points matching a positive seed, the Gaussian intensity is multiplied by 127, allowing a maximum value of 255. For those matching a negative seed, the corresponding final point intensity is 1. Therefore, if the point cloud is equipped with color channels, this intensity covers the same amplitude as the color channels and allows for a simpler normalization step (normalized between 0 and 1 for each channel by dividing by 255). The dynamic S-parameters, as the median of the neighbor distance array, provide some flexibility for working at different point cloud densities. Contributions from all points matching the seed are added, and the equipment includes clipping coordinates belonging to [0, 255] and linking them to the other coordinates of that point, and this is per point. The interaction channel / specification is a set of all linked coordinates. For example, for a 3D point cloud with N points having coordinates x, y, and z as unique features, the equipment will produce (N, 3+1) final point cloud dimensions. If color is present, the equipment produces (N, 6+1) dimensions, and so on. In the experiments conducted, using only point coordinates, i.e. without using color, showed very good results.

[0230] As a result, in this implementation, the 3D point cloud is equipped with an interaction channel. The pair consisting of the 3D point cloud and its interaction channel is then fed as input into the neural network to perform segmentation.

[0231] Prior to this, in a specific implementation of this approach, the implementation may include pruning the point cloud around the centroid of the positive seed. Replacing the entire point cloud with this region can reduce runtime. This specific implementation may also include uniform sampling to replicate points in low-density regions of the point cloud. This specific implementation may also include dividing the points of the 3D point cloud into a set of 2048 points to match the input size of PointCNN, which is the neural network in this specific implementation. As discussed earlier, in this specific implementation, the neural network then outputs predictions for these points. The segmentation method may include using network-based confidence scores to merge predictions. The segmentation method may also optionally include post-processing to smooth predictions that may appear relatively sparse. Smoothing can use a voxel-based method that relies on the confidence scores output by the neural network. In each voxel, smoothing compares the mean confidence score for an object category across all points contained in that object category with a fixed threshold. This smoothing allows indicating the voxel category and labeling all points contained in that voxel category accordingly. One advantage of this post-processing method is its time efficiency.

[0232] Now let's discuss experimental examples of the segmentation method.

[0233] In this experiment, the neural network is trained according to an example learning method. As previously discussed, the training dataset used for learning consists of a synthetic point cloud. In the example of the learning method used to train the neural network, the learning method includes the previously discussed preparation of the 3D point cloud for that learning method. The experiment then includes various implementations of the segmentation method.

[0234] Figure 6 and 7 The first implementation is shown. (See figure) Figure 6 As shown, a 3D point cloud 60, including object 62, is displayed to the user. The user performs multiple clicks on object 62 (620, 622, 624, and 626), each click generating a positive seed. This is achieved by applying an S50 neural network, such as... Figure 7 As shown, object 62 is extracted.

[0235] Figure 8 and 9 A second implementation is shown. (e.g.) Figure 8 As shown, a 3D point cloud 80, including object 82, is displayed to the user. The user performs multiple clicks 824 and 822 on the point cloud 80. Click 822 generates a negative seed, while click 824 generates a positive seed. By applying an S50 neural network, such as Figure 9 As shown, object 82 was extracted.

[0236] Figure 10 and 11 A third implementation is shown. (e.g.) Figure 10As shown, a 3D point cloud 100, including object 102, is displayed to the user. The user performs multiple clicks 1022 and 1024 on object 102, each click generating a positive seed. By applying an S50 neural network, such as Figure 11 As shown, object 102 is extracted.

[0237] Figure 12 and 13 The fourth implementation is shown. (See example.) Figure 12 As shown, a 3D point cloud 120, including object 122, is displayed to the user. The user performs multiple clicks 1222 and 1224 on object 122, with each click generating a positive seed. By applying an S50 neural network, such as Figure 13 As shown, object 122 is extracted.

[0238] Figure 14 A flowchart illustrating an example of the processes of ensemble learning and segmentation methods is shown. Figure 14 As shown, this process example performs the following steps:

[0239] 1. Offline Phase: The purpose of this phase is to train a model using a point cloud containing pairs of user indicators / segmentation masks that closely approximate real-world data and user behavior. This phase consists of two main steps. Note that this phase is transparent to the user.

[0240] 1) Generation of training dataset.

[0241] i. Providing point clouds and collecting consistent indicators from real users can be very difficult and time-consuming, and cannot guarantee coverage of diverse user behaviors. In this example, the training indicators are automatically generated by defining similar user behaviors. The training user indicators include:

[0242] - Positive training indicator, which includes at least one point in the training point cloud, the at least one point being part of an identified target object in the same point cloud.

[0243] - Negative training indicator, which includes at least one background point in the training point cloud that is not part of the identified target object in the same point cloud.

[0244] - Please note that at least one positive indicator must be generated for each target object.

[0245] ii. The generated indicators are converted into additional channels (i.e., interactive channels), which are added to the original point cloud as input to the neural network, thereby transforming the input into a format that can be fed into the neural network. The resulting training dataset then comprises a rich point cloud containing both training indicators and segmentation masks for the target objects.

[0246] 2) The neural network model learns based on the training dataset. This learning consists of a DNN-based point cloud model.

[0247] 2. Online Phase: Given a point cloud and user indicators, the data is first preprocessed to transform the indicators. Then, the trained model is applied to finally obtain the segmentation mask.

[0248] Figure 15 A flowchart illustrating the implementation of this process is shown.

[0249] Learning methods and segmentation methods are implemented by computers.

[0250] This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the execution of the steps by a computer may be fully automatic or semi-automatic. In the example, triggering of at least some steps in the method can be performed through user computer interaction. The required level of user computer interaction can depend on anticipating the level of automation and balancing it with the need to fulfill the user's wishes. In the example, this level can be user-defined and / or predefined.

[0251] A typical example of a computer implementation of the method is to use a system suitable for this purpose to execute the method. This system may include a processor coupled to memory and a graphical user interface (GUI), on which a computer program is recorded, containing instructions for executing the method. The memory may also store a database. The memory is any hardware suitable for such storage and may comprise multiple physically distinct parts (e.g., one part for the program and another for the database).

[0252] Figure 16 An example of a system is shown, where the system is a client computer system, such as a client's workstation.

[0253] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer is also equipped with a graphics processing unit (GPU) 1110, which is associated with video RAM 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to a mass storage device (e.g., a hard disk drive 1030). Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the above can be supplemented or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used on the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive tablet and / or a sensitive screen.

[0254] Any computer program described herein may include computer-executable instructions, including means for causing the system described above to perform one or more methods. The program may be recorded on any data storage medium, including the system's memory. The program may be implemented, for example, as digital electronic circuitry or computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing the instruction program to perform the function of the method by manipulating input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from the data storage system, at least one input device, and at least one output device, and to transfer data and instructions to the data storage system, at least one input device, and at least one output device. If desired, the application program may be implemented in a high-level procedural or object-oriented programming language, assembly language, or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation or update program. In any case, application of the program to the system results in instructions to perform one or more methods.

Claims

1. A computer-implemented machine learning method, the method comprising: A dataset of 3D point clouds is provided, each 3D point cloud including at least one object, each 3D point cloud being equipped with a specification for one or more graphical user interactions, each of the one or more graphical user interactions representing a corresponding selection operation on the same object in the 3D point cloud, wherein the specification defines the selection operation on the same object through one or more graphical user interactions; and The neural network is trained on the dataset and is configured to segment an input 3D point cloud including objects. The segmentation is based on the input 3D point cloud and on a specification of one or more input graphical user interactions, each of which represents a corresponding selection operation on the objects in the 3D point cloud. The dataset providing the 3D point cloud includes: Provide the 3D point cloud, and for each 3D point cloud, provide information about the position of the same object in the 3D point cloud; and For each 3D point cloud: The specifications are determined by simulating each graphical user interaction in the one or more graphical user interactions; and Equip the 3D point cloud with the specified rules for the one or more graphical user interactions.

2. The method according to claim 1, wherein, Each of the one or more graphical user interactions corresponds to one or more seed locations defined on the 3D point cloud, and simulating the graphical user interaction includes determining the one or more seed locations.

3. The method according to claim 2, wherein, The one or more graphical user interactions include a first graphical user interaction, which is used to select the same object and corresponds to one or more first seed positions customized on the same object.

4. The method according to claim 3, wherein, Determining one or more seed locations includes distributing the one or more first seed locations across the same object.

5. The method according to claim 3 or 4, wherein, For at least one 3D point cloud, the one or more graphical user interactions further include a second graphical user interaction, which is used to discard regions outside the same object and corresponds to one or more custom second seed locations outside the same object.

6. The method according to any one of claims 1 to 4, wherein, Equipping the 3D point cloud includes adding coordinates that quantify the intensity of the selection operation at each point to each point of the 3D point cloud based on simulating each graphical user interaction.

7. The method according to claim 5, wherein, Equipping the 3D point cloud includes adding coordinates that quantify the intensity of the selection operation at each point to each point of the 3D point cloud based on simulating each graphical user interaction.

8. The method according to claim 6, wherein, Each of the one or more graphical user interactions is used to select the same object or to discard a region outside the same object, and for each point in the 3D point cloud, adding the coordinates includes: Set the coordinates to initial values; For each graphical user interaction used to select the same object, the coordinates are increased based on the proximity between the graphical user interaction and the point; and For each graphical user interaction used to discard a region outside the same object, the coordinates are reduced based on the proximity between the graphical user interaction and the point.

9. A neural network product capable of learning using the method according to any one of claims 1 to 8.

10. A method of use, comprising a computer implementation of the neural network product according to claim 9, the method comprising: Provides 3D point clouds of objects; as well as One or more iterations of the following process: The selection operation on the object is performed by executing one or more graphical user interactions; as well as By applying the neural network product, the 3D point cloud is segmented based on the 3D point cloud and based on the specifications of the one or more graphical user interactions, wherein the specifications specify a selection operation on the same object through one or more graphical user interactions.

11. The method of use according to claim 10, wherein, The method further includes, after performing the selection operation and before applying the neural network product: The specification is determined by determining the position of one or more seed positions defined by the graphical user interaction for each of the one or more graphical user interactions; and Equip the 3D point cloud with the specified rules, which include adding coordinates that quantify the intensity of the selection operation at each point in the 3D point cloud to the point, wherein the specified rules specify the selection operation on the same object through one or more graphical user interactions.

12. The method of use according to claim 10 or 11, wherein, The one or more graphical user interactions include one or more of the following: Perform one or more clicks on the object; Perform a line drawing on the object; Define a bounding box on and / or around the object; Perform one or more clicks outside the object; and / or Perform a line drawing on the outside of the object.

13. A computer program product comprising instructions for performing the method according to any one of claims 1 to 8 and / or the method of use according to any one of claims 10 to 12.

14. An apparatus comprising a data storage medium on which the computer program product of claim 13 and / or the neural network product of claim 9 are recorded.

15. The apparatus of claim 14, further comprising a processor coupled to the data storage medium.

Images