Machine learning for 3D object detection

Abstract

The invention relates in particular to a computer-implemented method of machine learning for learning a neural network configured for encoding hyperpoints in a 3D point cloud into eigenvectors. The method comprises providing a dataset of hyperpoints. Each hyperpoint is a set of points in a 3D point cloud. The set of points represents at least a portion of an object. The method further comprises learning the neural network based on the dataset of hyperpoints. The learning comprises minimizing a loss. The loss penalizes inconsistencies between two hyperpoints. This constitutes an improved method of machine learning for 3D object detection.

Description

Technical Field

[0001] This invention relates to the field of computer programs and systems, and more particularly to methods, systems and programs for machine learning in 3D object detection. Background Technology

[0002] Numerous systems and programs are available on the market for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, which, for example, involves software solutions for designing objects. CAE stands for Computer-Aided Engineering, which, for example, involves software solutions for simulating the physical behavior of future products. CAM stands for Computer-Aided Manufacturing, which, for example, involves software solutions for defining manufacturing processes and operations. In such computer-aided design systems, graphical user interfaces play a crucial role in technical efficiency. These technologies can be embedded within Product Lifecycle Management (PLM) systems. PLM refers to a concept that extends across the enterprise to help companies share product data, apply common processes, and utilize enterprise knowledge for product development from concept to the end of its lifecycle. PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA, and DELMIA) provide an Engineering Hub for organizing product engineering knowledge, a Manufacturing Hub for managing manufacturing engineering knowledge, and an Enterprise Hub that enables enterprises to integrate and connect to both the Engineering Hub and the Manufacturing Hub. Together, the system provides an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, which drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this context and others, machine learning and 3D object detection are becoming increasingly important.

[0004] However, there is still a need for improved machine learning methods for 3D object detection. Summary of the Invention

[0005] Therefore, a computer-implemented method for machine learning is provided for learning a neural network configured to encode superpoints in a 3D point cloud as eigenvectors. The method includes providing a dataset of superpoints. Each superpoint is a set of points in the 3D point cloud. The set of points represents at least a portion of an object. The method also includes learning the neural network based on the dataset of superpoints. The learning includes minimizing a loss. The loss penalizes inconsistencies between two superpoints.

[0006] The method may include one or more of the following:

[0007] - The loss is a reconstruction loss, which penalizes the inconsistency between the reconstruction of the superpoint and the superpoint.

[0008] - The inconsistency is the distance between the superpoint and the reconstruction of the superpoint;

[0009] - The distance is either the Chamfer distance or the Earth-Mover distance; and / or

[0010] The learning described is unsupervised learning.

[0011] A neural network that can learn according to the method described is also provided.

[0012] A method of using a computer implementation of the neural network is also provided. The method includes providing one or more first superpoints of a first 3D point cloud. The first 3D point cloud represents a 3D scene. The method also includes providing one or more second superpoints of a second 3D point cloud. The second 3D point cloud represents a 3D object. The method further includes: encoding each of the one or more first superpoints as a corresponding first eigenvector by applying the neural network, and encoding each of the one or more second superpoints as a corresponding second eigenvector. The method further includes determining the similarity between each of the one or more first superpoints and each of the one or more second superpoints by: for each first superpoint and each second superpoint, calculating a similarity metric between the corresponding first eigenvector encoded for the first superpoint and the corresponding second eigenvector encoded for the second superpoint.

[0013] The method of use may include one or more of the following:

[0014] -The provision of the one or more first superpoints includes:

[0015] • Provide one or more initial superpoints in the first 3D point cloud;

[0016] • The one or more initial superpoints are filtered by selecting each of the following initial superpoints: for each initial superpoint:

[0017] ○ The inconsistency between the dimension of the superpoint and the dimension of at least one second superpoint is less than a predefined threshold; and / or

[0018] ○ The inconsistency between the position of the superpoint and the position of at least one second superpoint is less than a predefined threshold.

[0019] The selected superpoint is one or more of the first superpoints;

[0020] - The filtering includes selecting each of the initial hyperpoints as follows, for each initial hyperpoint:

[0021] • The distance between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than a predefined threshold; and / or

[0022] • The ratio between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than the maximum ratio and greater than the minimum ratio; and / or

[0023] • The difference between the relative height of the nearest support surface to the superpoint and the relative height of the nearest support surface to at least one second superpoint is less than a predefined threshold;

[0024] - The method further includes: determining one or more groups of first superpoints among first superpoints that all have a determined similarity to at least one second superpoint greater than a predefined threshold, each group of first superpoints having a shape similar to the second 3D point cloud;

[0025] -The determination of the one or more groups includes:

[0026] • Provide a graph of a second superpoint, the graph of the second superpoint having nodes and edges, each node representing a second superpoint, each edge representing a geometric relationship between two superpoints represented by nodes connected by the edge and having one or more geometric properties of the geometric relationship;

[0027] • One or more graphs are formed by the following operations, each graph having nodes, each node representing each first superpoint: constructing edges, each edge connecting two nodes, and each edge having one or more geometric properties similar to the edges of the graph of the second superpoint, each formed graph corresponding to a corresponding group; and / or

[0028] The method further includes: for each group, determining a similarity score by measuring the similarity between the first superpoint in the group and the one or more second superpoints.

[0029] A computer program including instructions for performing the method and / or the usage method is also provided.

[0030] An apparatus including a data storage medium is also provided, wherein the computer program and / or the neural network are stored thereon.

[0031] The device can form or act as a non-transitory computer-readable medium, such as on a SaaS (Software as a Service) or other server, or a cloud-based platform. The device may alternatively include a processor coupled to the data storage medium. The device thus forms a computer system, either entirely or partially (e.g., the device is a subsystem of an overall system). The system may also include a graphical user interface coupled to the processor. Attached Figure Description

[0032] Embodiments of the invention will now be described by way of non-limiting example and with reference to the accompanying drawings, in which:

[0033] Figure 1 A flowchart illustrating an example of the process of integrating the method is shown;

[0034] Figures 2 to 18 The method is illustrated; and

[0035] Figure 19 An example of the system is shown. Detailed Implementation

[0036] A computer-implemented method for machine learning is proposed for learning a neural network configured to encode superpoints in a 3D point cloud as eigenvectors. The method includes providing a dataset of superpoints. Each superpoint is a set of points in the 3D point cloud. The set of points represents at least a portion of an object. The method also includes learning the neural network based on the dataset of superpoints. The learning includes minimizing a loss. The loss penalizes inconsistencies between two superpoints. This method may be referred to as a "learning method".

[0037] The learning method described herein forms an improved approach to machine learning for 3D object detection.

[0038] It is noteworthy that the learning method enables 3D object detection within 3D point clouds. The neural network learned through this method is particularly applicable to numerous applications using 3D object detection, such as any application using 3D object detection according to the usage method discussed below. These applications can be (e.g., automatically) inventory counting and / or management (e.g., in a factory context, locating all robots to move or replace the robots or some of the robots), (e.g., automatically) object removal and / or replacement (e.g., in a factory context, automatically replacing all scans of similar machines), and / or (e.g., in a factory context) virtual twin generation. Virtual factory twins exist within a CAD context or a hybrid of a CAD context and 3D point clouds, the virtual factory representing, for example, a twin of a scanned or physically actual factory. Ultimately, virtual twin generation of a factory context can allow for the simulation and / or various optimizations of the factory.

[0039] Furthermore, the learning method learns a neural network capable of taking superpoints in a 3D point cloud as input and encoding the superpoints as eigenvectors. In other words, the learning method learns a neural network capable of projecting any 3D superpoint in a 3D point cloud onto its eigenvector representation. Since the neural network is trained / learned by minimizing the loss that penalizes inconsistencies between two superpoints, the neural network encodes the input superpoints as eigenvectors, which are meaningful representations of the superpoints in terms of geometric information.

[0040] In practice, the loss takes two superpoints as input and calculates the inconsistency between these two superpoints. Because the loss penalizes this inconsistency, it tends to have a large value when the inconsistency is high. Thus, during the learning process, for each superpoint used as a training sample, the superpoint is fed as input to the loss, and another superpoint, obtained from the eigenvector encoded as a training sample (i.e., the eigenvector obtained by applying the neural network to the superpoint), is also fed as input to the loss. In other words, the loss penalizes the inconsistency between the training sample superpoint and the other superpoint. The other superpoint can, for example, be a reconstruction of the superpoint by decoding the eigenvector encoded as a training sample. In this case, the neural network includes an encoder for encoding the superpoint as an eigenvector and a decoder for decoding the eigenvector as a superpoint, as discussed further below. This is, for example, the case when the neural network is an autoencoder. The learning process then calculates a loss value for the training sample superpoint and the other superpoint, with the loss taking a pair of superpoints as input. This value of the loss quantifies the geometric inconsistency between the superpoints (here, the superpoints used as training samples and another superpoint) that are input to the loss. Therefore, the geometric inconsistency is the inconsistency between the superpoint used as a training sample and the decoding of its encoded eigenvector. Since the loss penalizes such inconsistency, the value is large when the inconsistency is large. Because the learning minimizes the loss, the learning modifies the weights of the neural network when the value is large, since a large value captures the fact that the encoding of the superpoint used as a training sample is not sufficiently satisfactory. In other words, the eigenvectors encoding the superpoint used as a training sample do not accurately capture the geometry of the superpoint when the value is high, therefore the weights need to be modified to improve this accuracy.

[0041] Therefore, the learning according to the aforementioned learning method often ensures that superpoints with similar geometries are encoded using similar eigenvectors, and that two superpoints with different geometries are encoded using significantly different eigenvectors. In other words, the neural network is trained to project superpoints onto eigenvector representations that represent meaningful geometric information; that is, the learning aims to make the distance between the eigenvectors encoding two superpoints highly correlated with the geometric similarity of those two superpoints. Now, since superpoints in a 3D point cloud are subsets of points in the 3D point cloud that describe local portions of the 3D point cloud (e.g., local shapes, forming at least a portion of objects in the 3D point cloud), the neural network learned by the aforementioned learning method allows the encoding of local portions of the 3D point cloud into geometrically meaningful eigenvector representations.

[0042] Therefore, the neural network is a neural network capable of being used to perform 3D object detection in a 3D point cloud. Thus, the learning method is a machine learning method for 3D object detection. For example, given a first 3D point cloud including one or more first superpoints and a second 3D point cloud including one or more second superpoints, the neural network can encode the superpoints into eigenvectors, which can then be used to compute a similarity metric between the eigenvectors. This allows determining which first superpoints are geometrically similar to the second superpoints, which allows detecting local shapes in the first point cloud that are similar to the local shapes of the 3D object represented by the second point cloud. Ultimately, these similar local shapes can be used / processed to output 3D objects in the first 3D point cloud that are similar to the 3D object represented by the second 3D point cloud (i.e., formed by one or more detected local shapes). For example, if the first 3D point cloud represents a factory / manufacturer and the second 3D point cloud represents a 3D articulated robot, the neural network can be used to detect (e.g., all) 3D articulated robots in the factory that are similar to the 3D articulated robot represented by the second 3D point cloud. Furthermore, because of the learning operated by the aforementioned learning method, the detection is general: the neural network not only detects exact instances of the second 3D point cloud in the first 3D point cloud, but also detects objects in the first 3D point cloud that are partially similar to the object represented by the second 3D point cloud. In other words, given a single 3D object represented by a 3D point cloud, the neural network enables the localization of partially similar 3D objects in a larger 3D point cloud (e.g., a 3D point cloud with more than 20 times the number of points), considering factors such as geometric transformations (e.g., rotations or positional changes) of instances of the single 3D object, occlusion of instances of the single 3D object, and other types of 3D objects that are geometrically similar to the single 3D object.

[0043] Furthermore, the learning does not require any labeling of the training dataset. For example, the learning can be unsupervised learning, as is well known in the field of machine learning. This allows the learned neural network to be unaware of the category / type of the 3D object to be detected and the context / background of the detection (e.g., whether the object(s) to be detected are in an indoor scene or an outdoor scene). For example, once trained, the neural network can be used to detect any object in any context, such as objects not seen during the learning process and / or any object in a context not seen during the learning process. Additionally, the neural network is unaware of the type of point cloud to which the input superpoint belongs (i.e., no constraints on point cloud acquisition are required for effective detection). This collectively makes the neural network general and particularly robust for 3D object detection in 3D point clouds. The neural network can, for example, be trained on synthetic point clouds and then used to perform 3D object detection on real-world 3D point clouds, i.e., on 3D point clouds acquired by physical sensors.

[0044] Furthermore, the representation based on local learning (i.e., the learning trains the neural network to encode superpoints representing local shapes) makes the neural network robust to noise, occlusion, and point distribution, and invariant to different types of transformations (e.g., rotation) and object positions (e.g., articulated objects, such as articulated robots characterized by articulated robotic arms). This makes the neural network more robust for 3D object detection in 3D point clouds. It should be noted that the neural network can be used to detect objects in 3D point clouds that are partially similar to a given 3D object (i.e., not exact instances of it) (i.e., queries discussed below). Moreover, the neural network does not rely on the object's color for detection: even if the neural network has different colors, it detects similar shapes.

[0045] A neural network capable of learning according to the learning method is also provided, for example, a neural network already learned by the learning method. The neural network forms a computerized data structure with neuron layers, each neuron having weights (also referred to as parameters), the values ​​of which can be set (e.g., already set) according to the learning according to the learning method.

[0046] A method of using a computer implementation of the neural network is also provided. The method includes providing one or more first superpoints of a first 3D point cloud. The first 3D point cloud represents a 3D scene. The method also includes providing one or more second superpoints of a second 3D point cloud. The second 3D point cloud represents a 3D object. The method further includes: encoding each of the one or more first superpoints as a corresponding first eigenvector and encoding each of the one or more second superpoints as a corresponding second eigenvector by applying the neural network. The method further includes determining the similarity between each of the one or more first superpoints and each of the one or more second superpoints by: for each first superpoint and each second superpoint, calculating a similarity metric between the corresponding first eigenvector encoded for the first superpoint and the corresponding second eigenvector encoded for the second superpoint. The method may be referred to as a "detection method".

[0047] The detection method is a method for 3D object detection in a 3D point cloud. This detection method forms an improved method for 3D object detection in a 3D point cloud. Specifically, the method allows the detection of first superpoints representing a 3D scene in the first 3D point cloud, which are geometrically similar to second superpoints representing 3D objects in the second 3D point cloud. In other words, each of the second superpoints represents a local portion of the object, and the detection method detects local portions in the first point cloud that are geometrically similar to these local portions of the object (i.e., the first superpoints). This allows the detection of 3D objects in the 3D scene represented by the first 3D point cloud that are similar to 3D objects represented by the second 3D point cloud.

[0048] To detect similar 3D objects, the detection method uses the neural network to encode the hyperpoints into eigenvectors and then determines a similarity metric between the encoded hyperpoints. Thus, the detection method benefits from the robustness of the neural network, which, as previously explained, projects input hyperpoints onto geometrically meaningful eigenvector representations. The determination of the similarity between two hyperpoints is accurate because it computes a similarity metric between the eigenvectors encoded for the two hyperpoints, which, as previously discussed, are geometrically meaningful representations of the two hyperpoints. In other words, if similarity is detected between the eigenvectors, it implies a corresponding geometric similarity between the two hyperpoints. Therefore, the detection method allows for robust detection of similar 3D objects in a 3D point cloud.

[0049] The learning method and the detection method can be executed independently by, for example, different users and / or on different computers. Alternatively, they can be integrated into a process implemented on the same computer. (See now for further details.) Figure 1 To discuss the process described. Figure 1 A flowchart illustrating an example of the process is shown.

[0050] The process includes an offline phase that integrates the learning method. The offline phase includes providing a dataset of superpoints S10-S20 according to the learning method. The offline phase also includes learning the neural network S20 based on the superpoints according to the learning method. The offline phase may further include storing the learned neural network in memory.

[0051] The process further includes an online phase that integrates the detection method. The online phase includes providing one or more first superpoints and one or more second superpoints according to the detection method (S40-S50). Prior to this, the online phase may include providing a learned neural network, for example, by retrieving the learned neural network from memory after learning in the offline phase and storing it in the memory. The online phase also includes encoding the first and second superpoints according to the detection method (S60). The online phase further includes determining the similarity between the one or more first superpoints and the one or more second superpoints according to the detection method (S70).

[0052] The learning method described will now be discussed further.

[0053] The learning method described is a machine learning approach for 3D object detection in point clouds.

[0054] As is well known, a 3D point cloud is an unordered collection of 3D points with coordinates (typically 3D) that may optionally be accompanied by additional properties (e.g., included in additional coordinates), such as intensity or color. This unordered aspect of the data makes it difficult to analyze, especially compared to structured meshes such as images. For example, different formats of 3D point clouds exist depending on the sensor used to capture the 3D scan that generates the 3D point cloud, but as previously discussed, the neural network is unaware of these different formats. Currently, existing sensors can provide 3D point clouds with millions of points, giving extremely high-quality, ultra-dense 3D scans. Any 3D point cloud in this paper originates from such a dense 3D scan. Point clouds can also be generated from other 3D representations (such as CAD objects or mesh elements). Any 3D point cloud in this paper originates from such generation. This is discussed further below.

[0055] 3D object detection in 3D point clouds corresponds to the task of detecting / localizing specific 3D shapes or 3D objects in a 3D scene. The detection can use 3D bounding boxes, for example, by drawing a 3D bounding box around each detected object to indicate that detection has been performed. In the context of this disclosure, a 3D shape can be a 3D object or simply a part of it. The detection detects one or more shapes that are all similar to the input shape. In the context of this disclosure, the similarity between 3D objects includes both global geometric similarity and local geometric similarity.

[0056] As is generally known, a 3D point cloud is a collection of 3D points, where each 3D point is described by three Cartesian coordinates in 3D space. This is equivalent to saying that each point cloud in this paper can be associated with a set of triples, each triple representing the 3D coordinates of a point in the 3D point cloud. Any 3D point cloud in this paper can represent a 3D real-world scene, which is a segment of the real world (e.g., a factory scene or an office scene), and each point in the 3D point cloud represents a point in that scene. "Representing a real-world scene" does not necessarily mean that the 3D point cloud originates from physical measurements performed in the real world. For example, any 3D point cloud in this paper can be a composite representation of a real-world scene, as discussed further below. Any 3D point cloud in this paper can represent or include 3D objects or may include several 3D objects. Each object in the 3D point cloud is here a set of points in the 3D point cloud, each point representing a point of a 3D object as it would be in the real world, and the set of points together forms a representation of the real-world 3D object.

[0057] Any 3D point cloud in this paper can be a synthetic point cloud, i.e., a point cloud derived from a synthetic 3D point cloud generation process. In other words, a synthetic point cloud representing a real scene is not derived from physical measurements performed on the real scene to capture a representation of the real scene, but rather from a representation of the real scene generated in a virtual, digital manner. The synthetic point cloud generation process specifies any computer-implemented process that outputs a synthetic 3D point cloud. For example, the generation process can generate a 3D point cloud from any other 3D geometric representation, such as a CAD model or a 3D mesh. The generation process can be, for example, the process described below: Xiangyu Yue et al., “A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving”, 2018, which is incorporated herein by reference, and the article proposes a virtual scanner simulation to extract point clouds from 3D CAD scenes (e.g., video game environments).

[0058] Alternatively, any 3D point cloud described herein can originate from physical measurements on a real-world scene. Providing such a 3D point cloud can include providing one or more physical sensors (e.g., fixed or moving 3D scanners, such as fixed or moving LiDAR), each configured to: acquire a corresponding physical signal; and acquire one or more corresponding physical signals by operating the one or more physical sensors on a real-world scene (i.e., scanning the real-world scene using each sensor). Providing the 3D point cloud can then include automatically determining the 3D point cloud based on the measurements according to any known technique. It should be noted that existing sensors are capable of providing clouds of millions of points, giving extremely high-quality, ultra-dense 3D scans. Any 3D point cloud described herein can be acquired by sensors such as 3D scanners (e.g., LiDAR), which are widely used to collect 3D point cloud representations of real-world environments. Several point cloud datasets are now publicly available for outdoor contexts, such as those discussed in the following papers: Timo Hackel et al., “SEMANTIC3D.NET: A new large-scale point cloud classification benchmark,” ISPRS 2017; and Andreas Geiger et al., “Are we ready for Autonomous Driving? The KITTIVision Benchmark Suite,” CVPR 2012, both of which are incorporated herein by reference. For indoor contexts, see, for example, the dataset discussed below: Iro Armeni et al., “3D Semantic Parsing of Large-Scale Indoor Spaces,” CVPR 2016, which is also incorporated herein by reference. Any 3D point cloud in this paper can be obtained from any of these datasets.

[0059] Any 3D point cloud described in this article may include more than 10,000, 100,000, 1,000,000, 1,000,000, or 100,000,000 points. For example, a 3D point cloud representing a scene (e.g., a factory / manufacturer scene) may include more than 10,000,000 or 100,000,000 points. Any 3D point cloud representing a 3D object may include approximately 20 times fewer points.

[0060] Any 3D point cloud described herein can represent a manufacturing scene, such as the interior (or part thereof) of a factory / manufacturer. Such a scene can include objects in the manufacturing world, represented by multiple parts / components of the 3D point cloud. Any 3D point cloud described herein can also simply represent such objects. Such objects can be (e.g., mechanical) parts or components of parts (or equivalently, components of parts, since components of parts can be considered as parts themselves), or more generally, any rigid body component (e.g., a moving mechanism). The objects can belong to various industrial sectors: aerospace, construction, building, consumer goods, high-tech equipment, industrial instrumentation, transportation, marine, and / or offshore oil and gas production or transportation. Therefore, the object can represent an industrial product, which can be any mechanical component, such as components of land transportation vehicles (e.g., including automobile or light truck equipment, racing cars, motorcycles, truck and motorcycle equipment, trucks and buses, trains), components of air transportation vehicles (e.g., including aircraft fuselage equipment, aviation equipment, propulsion equipment, defense products, civil aviation equipment, space equipment), components of marine transportation vehicles (e.g., including marine equipment, commercial vessels, offshore equipment, yachts and workboats, marine equipment), general mechanical components (e.g., including industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment products, manufactured metal products, tire manufacturing products), electromechanical or electronic components (e.g., including consumer electronics products, safety and / or control and / or instrumentation products, computing and communication equipment, semiconductors, medical devices and equipment), consumer goods (e.g., including furniture, home and garden products, leisure goods, fashion products, products of durable goods retailers, products of consumable goods retailers), and packaging (e.g., including food and beverage and tobacco, beauty and personal care, and home product packaging). The object can also be a manufacturing tool, such as an articulated manufacturing robot.

[0061] The learning method described is used for machine learning.

[0062] As is known from the field of machine learning, the processing of input by a neural network involves applying operations to the input, the operations being defined by data including weight values. Learning a neural network therefore includes determining the values ​​of the weights based on a dataset configured for such learning, such a dataset may be referred to as a learning dataset or a training dataset. For this purpose, the dataset comprises multiple data slices, each forming a corresponding training sample. The training samples represent the diversity of situations in which the neural network will be used after being learned. Any training dataset in this document may include a large number of training samples exceeding 1,000, 10,000, 100,000, or 1,000,000. In the context of this disclosure, "learning a neural network based on a dataset" means that the dataset is a learning / training dataset for a neural network, and the values ​​of the weights (also referred to as "parameters") are based on the learning / training dataset.

[0063] The neural network involved in the method can be a deep neural network (DNN). DNNs are a powerful set of techniques for learning within neural networks, such as those discussed below: Dermemelhart et al., “Learning internal representations by error propagation, Parallel distributed processing: explorations in the microstructure of cognition,” Vol. 1: Foundations, MIT Press, Cambridge, MA, 1986, which is incorporated herein by reference. DNNs are biologically motivated programming paradigms that enable computers to learn from observed data. Compared to the artificially designed low-level features (Zernike moments, HOG, Bag-of-Words, or SIFT) used in other methods (minimum cut, SVM, Boosting, or random forests), DNNs have the ability to learn rich, intermediate-level media representations. Therefore, the use of DNNs is particularly relevant for object detection / recognition. More specifically, DNNs focus on end-to-end learning based on raw data. In other words, end-to-end optimization is performed, for example, by starting with raw features and ending with labels, which goes beyond the maximum possible range of feature engineering.

[0064] In the context of the learning method, the training dataset of the neural network is a dataset of provided superpoints, each superpoint forming a training sample of the training dataset. The training dataset consists of superpoints, each of which is a set of points in a 3D point cloud. The superpoints may originate from one or more 3D point clouds; that is, each point cloud includes one or more superpoints of the provided superpoints. A superpoint in a 3D point cloud is a set of points in the 3D point cloud representing a shape, i.e., at least a portion of a 3D object, such as the entire object or its local shape. The concept of superpoints is well known. Providing the superpoints in S10-S20 may include providing such superpoints, for example, those that have already been computed. Alternatively, providing the superpoints in S10-S20 may include providing one or more 3D point clouds in S10, and extracting the superpoints in S20 from the provided 3D point clouds.

[0065] Providing one or more 3D point clouds in S10 may include retrieving the 3D point clouds from memory. Alternatively, providing one or more 3D point clouds in S10 may include obtaining (e.g., all) at least some of the one or more 3D point clouds. Obtaining 3D point clouds may include performing a synthetic 3D point cloud generation process, in which case, as previously discussed, the obtained 3D point cloud is a synthetic 3D point cloud. The generation process may, for example, generate the 3D point cloud from a CAD scene. Obtaining 3D point clouds may alternatively include, as previously discussed, performing physical acquisition of the 3D point clouds by operating one or more physical sensors (e.g., one or more LiDARs) on a real scene (e.g., inside a factory / manufacturer). The provided 3D point clouds may therefore consist only of synthetic 3D point clouds, only of physically acquired 3D point clouds, or a mixture of synthetic and physically acquired 3D point clouds.

[0066] Extracting the superpoints from the provided 3D point cloud in step S20 includes: for each provided 3D point cloud, extracting one or more superpoints from the 3D point cloud. Extracting superpoints from the 3D point cloud can be performed by any known method for extracting superpoints from a 3D point cloud, such methods being well-known. The method for extracting the superpoints from the 3D point cloud can be distance-based extraction, wherein, as known in the prior art, the adjacency of points is considered (within a radius or given by the K-nearest neighbor method). Alternatively, the method for extracting the superpoints can be geometry-based extraction, as known in the art, for which the point cloud is divided into geometrically uniform superpoints.

[0067] The method for extracting the hyperpoints can be, for example, a distance-based method as described below: G. Elbaz et al., “3D Point Cloud Registration for Localization using a Deep Neural Network Auto-Encoder”, CVPR 2017, which is incorporated herein by reference. This method can be referred to as the RSCS (Random Sphere Cover Set) method. This method allows the extraction of hyperpoints as basic matching units for the registration problem. More specifically, this method includes an iterative procedure, which includes: given a predefined radius R... sphere At each iteration, a point P that does not yet belong to any superpoint SP is randomly selected, and then a new SP is defined as a point located at P, with radius R. sphere The set of points inside a sphere. Superpoint overlap is allowed, meaning a point can be included in several superpoints.

[0068] Figure 2The method of extracting superpoints from the 3D point cloud 20 using the RSCS method (where the radius is equal to 0.3m) is shown. Superpoints 201 to 209 have been extracted.

[0069] The method for extracting the superpoints can alternatively refer to the geometry-based method given below: L. Landrieu et al., “Cut pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs,” SIAM Journal on Imaging Sciences, 2017, which is incorporated herein by reference. This method allows for the extraction of superpoints for semantic segmentation problems. This method considers geometric features such as linearity, planarity, divergence (also discussed in: J. Demantk et al., “Dimensionality based scale selection in 3D lidar point clouds.” International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2011, which is incorporated herein by reference) and verticality (also discussed in: S. Guinard et al., “Weakly supervised segmentation-aided classification of urban scenes from 3d LiDAR point clouds,” ISPRS 2017, which is incorporated herein by reference) to distinguish between superpoints. Such features describe the local proximity of each point based on its K nearest neighbors. The method then involves constructing a neighborhood graph of nearest neighbors, as discussed in the following reference: Guinard et al., “Weakly supervised segmentation-aided classification of urban scenes from 3d LiDARpoint clouds,” ISPRS 2017, which is incorporated herein by reference. The method utilizes a cut pursuit algorithm to perform minimum partitioning on the geometric features of the graph, as discussed in the following reference: L. Landrieu et al., “Cut pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs,” SIAM Journal on Imaging Sciences, 2017, which is incorporated herein by reference. This algorithm incorporates regularized weights that control the coarsness of the partitioning.The method disclosed in the previously cited reference by Guinard et al. (“Weakly supervised segmentation-aided classification of urban scenes from 3d LiDAR point clouds”, ISPRS 2017) can first compute linearity, planarity, and divergence values ​​by calculating PCA (principal component analysis) on the adjacency of each point. This method can then compute the eigenvalues ​​of the covariance matrix of the point's location, and can be used to evaluate normals or derive other meaningful vectors by estimating the surface.

[0070] Figure 3 This demonstrates the use of the geometry-based method discussed in the current discussion (where the regularization weight is equal to 0.1m) to extract data from... Figure 2 Superpoint extraction was performed on the 3D point cloud 20 shown in the figure. Superpoints 301 to 308 have been extracted.

[0071] Figure 4 and Figure 5 The illustration shows a comparison between superpoints obtained from the same point cloud but using different methods. Figure 4 The method of obtaining superpoints from point cloud 40 using a distance-based approach is shown, and Figure 5 The diagram shows superpoints obtained from the same point cloud 40 but using a geometry-based method.

[0072] The linearity value can be given by the formula (λ1-λ2) / λ1, the planarity value can be given by the formula (λ1-λ3) / λ1, and the divergence value can be given by the formula λ3 / λ1, where λ1, λ2, and λ3 are the eigenvalues ​​of the covariance matrix of the point's position. Figure 6 The diagram illustrates the linear values ​​of points in a 3D point cloud. Figure 7 The planarity values ​​are illustrated, and Figure 8 The diagram illustrates the divergence value.

[0073] The method can preferably use a geometry-based approach for superpoint extraction, such as the method discussed above, to benefit from the advantages of this type of approach. With such an approach, superpoints tend to correspond to only a uniform portion of the point cloud (due to cuts obtained in adjacent images), which facilitates further processing: thus, based on the regularization weights of the cuts, superpoints tend to be portions of a single object. This provides properties of interest. For example, the superpoint extraction tends to be consistent between superpoints extracted from isolated point cloud objects and those extracted from the same (or similar) objects in the scene (i.e., with other objects). Similarly, superpoints obtained using such an approach regroup points with similar geometric properties, thus simplifying the complexity required for the learning steps.

[0074] Now we will discuss the learning of neural networks (S30).

[0075] The learned neural network is configured to take superpoints as input and encode them as eigenvectors, where the eigenvectors are vectors with a dimension lower than that of the superpoints but which form geometrically meaningful eigencodes of the superpoints. The neural network is learned to do this, i.e., to modify the weights of the neural network during the learning process until the neural network performs this task sufficiently well. The neural network may include an encoder, which is a neural network configured to take superpoints as input and project them onto an eigenvector space. In other words, the encoder is trained to compress the input superpoints while preserving their properties. By using a loss, learning the encoder results in projections onto eigenre representations that are meaningful in terms of geometric properties. For example, the learning by using the loss makes a correlation exist between the similarity between two superpoints and the similarity of their encodings. The neural network can be, or can include, any neural network that takes superpoints (or, more generally, 3D point clouds) as input and projects them onto a lower-dimensional space while remaining invariant to object transformations such as rotation. For example, the neural network architecture could be the kind disclosed below: R. Spezialetti et al., “Learning an Effective Equivariant 3DDescriptor Without Supervision”, ICCV 2019, which is incorporated herein by reference.

[0076] The learning of the neural network S30 may further include, for example, learning a decoder simultaneously with the learning of the encoder, the decoder being a neural network configured to take eigenvectors as input and decode them into superpoints. In other words, the decoder performs superpoint reconstruction based on the input eigenvectors (e.g., as encoded by the encoder). The neural network learned by the method can therefore exist in both the encoder and the decoder, the encoder being configured to take the input superpoints as input and encode them into eigenvectors, and the decoder being configured to take the eigenvectors as input and decode them into superpoints.

[0077] For example, the neural network can be an autoencoder comprising an encoder and a decoder. The autoencoder architecture is well known in the field of machine learning and is discussed, for example, in the following reference: G. Elbaz et al., “3DPoint Cloud Registration for Localization using a Deep Neural Network Auto-Encoder”, CVPR 2017, which is incorporated herein by reference. As is known in the field of machine learning, an autoencoder compresses information about its input (i.e., the superpoints for the autoencoder of this disclosure). More specifically, the autoencoder is trained to project the input (here, the input superpoints) into an intrinsic representation (e.g., a size smaller than the input) and reconstruct the input from this representation. The autoencoder compresses the information while attempting to preserve all properties of the input superpoints for several different superpoints. Learning the autoencoder thus implicitly results in projection into an intrinsic representation that is meaningful in terms of geometric properties. For example, the learning makes a correlation exist between the similarity between two superpoints and their encoded similarity.

[0078] The learning S30 includes minimizing the loss. The loss penalizes inconsistencies between two superpoints. In other words, the loss takes two superpoints as input and calculates the inconsistency between them, i.e., a quantification of the geometric inconsistency. The more geometrically dissimilar the superpoints are, the larger the loss value for those superpoints. Learning S30 uses the loss to measure whether the eigenvectors sufficiently accurately encode the input superpoints in the training dataset when the neural network is applied to the input superpoints. Specifically, learning S30 may include iterations of the following operations:

[0079] - Select the superpoints of the training dataset;

[0080] - The superpoints are encoded into eigenvectors by applying the neural network or the encoder of the neural network;

[0081] - For example, by reconstructing another hyperpoint from the eigenvector, for example, by decoding the eigenvector into another hyperpoint, for example, by applying the decoder of the neural network to the eigenvector to obtain another hyperpoint from the eigenvector;

[0082] - The loss value for the superpoint and the other superpoint is calculated by feeding the superpoint and the other superpoint as input to the loss;

[0083] - Evaluate whether the calculated value is too high, i.e., by evaluating whether the calculated value is higher than a predefined threshold (e.g., a convergence threshold). A high value in the loss captures the fact that the eigenvectors have not yet encoded the superpoint with sufficient accuracy, i.e., the eigenvectors are not geometrically meaningful enough to be the eigenre representation of the superpoint;

[0084] - If the calculated value is too high, for example, modify the weights of the neural network by changing the weights of the encoder and / or decoder.

[0085] The above steps can be repeated as long as the convergence criterion for learning S30 (as is well known in the field of machine learning) has not been met and / or as long as a sufficient (i.e., sufficiently large) number of superpoints in the dataset have not been selected. The selection of superpoints during iterations ensures that each superpoint in the dataset is selected at least once. Any known learning strategy can be used for selection.

[0086] The loss can be a reconstruction loss that penalizes inconsistencies between reconstructions of superpoints. The reconstruction of a superpoint is another superpoint. The reconstruction of a superpoint is obtained from the eigenvectors encoded by the superpoint, i.e., the eigenvectors are obtained by applying the neural network (or its encoder) to the superpoint. The reconstruction of a superpoint can, for example, be the result of applying a decoder of the neural network to the eigenvectors encoded by the superpoint. This is, for example, as previously discussed, when the neural network is an autoencoder comprising an encoder and a decoder. Using a reconstruction loss allows for the evaluation of whether the reconstruction of the superpoint encoded by the neural network is geometrically sufficiently similar to the superpoint, which implicitly enables the evaluation of whether the superpoint is encoded as a geometrically meaningful eigenvector. The reconstruction loss also allows for ensuring accurate reconstruction by the decoder.

[0087] The inconsistency can be any quantification of geometric inconsistency / dissimilarity between two superpoints. For example, the inconsistency can be the distance between superpoints. In other words, the loss can be a reconstruction loss that penalizes the distance between the superpoint and its reconstruction. The distance can be any distance between superpoints, such as any distance between 3D point clouds. For example, the distance can be the Chamfer distance, which provides economic efficiency for the computational resources required to perform the learning S30. Chamfer distance (d CD The following formula is given:

[0088]

[0089] Here, S1 and S2 are two sets of superpoints, where x and y specify points in S1 and S2 respectively (i.e., 3D points with 3D coordinates), and where ||||2 specifies the Euclidean 3D norm. Alternatively, the distance can be the Earth-Mover distance.

[0090] Prior to the learning S30, the learning method may include adjusting the size of the superpoints. In other words, the learning method may include undersampling and / or oversampling the points in each superpoint of the training dataset before the learning S30, such that each superpoint matches the input size of the neural network. For example, the points in each superpoint may be such that, after undersampling and / or oversampling, each superpoint comprises N points, each equipped with 3D coordinates. In such a case, the size of the eigenvectors may be equal to N / 2. Figure 9 A schematic representation of an autoencoder is shown, with an input size of N = 128 and an eigenvector size of N / 2 = 64.

[0091] As previously explained, the learning S30 modifies the weights (also referred to as parameters) of the neural network until the loss has a sufficiently low value on the samples (i.e., superpoints) (e.g., all or most) of the training dataset. These concepts are well known in the field of machine learning. The neural network can be a deep learning model, which is typically characterized by millions of parameters whose values ​​cannot be manually set to characterize these models. These parameters are set using a learning algorithm during the learning of the model. The model is said to be in "training mode" while the learning algorithm is updating the model parameters. It exists in the process of successively "correcting" the model based on its output for each input, thanks to the inherent characteristics of each input and optionally, annotations (if any) associated with it. Once the model has been trained, updates to its parameters cease. The model is then used only in an online phase to process new inputs (i.e., inputs not seen during training mode) and return results, which is said to be in "test mode." In this disclosure, the model (i.e., the neural network) can be a DNN, so these considerations can be applied to the model.

[0092] The learning S30 can be unsupervised, meaning that the learning S30 can be unsupervised learning. In the unsupervised learning paradigm for 3D point cloud feature extraction, no annotations are given regarding the point cloud used during the learning process. This means that the feature extraction will rely solely on point cloud properties (e.g., coordinates of 3D points) without relying on any labels (e.g., extracting similar features for all point clouds from categories with the same labels). Using such training / learning often ensures that the learned neural network is unaware of the type of point cloud it will use as input, the category / type of the object(s) or scene(s) represented by the point cloud, and / or its context (e.g., indoor or outdoor).

[0093] The detection method will now be discussed further.

[0094] Providing the one or more first superpoints and the one or more second superpoints in S40-S50 may include directly (e.g., without computation) providing the already computed one or more first superpoints and the one or more second superpoints. Alternatively, providing S40-S50 may include providing the first 3D point cloud and the second 3D point cloud in S40, and extracting the one or more first superpoints in S50 from the first 3D point cloud and extracting the one or more second superpoints in S50 from the second 3D point cloud. The extraction S50 may be performed as the extraction S20 previously discussed, for example, using the same distance-based or geometry-based methods. Providing the first 3D point cloud and the second 3D point cloud in S40 may be performed as the providing S10 previously discussed.

[0095] The first 3D point cloud represents a 3D scene. The scene is a real-world scene, i.e., a part of the real world. The first 3D point cloud can be measured, i.e., it can originate from physical measurements; for example, as previously discussed, the first 3D point cloud can be obtained by operating one or more physical sensors (e.g., LiDAR) on the scene (as it would in the real world). Alternatively, as previously discussed, the 3D point cloud can be generated from a virtual 3D scene (e.g., a CAD environment). The 3D scene can be a manufacturing scene, e.g., the interior of a factory / manufacturer or a portion thereof. The 3D scene can therefore include 3D objects of the manufacturing world, e.g., mechanical parts being manufactured or already manufactured, or manufacturing tools, such as articulated manufacturing robots.

[0096] The second 3D point cloud represents a 3D object, such as a 3D object in the fabricated world. As previously discussed, the second 3D point cloud may be synthetically generated or may be derived from physical measurements, i.e., it may be a measured 3D point cloud. The 3D object may be similar to or partially similar to (e.g., until occlusion and / or movement such as rotation) one or more 3D objects in the first 3D point cloud. As discussed below, the detection method detects these 3D objects, or at least a portion thereof.

[0097] The one or more second superpoints can be referred to as "queries" because they are superpoints for which the detection method searches for similar superpoints in the first 3D point cloud, as discussed below. This is equivalent to saying that the detection method detects 3D shapes / objects similar to the query. The query can be provided by the user. The first point cloud can be referred to as a "search region" because the detection method searches for objects in the first point cloud that are similar to objects in the query region. The first 3D point cloud can be a large point cloud; for example, the ratio between the number of points in the 3D point cloud and the size of the memory of the computer performing (one or more) methods can be greater than a predefined threshold. For example, the first 3D point cloud for a 16 gigabyte memory size (which corresponds to a standard or inexpensive computer) can include approximately or more than 100 million points, and / or can include 20 times the number of points in the second 3D point cloud.

[0098] The detection method determines the similarity between the extracted superpoints. To reduce the number of extracted superpoints for which similarity has been determined, the one or more first superpoints provided in S40-S50 may include prior optional filtering, which will now be discussed.

[0099] Specifically, providing the one or more first superpoints in S40-S50 may include, for example, providing one or more initial superpoints of the first 3D point cloud by extracting S50 from the first 3D point cloud provided at S40 (as previously discussed). Then, providing S40-S50 may include filtering the one or more initial superpoints by selecting each of the initial superpoints as follows, for each initial superpoint:

[0100] - The inconsistency between the dimension of the superpoint and the dimension of at least one second superpoint is less than a predefined threshold; and / or

[0101] - The inconsistency between the location of the superpoint and the location of at least one second superpoint is less than a predefined threshold.

[0102] The selected superpoints constitute the one or more first superpoints.

[0103] Therefore, the provisioning steps S40-S50 can begin by providing initial superpoints, for example, by extracting initial superpoints from the first 3D point cloud, and then selecting those that are geometrically sufficiently similar to at least one second superpoint. This allows filtering of superpoints in the first 3D point cloud, which may contain many superpoints (e.g., the initial superpoints may collectively cover all points in the first 3D point cloud). Thus, the detection method then only needs to evaluate the similarity between the filtered (i.e., smaller) set of first superpoints and the second superpoints.

[0104] The selection of each initial superpoint that is geometrically sufficiently similar to at least one second superpoint is based on the inconsistency between the initial superpoint and the second superpoints in terms of dimensions and / or location. Specifically, for each initial superpoint, the detection method evaluates whether the inconsistency between the dimensions (e.g., each dimension) of the initial superpoint and the dimensions (e.g., each dimension) of at least one second superpoint is less than a predefined threshold, and / or the detection method evaluates whether the inconsistency between the location of the initial superpoint and the location of at least one second superpoint is less than a predefined threshold. The method then selects initial superpoints whose inconsistency between their dimensions and the dimensions of at least one second superpoint is less than the predefined threshold and / or whose inconsistency between their locations and the locations of at least one second superpoint is less than the predefined threshold as one or more first superpoints. Each inconsistency can be an arbitrary quantification of geometric inconsistency, such as distance or ratio.

[0105] The filtering may include selecting each of the initial superpoints as follows: for each initial superpoint:

[0106] - The distance between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than a predefined threshold; and / or

[0107] - The ratio between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than the maximum ratio and greater than the minimum ratio; and / or

[0108] - The difference between the relative height of the nearest support surface from the superpoint and the relative height of the nearest support surface from at least one second superpoint is less than a predefined threshold.

[0109] The distances between each dimension, the ratios between each dimension, and the differences in relative height listed above are all quantifications / measures of geometric inconsistencies between superpoints. The dimensions of a superpoint can be the dimensions of that superpoint in its local reference frame (LRF, as discussed in the following reference: J. Yang et al., "Toward the Repeatability and Robustness of the Local Reference Frame for 3D Shape Matching: An Evaluation", IEEE Transactions on Image Processing, Vol. 27, No. 8, pp. 3766-3781, August 2018, which is incorporated herein by reference).

[0110] For each initial superpoint, if the distance between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than a predefined threshold, the detection method can, for example, select the initial superpoint by determining the existence of second superpoints where the distance between each dimension of the initial superpoint and the corresponding dimension of the second superpoint is less than the predefined threshold. To do this, the method can determine whether there exists a second superpoint satisfying the following: for each dimension i of the second superpoint:

[0111]

[0112] Among them, spq dimi It is the dimension i of the second superpoint, sprz dimi The dimensions i and d of the initial superpoints of the search region. max d is a predefined threshold, also known as the maximum acceptable difference between the sizes of the two superpoints, and ||.|| is any appropriate norm. max It can belong to a certain value range, which is defined, for example, as a function of the variability of the target in the query and 3D detection. The range can, for example, be a function of the reality represented by the second 3D point cloud, such as the context and / or quality of the point cloud acquisition. For example, in the case where all the 3D objects involved have superpoints with minimum and maximum dimensions including between 1 cm and 1 m, d max It can be equal to 20cm. If the 3D object involved in the learning method has a larger dimension, d max It can be larger. If the point cloud involved in the learning method is a synthetic point cloud, or if the point cloud is acquired using a very accurate scan, then d can be reduced. max Because it is characterized by lower noise. For example, d maxIt can be within the range [5cm; 1m], and can be a function of the regularization weights of the graph cuts discussed earlier.

[0113] Alternatively, if the ratio between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than a maximum ratio and greater than a minimum ratio, the detection method may, for example, select the initial superpoint by determining that there exists a second superpoint where the ratio between each dimension of the initial superpoint and the corresponding dimension of the second superpoint is less than a maximum ratio and greater than a minimum ratio. To do this, the method may determine whether a second superpoint exists where, for each dimension i of the second superpoint:

[0114]

[0115] Among them, spq dimi It is the dimension i of the second superpoint, sprz dimi The initial superpoints of the search region are of dimension i and r. min It is the minimum ratio, also known as the minimum acceptance ratio, and r max The maximum ratio, also known as the maximum acceptance ratio, can be selected based on the quality of the point cloud involved in the method, the method of selecting superpoints, and the variability of the query as the target. max and r min That is, the worse the quality of the point cloud, the less accurate the cutting of the superpoints and / or the more important the target variability, the more effective it is to reduce r. min The more you reduce it, the more you can reduce r. max The more it increases. max It can be greater than 1.5 and / or less than 6, for example, equal to 4, and / or r min The value can be greater than 0.1 and / or less than 0.9, for example, equal to 0.2. Alternatively or additionally, if the difference between the relative height of the initial superpoint from its nearest support surface and the relative height of the second superpoint from its nearest support surface is less than a predefined threshold, the detection method can, for example, select the initial superpoint by determining the existence of a second superpoint where the difference between the relative height of the initial superpoint from its nearest support surface and the relative height of the second superpoint from its nearest support surface is less than a predefined threshold. The relative height captures the difference between the desired position in the query and the position in the search area. The desired position can be defined as a function of the parameters of movement in the query. The predefined threshold can be selected as a function of the uncertainty of the superpoint accuracy and the variability accepted by the query. For example, the predefined threshold can be equal to the query height divided by 4. The predefined threshold can, for example, be equal to 30 cm.

[0116] The nearest supporting surface of a superpoint is a planar surface with the vertical axis or any vector sufficiently close to the vertical axis (i.e., whose cross product norm with the vertical axis is below a predefined threshold) as its normal axis. Therefore, the nearest supporting surface can represent a floor, or any other real-world physical support of the shape represented by the superpoint, such as a table, desk, conveyor belt, or workbench. The detection method can calculate or detect the nearest supporting surface of the superpoint involved in the detection method for selection among initial superpoints. Detection of the nearest support can be performed by any known method. For example, it can be automatically detected during the aforementioned geometry-based extraction of superpoints by performing PCA on the adjacency and minimum cut in the adjacent graph for each point, and detecting superpoints with horizontal plane characteristics (e.g., applying a planarity value to the horizontal plane of each superpoint).

[0117] In other words, for each initial hyperpoint, the detection method can be used to select one or more of the following listed above: distance between each dimension, ratio between each dimension, and difference between relative heights. The selected initial hyperpoint(s) form the one or more first hyperpoints.

[0118] The detection method then encodes each first superpoint S60 into a corresponding first eigenvector by applying the neural network or its encoder to the first superpoints. The method also encodes each second superpoint into a corresponding second eigenvector by applying the neural network or its encoder to the second superpoints.

[0119] The detection method then determines the similarity of S70 between each of one or more (e.g., all) first superpoints and each of one or more (e.g., all) second superpoints. To this end, for each of one or more (e.g., all) first superpoints and each of one or more (e.g., all) second superpoints, the method calculates a similarity metric between the corresponding first eigenvector encoded for the first superpoint and the corresponding second eigenvector encoded for the second superpoint. Determining the similarity of S70 may involve all first and second superpoints. It should be noted that when the detection method envisions prior filtering of the first superpoints, as previously discussed, then determining the similarity of S70 may then involve all first and second superpoints, because the prior filtering has already ensured the relevance of these superpoints used for detecting geometric similarity. Therefore, involving all superpoints is relevant for 3D detection while being relatively economical in terms of computational resources used to determine S70.

[0120] The similarity measure can be any measure / quantization of the similarity between eigenvectors. Now, since eigenvectors are geometrically meaningful eigenre representations of the superpoints they encode, calculating the similarity measure between them is equivalent to determining the geometric similarity between the superpoints. The similarity measure can be a cosine similarity measure cs(e... i e j Its type is:

[0121]

[0122] Among them, e i e j It is the eigenvector that encodes the two superpoints. The similarity measure can be any other similarity measure, such as Euclidean distance, for example, in the case where the neural network has been learned by regularizing the eigenspace.

[0123] The calculation of the similarity metric yields a similarity score, each similarity score being the value of the similarity metric between two eigenvectors (i.e., calculated at S70) between the first superpoint and the second superpoint. Therefore, determination S70 yields a similarity score between the shape of the first 3D point cloud (i.e., the first superpoint representing a local portion of a 3D object therein) and the shape of the second 3D point cloud (i.e., the second superpoint representing a local portion of a 3D object represented by the second 3D point cloud). The method may also include evaluating similarity scores higher than a predefined threshold among the calculated similarity scores. Each corresponding similarity score represents a pair of first and second superpoints whose similarity metric between their two encoded eigenvectors is higher than the predefined threshold. In this case, the first superpoint and the second superpoint are said to match. The predefined threshold can be defined relative to encoder projection properties and / or the user-desired level of similarity. Alternatively, the predefined threshold can be automatically obtained based on a dataset of superpoints manually annotated as similar (e.g., for the target application). In this case, the threshold can be obtained by calculating the minimum similarity between two superpoints that are marked as similar. Using this automatic acquisition, the predefined threshold can fall within the value range [0.7; 1] (e.g., and can be selected based on target similarity and / or the architecture used), for example, it can be equal to 0.8.

[0124] The detection method can therefore provide a search area after S70 is determined. The set of the first superpoints matched in the query ((mq) i , ..., mq m ), ..., (mq i , ..., mq kThe corresponding match / similarity score of the superpoint in the )) i .

[0125] The detection method may further include grouping the matched first superpoints together S80, thereby sorting the first points into several groups, and each group representing a 3D object of the first 3D point cloud that is similar to a 3D object of the second point cloud (i.e., the query) formed by the second superpoints. Thus, after determining the local shapes (i.e., the matched first superpoints) in the first 3D point cloud that are similar to the local shapes in the query (i.e., the second superpoints), the detection method finds 3D objects in the first 3D point cloud that are geometrically similar to the 3D object represented by the query by grouping the matched first superpoints together S80.

[0126] The grouping S80 may include determining one or more groups of first superpoints among first superpoints that all have a determined similarity greater than a predefined threshold with at least one second superpoint (i.e., among matched first superpoints), each group of first superpoints having a shape similar to the second 3D point cloud. In other words, the determination S80 may compare the geometric properties of the matched first superpoints, such as their relative orientation and / or relative position with respect to each other and / or with respect to a reference point (e.g., the center of the query), with the corresponding properties of the query's superpoints, thereby grouping the matched first superpoints into groups that are geometrically similar to the query. The detection method may utilize any known grouping method to perform the determination S80. The grouping may be rotation-invariant, i.e., the grouping detects a group of first superpoints similar to the query, even if the group represents a rotation of a 3D object represented by the 3D point cloud.

[0127] Now let's discuss an example of grouping S80.

[0128] In the example, the determination S80 for one or more sets can be performed according to the method used to compute the graph from the set of superpoints in the query, which will now be discussed. This method yields a complete graph of the following type: G q =(SPs) q ,E), where SPs qLet E be a node representing a superpoint, and E be a set of edges connecting two nodes. This approach involves providing (e.g., determined by any known method) a graph of a second superpoint. The graph of the second superpoint has nodes and edges, each edge connecting two nodes. Each node represents a second superpoint. Each node may have attributes, which may be one or more of encoder features and / or dimensions. Each edge represents a geometric relationship between two superpoints represented by the nodes connected by the edge. The geometric relationship may be the relative positioning and / or orientation of the two superpoints relative to each other. The edge has one or more geometric attributes of the geometric relationship. The attributes may form data associated with the edge. The attributes may include one or more of the following:

[0129] - The distance between the centroids of two hyperpoints represented by the nodes connected by the edge, for example, the likelihood distance when the hyperpoints belong to a 3D point cloud representing an articulated object (e.g., an articulated manufacturing robot), and / or

[0130] - The relative orientation between the surfaces of two superpoints represented by the nodes connected by the edge, for example, when the superpoints belong to a 3D point cloud representing an articulated object (e.g., an articulated manufacturing robot), is a likelihood orientation.

[0131] Providing the graph may include, for example, determining (e.g., computing) the geometric properties for each edge. For example, the surface of the hyperpoints can be obtained through local reference frame (LRF) estimation, as is known in the art, for example, an LRF estimation performed in the following reference: J. Yang et al., "Toward the Repeatability and Robustness of the Local Reference Frame for 3D Shape Matching: An Evaluation", IEEE Transactions on Image Processing, Vol. 27, No. 8, pp. 3766-3781, August 2018, which is incorporated herein by reference.

[0132] In addition to providing the graph, the method for computing a graph from the set of superpoints in the query then includes forming one or more graphs. Each graph has nodes and edges, each edge connecting two nodes. Each node represents a first superpoint. Each edge represents a geometric relationship between the two superpoints represented by the nodes connected by the edge. The geometric relationship can be of the same nature as the geometric relationship in the provided graph, for example, it can be the relative positioning and / or orientation of the two superpoints. The formation of the one or more graphs is performed, for example, by constructing the edges of each graph simultaneously for all graphs or by constructing the edges of each graph individually. Each constructed edge connects two nodes (i.e., represents each matching first superpoint). Each constructed edge has one or more geometric properties similar to the geometric properties of the edges of the graph of the second superpoint. For example, the construction of the edges may include identifying subgraphs of the provided graph and reconstructing the subgraphs only for the first superpoints, for example, in a bottom-up manner.

[0133] Specifically, the construction of the edge may include iteratively exploring matching first superpoints, and for each explored first superpoint, finding one or more likelihood edges, each likelihood edge used to connect a node representing the explored first superpoint to another node representing another first superpoint. "Likelihood edge" means an edge whose attributes match the attributes of the edges connecting two nodes of two second superpoints that match two first superpoints in the provided graph. The discovery of the edge may include evaluating (if constructed) whether the edge will have attributes similar to those present in the provided graph and whose edges connect two nodes of two second superpoints that match two first superpoints that will be connected by the edge (if constructed). If this is the case, the discovery constructs the edge. The construction of the edge may therefore include calculating its attributes, which can be performed exactly as in the provided graph. "Similarity attribute" or "matching attribute" means, in this example, an attribute that is equal or substantially equal, or whose difference (e.g., measured by distance and / or ratio) is below a predefined threshold, for example, a threshold defined relative to the number of points in the query. In the example, the discovery of the likelihood edges can be optimized by comparing only the edges of distant superpoints whose distance is less than the maximum likelihood distance between two superpoints of the query (or less than the expected maximum size of the query).

[0134] Each formed graph originates from the construction of its edges (i.e., the construction of edges results in one or more graphs being formed). Each graph corresponds to a specific group of first superpoints, i.e., the nodes of the graph represent the superpoints of the group. The edges of the graphs are all similar to the edges of the provided graph.

[0135] The previously discussed example of grouping S80 (i.e., by providing a graph of the second superpoint, and one or more graphs forming the first superpoint) is particularly robust because it can form groups of first superpoints, each representing a 3D object in the search region that is similar to the 3D object in the query, regardless of whether the 3D object is non-articulated or articulated.

[0136] In another example, the determination of one or more groups (S80) can be performed by using the Hough Voting method with respect to the center of the object, as discussed in the following reference: F. Tombari et al., “Hough Voting for 3D Object Recognition under Occlusion and Clutter,” IPSJ Transactions on Computer Vision and Application March 2012, which is incorporated herein by reference. This method attempts to determine the centers of potentially similar objects by voting for each superpoint in the search region based on one or more superpoints of its matching query. This may include capturing the relative orientation of the query superpoints to superpoints in the search region so that voting is conducted in the same direction and orientation. Such relative orientation can be obtained through LRF estimation, for example, LRF estimation performed in the following reference: J. Yang et al., "Toward the Repeatability and Robustness of the Local Reference Frame for 3D Shape Matching: An Evaluation", IEEE Transactions on Image Processing, Vol. 27, No. 8, pp. 3766-3781, August 2018, which is incorporated herein by reference.

[0137] As referenced earlier by F. Tombari et al., “Hough Voting for 3D Object Recognition under Occlusion and Clutter,” IPSJ Transactions on Computer Vision and Application, March 2012, the Hough Voting method can include defining a 3D mesh, which is decomposed into a set of voxels in the 3D space of the search region. The optimal size of the 3D mesh can be defined relative to the size of the query and the accuracy used to estimate LRF votes. In the example, the size of the voxels can be given by the following formula:

[0138] size = max(querydims) * f * corr LRF ,

[0139] Where querydims is the dimension of the query when the axis is aligned, f is a factor used to determine the precision with respect to the center (e.g., f can be equal to 0.2), and corr LRF It is a factor used to attenuate the effects of the approximate LRF (especially for large objects) (e.g., for a standard LRF implementation based on the covariance matrix, corr LRF It can be equal to 1.5).

[0140] As referenced earlier by F. Tombari et al., “Hough Voting for 3D Object Recognition under Occlusion and Clutter,” IPSJ Transactions on Computer Vision and Application, March 2012, the Hough Voting method can then aggregate voting voxels, for example, by considering the number of distinct matching superpoints for each voxel's query. This can involve using a threshold defined relative to the number of superpoints in the query to determine which voxels represent potential object centers similar to the query. This threshold controls the amount of likelihood occlusion and, possibly, the variability of the target for a specific part of the object. Therefore, it can be directly accessed as a parameter that the user can modify during inference / detection.

[0141] Figures 10 to 12 The Hough Voting method is illustrated. Figure 10 It has been shown that Figure 2 The point cloud shown above has its extracted superpoints. These superpoints represent queries. Figure 10 The diagram shows the distances from the center of each superpoint and the query center. For example, reference numeral 100 indicates such distances. Figure 11 The search area shows the formation and composition of... Figure 10 A point cloud represents a set of superpoints of similar objects. Figure 11 The figure also shows a vote cast from the center of each superpoint. For example, reference numeral 110 indicates such a vote. Figure 12 This is an illustration of the Hough Voting method, which involves casting a rotation-awarevote.

[0142] It should be understood that grouping S80 is not necessarily performed in the detection method. For example, the second point cloud may consist of only a single superpoint, and therefore grouping is not required.

[0143] The detection method may further include determining bounding boxes S90 for each defined group. The bounding boxes surround / encircle the group. Determining the bounding boxes S90 may include drawing the bounding boxes around the center of each group, and, for example, determining that center beforehand. The size and / or dimensions of each bounding box may be equal to or substantially equal to the size of the (e.g., minimum) bounding box used to cover the second 3D point cloud.

[0144] In the case where the graph of the superpoints is computed using the previously discussed method in group S80, each bounding box can be drawn to enclose the corresponding group of the first superpoints, and the size of the bounding box can be equal to or substantially equal to the size of the (e.g., minimum) bounding box used to cover the query.

[0145] When using the Hough Voting method in group S80, the bounding box can be drawn in the search area when determining the likelihood object center: centered on the likelihood center (with the same dimensions as the bounding box query) and / or around the superpoint for the voxel vote representing the likelihood object center.

[0146] It should be understood that when grouping S80 is not performed by the detection method, as previously discussed, the determination S90 can simply determine the bounding box around each matched superpoint.

[0147] The determination S90 can also determine bounding boxes surrounding the second 3D point cloud. The detection method can also include displaying each determined bounding box. This provides the user of the detection method with visual feedback on the 3D objects detected in the search area.

[0148] The detection method may further include: for each group, determining a similarity score for the group by measuring the similarity between a first superpoint in the group and one or more second superpoints. The similarity score for each group is a score of geometric similarity between the group and a group of superpoints formed by one or more second superpoints. The detection method can determine the similarity score by calculating any such score of geometric similarity. The similarity may depend on the application and / or be based on several cues. The similarity score may include one or more of the following:

[0149] - Superpoint Match Similarity Score (SPMS): This is a combination of the matching scores of superpoints (with the identified superpoints in the query) for a given group of matches against the first superpoint. Here, n is the number of superpoints in the query, and m is the number of superpoints. i It is the matching score for each of the groups with the first superpoint (if there is no match, m) i =0), and the score is given by the following formula:

[0150]

[0151] - Color histogram comparison, for example, for applications where color information is available (e.g., fire extinguisher detection). The histogram can be calculated at all points from the search area included in the bounding box or only at matching superpoints, and / or

[0152] - Any global shape descriptor that can be used on all points from the search area included in the bounding box or only on matching superpoints.

[0153] The similarity scores (also known as similarity measures) listed above can be used together to obtain a global similarity score or used independently depending on the target application.

[0154] In addition to determining a similarity score for each group, the detection method may also include displaying the similarity scores of the groups. The detection method may, for example, display the similarity scores of all groups simultaneously, arranged in descending order of similarity scores. The detection method may also display bounding boxes surrounding each group and optionally surrounding a second point cloud simultaneously with the display of the similarity scores for each group. The detection method may also display the dimensions of the bounding boxes. The method may also display a visual indication for each group that the similarity score of that group is higher than a similarity score threshold.

[0155] The similarity score threshold can be a percentage of similarity. The similarity score threshold can be selected by the user. For example, the similarity score threshold can be displayed to the user as a slider having lines representing all possible values ​​of the threshold (e.g., from 0% to 100%) and a handle (of any shape), which the user can set by graphically selecting (e.g., using a click or touch on the handle) and sliding along the lines (e.g., using a drag-and-drop operation). The slider can be part of a desktop applet displayed on the graphical user interface of a computer used by a user of the detection method, the desktop applet being a software component stored in memory. The user's selection of the threshold can occur before executing the detection method, for example, as a starting step of the method, allowing the user to predefine his / her desired similarity score and the bounding boxes he / she wants to display. Alternatively, the selection can also occur after the determination of the bounding boxes S90, allowing the user to trigger the display of only the bounding boxes surrounding groups with sufficiently high similarity scores. Alternatively, the user can continuously select the similarity score threshold during the execution of the method. For example, when the user selects the similarity score threshold, the steps of the detection method are executed to detect bounding boxes around each group that all have similarity scores higher than the threshold (i.e., the detected groups, and then create the bounding boxes), and display them in real time as a result of the user's selection, continuously and in real time adjusting the display of the bounding boxes according to the user's continuous changes to the similarity score threshold.

[0156] Figure 13 A screenshot of a desktop applet 130 displayed on a graphical user interface of a computer used by a user of the detection method is shown. The desktop applet allows selection of the similarity score threshold. The desktop applet 130 displays the first superpoints of each group detected in column 1300, each of which is assigned a name. The desktop applet 130 also displays column 1304 of the dimensions of the bounding box for each group. The desktop applet 130 also displays column 1302 of the similarity score for each group. The desktop applet also displays a column 1310 for each group with a visual indication 1310 that the similarity score of the group is higher than (or lower than) the similarity score threshold. The desktop applet includes a slider consisting of a line 1308 and a handle 1306. The user can change the value of the similarity score threshold by graphically selecting (e.g., using a click or touch on the handle) the handle 1306 and sliding the handle along the line 1308 (e.g., using a drag-and-drop operation). Figure 13 In the screenshot shown above, the threshold is set to 84%.

[0157] The following describes how to implement the process of integrating learning and detection methods.

[0158] The described implementation allows for example-based 3D object detection, which involves locating similar shapes / objects to a given point cloud example within a larger point cloud. Localization is based on 3D bounding boxes surrounding the detected object. The implementation employs a deep learning scheme for detection. The evaluation criteria used for this implementation are as follows:

[0159] Given a point cloud query (example) and a larger point cloud (search area) as input,

[0160] The described implementation attempts to output 3D bounding boxes surrounding objects similar to the query within a search area. However, the number of objects to be detected is unknown beforehand. It should be noted that the search area may not contain objects similar to the selected query.

[0161] Example-based object detection should be unaware of categories; that is, it should be able to detect objects similar to any object query.

[0162] Example-based object detection should be unaware of the context; that is, the detection is independent of the object's context / environment (e.g., indoor, outdoor). It does not use point cloud context-specific prior knowledge.

[0163] The point cloud representation should be robust to noise, occlusion, and point distribution, and invariant to different types of transformations (e.g., rotation) and object positions (e.g., articulated objects such as robotic arms). This is achieved through the locally learned representation used.

[0164] - This implementation aims to locate similar objects that may not necessarily represent an exact instance of the query. In fact, the query may not even belong to the search area.

[0165] This implementation provides genericity, meaning the detection is independent of the object (even objects / context that are not visible during offline training) and the point cloud type (no special constraints are required on point cloud acquisition). This implementation is capable of locating partially similar shapes given a single shape query (i.e., not just locating the exact instance corresponding to a given shape query). This implementation allows for 3D shape detection, where the objective is to detect shapes within the point cloud.

[0166] This implementation provides a learning-based framework for performing efficient and general 3D detection of all shapes similar to a given shape query in a point cloud. The learning portion of the framework is based on deep learning techniques (i.e., using deep neural networks).

[0167] The implementation method follows Figure 14The framework described in [the document / document]. As in [the document / document]... Figure 14 As shown above, the implementation method includes an offline phase and an online phase:

[0168] - Offline Stage: This stage trains the encoder based on superpoints. It consists of two main steps. It's important to note that this stage is transparent to the user. The two main steps are:

[0169] ○ Superpoint extraction. Superpoints are subsets of points that describe a local portion of a given point cloud. Consistency exists between superpoints extracted from isolated point cloud objects and those extracted from the same objects in the scene (i.e., objects with other objects).

[0170] ○ Encoder learning based on superpoint training datasets. This includes point cloud DNN-based models.

[0171] - Online Phase: Given a point cloud shape query and a point cloud scene, superpoints are extracted from both inputs and then encoded using the learned encoder during the offline phase. The superpoint encodings for the query are compared with the superpoint encodings for the scene. Those most similar to a given superpoint encoding are matched.

[0172] Offline learning

[0173] An encoder is trained offline to project any 3D superpoint into an intrinsic representation that presents meaningful geometric information, i.e., such that the distance between the encodings of two superpoints is highly correlated with the geometric similarity of those two superpoints (e.g., relative to the orientation of the superpoints). The distance is defined as a similarity measure between two vectors, such as cosine similarity or Euclidean distance.

[0174] Dataset generation: Superpoint extraction

[0175] First, a dataset of superpoints is constructed to train the encoder. The diversity of the superpoints on which the encoder is trained is directly responsible for its ability to construct meaningful projections of any 3D geometry. The dataset of superpoints can be extracted from any set of one or more point cloud scenes containing a variety of geometries. The extraction of superpoints is completely unsupervised; therefore, no annotations are required for the point cloud scene.

[0176] (one or more) point cloud scenarios

[0177] Point cloud scenes are generated for the training without simulating a virtual scanner. Given CAD models of various objects, the generation involves converting the CAD models into point clouds and then inserting them into a point cloud background. When inserting objects, possible occlusion (object / object occlusion or object / background occlusion) and point density normalization for a realistic synthetic point cloud representation are considered. One hundred scenes are used for the learning / training.

[0178] Superpoint extraction

[0179] This implementation uses a geometry-based approach for superpoint extraction, as previously discussed. Such a approach yields properties of interest. Superpoints typically correspond to only a uniform portion of the point cloud (due to cuts obtained in adjacent graphs), a fact that facilitates further processing: it generally implies that, based on the regularization weights of the cuts, a superpoint is a portion of a single object. This provides at least two properties of interest for this implementation:

[0180] The superpoint extraction is fairly consistent between superpoints extracted from isolated point cloud objects and those extracted from the same (or similar) objects in the scene (i.e., other objects).

[0181] - Superpoints obtained using this method are regrouped into points with similar geometric properties, thus simplifying the complexity required for the encoder learning steps.

[0182] Encoder learning

[0183] The loss used in this implementation is the reconstruction loss. This loss is the Chamfer loss, which offers efficiency in terms of computational cost. Given two point sets S1 and S2, the loss is given by the following formula:

[0184]

[0185] This allows for the use of consumable point clouds as input and their projection into a lower-dimensional space, while remaining rotation-invariant for any neural network, such as those described in the following reference: R. Spezialetti et al., “Learning an Effective Equivariant 3D Descriptor Without Supervision”, ICCV 2019, which is incorporated herein by reference. In this implementation, the training is unsupervised.

[0186] The model is trained using a previously generated superpoint dataset. The model is trained with a corresponding input size based on the number of features to be considered for the point cloud. In this embodiment, the number of points in the superpoints is undersampled or oversampled to match the input size of the neural network. The input size in this embodiment is N = 128 points per superpoint, where the features are 3D coordinates and the encoding size is N / 2.

[0187] Online inference

[0188] Superpoint coding matching

[0189] When obtaining the set of superpoints for the query and search regions, this implementation calculates the similarity between each superpoint in the search region and each superpoint in the query. Since this step can be quite time-consuming, this implementation may include the filtering described below to avoid useless comparisons:

[0190] - Dimensional comparison between superpoints when projected into its LRF is as described in the following reference: J. Yang et al., "Toward the Repeatability and Robustness of the Local Reference Frame for 3D Shape Matching: An Evaluation", IEEE Transactions on Image Processing, Vol. 27, No. 8, pp. 3766-3781, August 2018, which is incorporated herein by reference. If the ratio between dimensions on each corresponding axis is far greater than 1, and the difference in each dimension exceeds a certain threshold, then the superpoints in the search region are not considered potential matching candidates for a given superpoint of the query. In other words, this implementation attempts to have i:

[0191]

[0192] Among them: spq dimi It is the dimension i of the query superpoint, sprz dimi The dimensions i and d of the superpoints of the search region. max rmin is the maximum acceptable difference between the sizes of the two superpoints (e.g., 20 cm), rmax is the minimum acceptable ratio (e.g., 0.2), and rmax is the maximum acceptable ratio for each dimension i of the superpoint in its LRF (e.g., 4); and / or

[0193] - The relative height from the floor to the nearest supporting surface. If the difference between the center heights of the superpoints from the floor (or the nearest supporting surface) is greater than a certain threshold (e.g., 30 cm), the superpoints in the search area are not considered potential matching candidates for the given superpoint for the query. It should be noted that the nearest supporting surface can be detected by any location detection algorithm (e.g., planarity values ​​on the horizontal plane for each superpoint).

[0194] The similarity metric used in the above implementation is cosine similarity, given by the formula below:

[0195] Among them, e i e j It is the encoding of two superpoints.

[0196] It is important to note that, depending on the encoder, another similarity metric can be selected (e.g., if regularization of the eigenspace of the autoencoder is performed).

[0197] In this implementation, if the similarity (e.g., cosine similarity) is above a threshold, then a superpoint in the search region is matched to a superpoint in the query region. This threshold is defined relative to the encoder's projection properties and the user's expected similarity. Using a set of superpoints manually annotated as similar or dissimilar for the target application, such a threshold can be obtained by calculating the minimum similarity achievable between two superpoints marked as similar. In this implementation, the threshold is fixed to 0.8 based on such an experiment.

[0198] At the end of this step, the search area is obtained. The set of n matching superpoints in the query, along with their corresponding matches mq to the matching superpoints in the query. i Score((mq) i , ..., mq m ), ..., (mq i , ..., mq k )).

[0199] Superpoint coding grouping

[0200] This step in the described implementation involves 3D bounding box detection from the set of matching superpoints. It should be noted that when an object consists of only one superpoint, a bounding box proposal can be drawn directly around the matching superpoint.

[0201] Up to this step, local descriptors (superpoint features) are computed and compared between the query and search regions. To infer the set of likelihood bounding boxes, the phase positions of the superpoints in the query (e.g., relative to a reference point such as the query center) are used.

[0202] This relative position can be obtained by using the Hough Voting method with respect to the center of the object, as previously explained. Alternatively, this relative position can be obtained using methods for computed graphs from sets of superpoints in the query, as previously discussed.

[0203] This implementation of the process outputs bounding box proposals. Since the similarity can depend on the application or object, this implementation may include adding similarity indicators about the proposals. This helps the user control how similar he / she wants the detected objects to be to the query. The method may, for example, include displaying already discussed... Figure 13 The desktop mini-program 130 allows users to refer to the previous example. Figure 13The discussion focuses on using a desktop app to set similarity score thresholds. Each detection result is associated with a similarity score (similarity rate), and the user can select thresholds to retain only those that are most convenient. Figure 13 In the desktop mini-program, a threshold of 84% was selected as the similarity rate.

[0204] This similarity can be application-dependent. It can then be based on several clues. The implementation can use several methods to calculate the similarity metric, which can be obtained from the proposed bounding boxes and / or matching superpoints listed below. These similarity metrics can be used together to obtain a global similarity score or used independently depending on the target application. The metrics are as follows:

[0205] - Superpoint Match Similarity Score (SPMS): A combination of matching scores for a given proposal's matching superpoints (compared to the identified superpoints in the query). Here, n is the number of superpoints in the query, and m... i It is the matching score for each of the proposals (if there is no match, m) i =0):

[0206]

[0207] - Color histogram comparison: This is very useful for specific applications where geometric information is insufficient but color information is available (e.g., fire extinguisher detection). It can be calculated on all points from the search area included in the bounding box or only on matching superpoints; and

[0208] - Similarly, any global shape descriptor can be used on all points from the search area included in the bounding box or only on the matched superpoints.

[0209] Now for reference Figures 15 to 18 Examples of 3D object detection based on the described detection method are discussed. For example, these examples can be obtained using the aforementioned implementations of the ensemble learning method and the detection method process.

[0210] Figure 15 A screenshot shows the 3D object from which the query was extracted. The 3D object is a trash can / dust bin. Figure 15 The bounding box around the trash can is shown. Figure 16 The superpoints extracted from the trash can are shown, i.e., the superpoints that form the query. Figure 17 The search area 170 is shown. (As in...) Figure 17 As shown above, as a result of performing the detection method, groups 172, 174, and 176 are superpoints in the detection search region. These groups represent... Figure 15 Similar trash cans. Figure 18The image shows the bounding box surrounding a detected similar trash can within search area 170. For illustrative purposes, in... Figure 18 Only one bounding box is shown, but it's important to understand that more bounding boxes can be displayed.

[0211] The learning method, the detection method, and the process of integrating these methods are computer-implemented methods. This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Therefore, the steps of the method may be performed by a computer fully or semi-automatically. In the example, at least some steps of the method can be triggered through user-computer interaction. The required level of user-computer interaction can depend on the level of automation anticipated and invested in in balance with the user's desired outcome. In the example, this level can be user-defined and / or predefined.

[0212] A typical example of a computer implementation of the method is to execute the method using a system adapted for this purpose. The system may include a processor coupled to memory and a graphical user interface (GUI), on which a computer program comprising instructions for performing the method is recorded. The memory may also store a database. The memory is any hardware suitable for such storage and may comprise several physically distinct parts (e.g., one for the program and possibly one for the database).

[0213] Figure 19 An example of a system is shown, wherein the system is a client computer system, such as a user's workstation.

[0214] The client computer in the example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer also includes a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard disk drive 1030. Mass storage devices suitable for tangibly implementing computer program instructions and data include all forms of non-volatile memory, such as semiconductor storage devices like EPROM, EEPROM, and flash memory; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090 such as a cursor control device, a keyboard, etc. A cursor control device in the client computer is used to allow the user to selectively position the cursor at any desired location on the monitor 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0215] A computer program may include computer-executable instructions, which include modules for causing the system to perform one or more methods. The program may be recorded on any data storage medium, including the system's memory. For example, the program may be implemented in digital electronic circuits or computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, for example, tangibly embodied in a machine-readable storage device, for execution by a programmable processor. The method steps may be executed by a programmable processor, which executes the instruction program to perform the function of the method by manipulating input data and generating output. The processor may thus be programmable and coupled to at least one programmable processor to receive data and instructions from and send data and instructions to the data storage system. The application program may be implemented in a high-level procedural or object-oriented programming language, or, if necessary, in assembly or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation program or update program. On the system, the application program in any case causes the instructions to execute the method.

Claims

1. A method of using a computer-implemented neural network, the neural network being configured to encode superpoints in a 3D point cloud as eigenvectors, wherein, The neural network is learned in the following way: Provide a dataset of (S10-S20) superpoints, each superpoint being a set of points in a 3D point cloud, the set of points representing at least a portion of an object, and The neural network is learned (S30) based on the dataset of the superpoints, the learning including minimizing the loss, the loss penalizing the inconsistency between two superpoints, wherein the loss is a reconstruction loss, and the loss penalizing the inconsistency between the superpoint and the reconstruction of the superpoint obtained by decoding the eigenvector encoded by the superpoint. The method of use includes: - Provided (S40-S50): One or more first superpoints in a first 3D point cloud, wherein the first 3D point cloud represents a 3D scene, and One or more second superpoints in a second 3D point cloud, wherein the second 3D point cloud represents a 3D object. Providing the one or more first superpoints includes: Provide one or more initial superpoints in the first 3D point cloud; and The one or more initial superpoints are filtered by selecting each of the following initial superpoints: for each initial superpoint: The inconsistency between the dimension of the initial superpoint and the dimension of at least one of the one or more second superpoints is less than a predefined threshold; and / or The inconsistency between the position of the initial superpoint and the position of at least one of the one or more second superpoints is less than a predefined threshold. The selected initial superpoint is one or more of the first superpoints. - By applying the neural network, each of the one or more first superpoints is encoded (S60) as a corresponding first eigenvector, and each of the one or more second superpoints is encoded as a corresponding second eigenvector; - The similarity between each first superpoint in one or more first superpoints and each second superpoint in one or more second superpoints is determined by the following operation (S70): for each first superpoint and each second superpoint, a similarity measure is calculated between the corresponding first eigenvector encoded for the first superpoint and the corresponding second eigenvector encoded for the second superpoint; and - Determine (S80) one or more sets of first superpoints from a plurality of first superpoints that all have a determined similarity greater than a predefined threshold with at least one second superpoint, each set of first superpoints representing a 3D object in a 3D scene represented by the first 3D point cloud, the 3D object having a shape similar to a 3D object represented by the second 3D point cloud.

2. The method according to claim 1, wherein, The inconsistency is the distance between the superpoint and the reconstruction of the superpoint.

3. The method according to claim 2, wherein, The distance is either the Chamfer distance or the Earth-Mover distance.

4. The method according to claim 1, wherein, The learning described is unsupervised learning.

5. The method according to claim 1, wherein, The filtering includes selecting each of the initial hyperpoints as follows, for each initial hyperpoint: - The distance between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than a predefined threshold; and / or - The ratio between each dimension of the superpoint and the corresponding dimension of at least one second superpoint is less than the maximum ratio and greater than the minimum ratio; and / or - The difference between the relative height of the nearest support surface from the superpoint and the relative height of the nearest support surface from at least one second superpoint is less than a predefined threshold.

6. The method according to claim 1, wherein, The determination of the one or more groups (S80) includes: - Provide a graph of a second superpoint, the graph of the second superpoint having nodes and edges, each node representing a second superpoint, each edge representing a geometric relationship between two superpoints represented by the nodes connected by the edge and having one or more geometric properties of the geometric relationship; - One or more graphs are formed by the following operations, each graph having nodes, each node representing a first superpoint: constructing edges, each edge connecting two nodes, and each edge having one or more geometric properties similar to the edges of the graph of the second superpoint, each formed graph corresponding to a corresponding group.

7. The method according to claim 6, further comprising: For each group, a similarity score is determined by measuring the similarity between the first superpoint in the group and the one or more second superpoints.

8. A computer program product comprising instructions for performing the method according to any one of claims 1 to 7.

9. An apparatus comprising a computer-readable data storage medium having instructions thereon recorded thereon for performing the method according to any one of claims 1 to 7.

10. The device of claim 9, further comprising a processor coupled to the data storage medium.

Images