Machine Learning for 3D Segmentation

Abstract

To provide a learning method of a neural network for segmenting 3D modeled objects.SOLUTION: A learning method comprises a step of providing a dataset of training samples. Each training sample includes a pair of 3D modeled object portions labelled with respective values. The respective values indicate whether or not the two portions belong to the same segment of a 3D modeled object. The learning method further comprises a step of causing a neural network to learn based on the dataset. The neural network is configured to take, as input, two portions of the 3D modeled object representing a mechanical component and to output individual values. The individual values indicate to what extent the two parts belong to the same segment of the 3D modeled object. The neural network having learned through this learning method is usable for 3D segmentation.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This disclosure relates to the field of computer programs and systems, more specifically to machine learning methods, systems, and programs for 3D segmentation. [Background technology]

[0002] The market offers numerous systems and programs for the design, engineering, and manufacturing of objects. CAD stands for Computer-Aided Design, and refers to software solutions for designing objects, for example. CAE stands for Computer-Aided Engineering, and refers to software solutions for simulating the physical behavior of future products, for example. CAM stands for Computer-Aided Manufacturing, and refers to software solutions for defining manufacturing processes and operations, for example. In such computer-aided design systems, graphical user interfaces play a crucial role in terms of technical efficiency. These technologies can be integrated into Product Lifecycle Management (PLM) systems. PLM is a business strategy that helps companies share product data, apply common processing, and leverage enterprise knowledge to help develop products from concept to lifecycle, across the concept of an extended enterprise. Dassault Systèmes' PLM solutions (under the trademarks CATIA, ENOVIA, and DELMIA) provide an Engineering Hub for organizing product engineering knowledge, a Manufacturing Hub for managing manufacturing engineering knowledge, and an Enterprise Hub that enables enterprise integration and connectivity to the Engineering and Manufacturing Hubs. Together, these systems provide an open object model that links products, processes, and resources to enable dynamic, knowledge-based product creation and decision support, facilitating product definition, manufacturing preparation, production, and service optimization. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] Against this and other backdrops, the segmentation of 3D modeled objects (also simply called "3D segmentation" or "segmentation") is becoming increasingly important.

[0004] Given a 3D modeled object (such as a mesh), 3D segmentation typically involves dividing the geometric elements of that object (such as faces, and sometimes other elements like vertices and edges) into multiple connected clusters called segments, where each segment maximizes internal consistency while minimizing similarity to other segments. Through the segmentation process, a face is divided into several sets (segments) such that each segment has a clear boundary with respect to other segments and represents a semantically consistent set.

[0005] Segmentation can be used in the process of abstracting shapes, in which the 3D geometry and / or features are extracted from captured 3D data, as described, for example, in the reference "Kaiser A. et. al., A survey of Simple Geometric Primitives Detection Methods for Captured 3D data, Computer Graphics Forum, 2018". Segmentation can also be used in the process of constructing CSG (Spatial Domain Constructing) models (such as CSG trees) from raw 3D data, as discussed, for example, in the references "Wu. Q. et. al., Constructing 3D CSG Models from 3D Raw Point Clouds, Computer Graphics Forum, 2018, and Shapiro V. et. al., Separation for Boundary to CSG Conversion, ACM Transactions on Graphics, Vol. 12, No. 1, January 1993, Pages 35-55". These two references, in particular, use methods based on RANSAC (Random Sample Consensus) to detect the initial pool of features from which the feature tree is constructed.

[0006] However, 3D segmentation still requires improved solutions. [Means for solving the problem]

[0007] Therefore, a computer implementation method for machine learning is provided. The method includes the step of providing a dataset of training samples. Each training sample contains pairs of 3D modeled object parts labeled with a value, which indicates whether the two parts belong to the same segment of the 3D modeled object. The method further includes training a neural network on this dataset. The neural network is configured to take two parts of a 3D modeled object representing a machine part as input and output a distinct value, which indicates to what extent the two parts belong to the same segment of the 3D modeled object. This allows the neural network to be used for 3D segmentation.

[0008] The method may include one or more of the following: -Neural networks are, • An encoding neural network configured to take a portion of a 3D modeled object as input and encode it into a latent vector, and This includes a predictive neural network configured to take pairs of latent vectors output by an encoding neural network as input and output distinct values ​​indicating the extent to which the two parts encoded by the latent vectors belong to the same segment of a 3D modeled object. - The encoding neural network is configured to accept a portion of a 3D modeled object, as well as adjacent portions of its input, as input. - The learning of the neural network includes minimizing a loss that includes a term that penalizes the error between, for each pair in the dataset, each value indicating whether the two parts of a pair belong to the same segment of the 3D modeled object, and the individual values output by the neural network for the two parts. - The neural network converts the two input parts into corresponding intermediate features, and the loss further includes another term that penalizes the error between, for each pair in the dataset, each value indicating whether the two parts of a pair belong to the same segment of the 3D modeled object, and the approximation between the intermediate features corresponding to the two parts. - The terms are as follows:

Number

Number

number

[0009] Furthermore, a neural network that can be trained according to this method, for example, a neural network trained by this method, is also provided. The neural network, through training according to this method, forms a computer implementation data structure having layers of neurons whose values ​​are configurable (for example, set) weights (also called parameters).

[0010] Furthermore, a computer implementation method is provided that uses a neural network for 3D segmentation. This method includes the step of providing a 3D modeled object. This method further includes the step of applying the neural network to pairs of parts of the 3D modeled object, thereby outputting a set of values. Each value is for each pair of parts of the 3D modeled object. This value indicates the extent to which the two parts belong to the same segment of the 3D modeled object. This method further includes the step of performing segmentation of the 3D modeled object based on a set of one or more similarity indices between adjacent parts of the 3D modeled object. The set of similarity indices includes a set of values.

[0011] The steps for performing segmentation may include the following: - A step of performing network-based segmentation of the 3D modeled object based on the set of values, and then performing another segmentation of the 3D modeled object based on at least one other similarity metric, or a step of performing the other segmentation and then performing network-based segmentation, or - A step of performing segmentation of the 3D modeled object based only on the set of values, or - A step of performing segmentation of a 3D modeled object based on a set of such values ​​combined with at least one other similarity metric.

[0012] A computer program including instructions for performing this method and / or a method of using the same are further provided.

[0013] Furthermore, a device is provided that includes a computer-readable data storage medium on which the computer program and / or the neural network described above are recorded.

[0014] The device may, for example, form or function as a non-transient computer-readable medium in a SaaS (Software as a Service), other server, or cloud-based platform. Alternatively, the device may comprise a processor coupled with a data storage medium. Thus, the device may form a computer system, either entirely or partially (for example, the device is a subsystem of the entire system). The system may further comprise a graphical user interface coupled with the processor. [Brief explanation of the drawing]

[0015] [Figure 1] This method is illustrated as an example. [Figure 2] This method is illustrated as an example. [Figure 3] This method is illustrated as an example. [Figure 4] This method is illustrated as an example. [Figure 5] This method is illustrated as an example. [Figure 6] This method is illustrated as an example. [Figure 7] This method is illustrated as an example. [Figure 8] An example of the system is shown. [Modes for carrying out the invention]

[0016] A computer implementation method for machine learning is provided. This method includes the step of providing a dataset of training samples. Each training sample contains pairs of 3D modeled object parts labeled with a value (i.e., each pair is labeled with a value). The value indicates whether the two parts belong to the same segment of the 3D modeled object. The method further includes the step of training a neural network on this dataset. The neural network is configured to take two parts of a 3D modeled object representing machine parts as input and output distinct values. The distinct values ​​indicate the extent to which the two parts belong to the same segment of the 3D modeled object. This allows the neural network to be used for 3D segmentation. This method is sometimes referred to as a “training method”.

[0017] This learning method constitutes an improved solution for 3D segmentation.

[0018] In particular, this learning method trains a neural network that can be used for 3D segmentation. In other words, the neural network constitutes a tool for 3D segmentation, and can therefore be used in 3D segmentation methods.

[0019] A neural network is configured and learns to take any pair of parts of a 3D modeled object representing a machine part as input and to output a distinct value indicating the extent to which the two parts belong to the same segment of the 3D modeled object. In other words, the output of a neural network indicates the extent to which two parts can be classified as belonging to the same segment of the 3D modeled object, for example, a probability or confidence score that the two parts belong to the same segment. The term "segment" is understood as a segmentation segment, i.e., a geometrically consistent surface portion of a 3D modeled object. Thus, the output of a neural network can be directly used by segmentation algorithms or methods that use information about the extent to which two adjacent parts of a given pair belong to the same segment for (e.g., all) pairs of adjacent parts of a 3D modeled object being segmented.

[0020] Specifically, many segmentation algorithms or methods perform segmentation of a 3D modeled object based on a set of one or more similarity indices between adjacent parts of the 3D modeled object, where each similarity indice includes a set of values ​​indicating how much two adjacent parts of the 3D modeled object belong to the same segment, i.e., a set of shape similarity values ​​between adjacent parts of the 3D modeled object. A neural network may be applied to (e.g., all) pairs of adjacent parts of the 3D modeled object, and this application causes the neural network to output distinct values ​​for each pair, each indicating how much the two parts of that pair belong to the same segment of the 3D modeled object. These distinct values ​​together form a set of values ​​that constitutes a similarity indice, and segmentation algorithms or methods may be based on this set of values. Thus, the output of the neural network can be used directly in a segmentation algorithm, i.e., without preprocessing (e.g., pre-segmentation). Furthermore, the output of a neural network already provides the stopping criteria for such an algorithm: to stop the algorithm, it is sufficient to set a criterion such as, for example, that two parts are merged / combined into one segment only if the individual values ​​(output by the neural network) are greater than an acceptable threshold, and as a result, segmentation stops when only pairs of parts remain in the segmented 3D modeled object where the individual values ​​(output by the neural network) are below the threshold.

[0021] Furthermore, using the set of output values ​​from the neural network in the segmentation algorithm enables relatively fast segmentation. In fact, the similarity index is only obtained by applying the neural network to pairs of parts of the 3D modeled object, but this can be done relatively quickly once the neural network has been trained. In other words, the similarity index becomes available relatively quickly simply by applying the neural network. Therefore, once the neural network has been trained, the complex calculations required to obtain the similarity index during segmentation are avoided.

[0022] Furthermore, this learning method provides robustness to neural networks for the following reasons: Firstly, this learning method relies solely on the individual values ​​of pairs in the training dataset to perform learning, and not on explicit definitions of the parts of a pair (e.g., parameterization and / or primitive expressions). Therefore, the neural network does not require explicit definitions of its inputs to function efficiently. Secondly, the neural network outputs individual values ​​for pairs of parts based only on the part and, for example, its local context, without knowledge of the entire 3D modeled object to which the part belongs. Thus, the neural network does not depend on overall class-specific features, allowing it to generalize to shapes it has never seen before, thus improving the generalization ability of the neural network. Thirdly, the robustness of the neural network depends on the training dataset (e.g., its comprehensiveness and diversity), so if the training dataset is appropriately selected for its purpose, the neural network can be used to accurately segment any type / class of 3D modeled object.

[0023] As mentioned above, the output of a neural network can be directly used for 3D segmentation methods, that is, methods for segmenting 3D modeled objects / methods for performing 3D segmentation on 3D modeled objects. As is well known, given a 3D modeled object (such as a mesh), 3D segmentation typically involves dividing / dividing that 3D modeled object (or its faces) into multiple connected clusters called segments, where each segment maximizes internal consistency while minimizing similarity to other segments. The clusters / segments are the result of segmentation and are sometimes referred to as "segmentation."

[0024] The output of the neural network may be used in any 3D segmentation method that implements this general segmentation concept. Specifically, the output of the neural network may be used in any 3D segmentation method that takes a 3D modeled object as input and outputs a set of segments of the 3D modeled object based on one or more similarity metrics.

[0025] A set of segments is sometimes referred to as the "segmentation output / generated by a 3D segmentation method" or simply the "segmentation" (of a 3D modeled object). Each segment identified / output by a 3D segmentation method is a part of a 3D modeled object that constitutes a surface forming a consistent part of the shape of the 3D modeled object. In other words, a segment may be defined as a geometrically consistent part of the shape of a 3D modeled object. Each segment identified by a 3D segmentation method has a clear boundary with other segments and tends to maximize internal consistency while minimizing similarity with other segments. For example, each segment may form the underlying surface of a 3D modeled object that can be represented by a single CAD feature (e.g., a CATIA feature) (e.g., fitted and / or parameterizable).

[0026] Each segment may form a consistent surface from the perspective of manufacturing the machine part. In other words, in the real world, a machine part can be divided into parts, each part having a geometry that requires or is suited to a particular manufacturing process (e.g., molding, additive manufacturing, or machining), and each of these parts is represented by one or more segments in the segmentation output by the 3D segmentation method. To put it another way, the set of segments output by the 3D segmentation method may consist of one or more subsets of segments, and for each subset, one or more segments of the subset as a whole represent a consistent surface shape (i.e., material layout) of a part of the machine part from a manufacturing perspective, i.e., a surface shape that requires or is suited to a particular manufacturing process (e.g., one of several suitable manufacturing processes may be selected). For example, one or more such subsets may each represent a part of a machine part that is manufactured by machining and each has a shape (i.e., geometry) that is suited to the path of the machining tool. Additionally or alternatively, one or more other subsets may each represent parts of a machine part manufactured by molding, each having a shape (i.e., geometry) corresponding to the shape (i.e., geometry) of a consistent part of the mold. The mold itself may be manufactured by machining, and each such part may represent a shape (i.e., geometry) adapted to the path of the machining tool. Additionally or alternatively, one or more other subsets may each represent parts of a machine part manufactured by additive manufacturing, each having a shape (i.e., geometry) corresponding to the shape (i.e., geometry) of the outer final layer formed by additive manufacturing. Thus, a segmentation method using the output of a neural network enables the identification of parts of a machine part that are consistent from a manufacturing perspective.

[0027] Any method of 3D segmentation as described herein may be any 3D segmentation that segments a 3D modeled object based on one or more similarity metrics (i.e., outputs a set of segments of an input 3D modeled object). Such a method may include one or more segmentation steps. Each segmentation step performs at least a portion of the segmentation of the 3D modeled object based on one or more similarity metrics. Each segmentation step outputs segments, and therefore each segmentation step may also be referred to as a segmentation. A segmentation step may output segments of only a portion of the 3D modeled object (for example, a segmentation step performs only partial segmentation of the 3D modeled object), or a segmentation step may output initial or intermediate segments of the entire 3D modeled object that may later require at least one other segmentation step, and this segmentation step may also be referred to as “partial” (i.e., an intermediate state / result). This partial segmentation may complete one or more previous partial segmentations, each output by a separate segmentation step, and / or be completed by one or more subsequent partial segmentations, each output by a separate segmentation step. Each segmentation step may be based on a single similarity metric, i.e., the segmentation step identifies segments based on only a single similarity metric. Alternatively, the segmentation step may be based on multiple similarity metrics, in which case the segmentation step identifies segments based on these similarity metrics. A 3D segmentation method may consist of only one single segmentation step, so that a single segmentation step segments the entire 3D modeled object. Optionally, a 3D segmentation method may include any filtering of segments, as is widely known in the field of 3D segmentation.The 3D segmentation method, or any segmentation step thereof, may use any 3D segmentation algorithm. Such an algorithm may consist of iterative searching and merging of pairs of parts of a 3D modeled object based on one or more similarity metrics. The iterative searching and merging includes a step of updating the similarity metric after each partial merge, which includes a step of reapplying the neural network after each merge to update / recalculate the individual values ​​after the merge, if the output of the neural network is used as the similarity metric. Such an algorithm may be based on the 3D modeled object itself, or on any data structure representing the 3D modeled object (e.g., a dual graph for a mesh, a nearest neighbor graph for a point cloud). The algorithm may be any suitable algorithm, including, for example, hierarchical clustering or region growth algorithms.

[0028] Any similarity metric as used herein may be a set of values ​​that indicate (e.g., probability, confidence score, quantification, measurement) the degree to which two adjacent parts (also called "surfaces") of a segmented 3D modeled object belong to the same segment. A set of output values ​​from a neural network constitutes a similarity metric as described above. Alternatively, any similarity metric as used herein may be the distance between two adjacent parts, a set of distance values ​​(i.e., a set of distance values) that quantifies the shape similarity between adjacent parts. Any similarity metric as used herein may be a set of centroid curvature distance values ​​that penalize mismatches in mean curvature between surfaces (i.e., encourage / facilitate the integration of surfaces with small differences in curvature at the boundary), the distance of which can be obtained, for example, by the following formula:

number

number

number

number

number

number

number

number

number

number

number

[0029] Further applications of neural networks for 3D segmentation are provided, specifically methods for using neural networks for 3D segmentation, which include 3D segmentation methods based on one or more similarity metrics, one of which is a set of values ​​output by a neural network. This method is sometimes referred to as a "segmentation method."

[0030] A segmentation method includes the step of providing 3D modeled objects representing mechanical parts, i.e., 3D modeled objects to be segmented. The step of providing 3D modeled objects may include the step of measuring or acquiring 3D modeled objects, for example, by providing physical sensors, operating the physical sensors on the mechanical parts (e.g., scanning the mechanical parts), and performing a 3D reconstruction process to acquire 3D modeled objects. Alternatively, the step of providing 3D modeled objects may include the step of creating 3D modeled objects, for example, by sketching. In yet another alternative, the step of providing 3D modeled objects may include the step of acquiring 3D modeled objects from a database (e.g., located remotely) where the 3D modeled objects are stored after they are created or acquired.

[0031] This segmentation method then includes the step of applying a neural network to pairs of parts of a 3D modeled object. This outputs a set of distinct values ​​for each pair of parts (i.e., adjacent parts) of the 3D modeled object, indicating the extent to which the two parts belong to the same segment of the 3D modeled object.

[0032] Next, the segmentation method includes the step of performing segmentation of the 3D modeled object based on a set of one or more similarity metrics between adjacent parts of the 3D modeled object, i.e., the step of applying / performing a 3D segmentation method. The set of similarity metrics includes a set of values, i.e., the output of a neural network applied to pairs of adjacent parts of the 3D modeled object, which is used by the 3D segmentation method.

[0033] The segmentation step may include a step of segmenting the 3D modeled object based only on the set of values. In other words, the segmentation performed consists of only a single segmentation step based on the set of output values ​​of the neural network. The segmentation step may use any segmentation algorithm, such as hierarchical clustering or a region growth algorithm.

[0034] Alternatively, the step of performing segmentation may include performing network-based segmentation of the 3D modeled object based on the set of values ​​(i.e., segmentation based on the use of a neural network), and then performing another segmentation of the 3D modeled object based on at least one other similarity metric. In other words, the segmentation performed includes two segmentation steps: network-based segmentation and other segmentation. Network-based segmentation is based on a set of output values ​​of a neural network and may use any segmentation algorithm, such as hierarchical clustering or a region growth algorithm. Other segmentation is based on one or more other similarity metrics (i.e., similarity metrics other than the set of output values ​​of a neural network) and may use any segmentation algorithm, such as hierarchical clustering or a region growth algorithm. Other segmentation may use the same segmentation algorithm as used in network-based segmentation.

[0035] Other segmentation may be any example of the hierarchical segmentation disclosed in European Patent Application EP21305195.6 filed on 16 February 2021. This hierarchical segmentation includes a first segmentation that forms the segmentation step of the hierarchical segmentation. The first segmentation includes the step of identifying a first segment from the surface of the 3D modeled object, each corresponding to a simple geometric surface of the 3D modeled object. A simple geometric surface is a primitive exhibiting at least one slippable motion. Next, the hierarchical segmentation includes a second segmentation that forms the next segmentation step of the hierarchical segmentation. The second segmentation includes the step of identifying a second segment from the surface of the unidentified 3D modeled object, each corresponding to a free-form surface of the 3D modeled object. As described in European Patent Application EP21305195.6, the first segmentation comprises three segmentation steps (referred to in the said application as “executions of the exploring and merging”), each step based on a first distance, which is a value that forms a similarity index. As further described in European Patent Application EP21305195.6, the second segmentation may form a single segmentation step based on a second distance. The contents of European Patent Application EP21305195.6, in particular the description of hierarchical segmentation and its features and examples, including the implementation using the disclosed hierarchical graph clustering, are incorporated herein by reference.

[0036] Alternatively, the step of performing segmentation may include a step of performing segmentation of the 3D modeled object based on a set of values ​​combined with at least one other similarity metric. In other words, the segmentation performed includes a single segmentation step based on a set of values ​​combined with at least one other similarity metric, i.e., a similarity metric formed by combining a set of values ​​with one or more other similarity metrics. This similarity metric may be, for example, a set of values ​​that indicate (e.g., probability, confidence score, quantification, measurement) how much two adjacent parts (also called "surfaces") of the 3D modeled object belong to the same segment, where each value is a combination of an individual value output by a neural network for the two parts and one or more values ​​of each similarity metric indicating how much the two parts belong to the same segment. For example, this value may be the sum of the individual value and one or more similarity values ​​that quantify the shape similarity between the two parts (e.g., each value obtained from their respective distances). Alternatively, this value may be the individual value multiplied by one or more similarity values. Alternatively, this value may be the average of the individual value in question and one or more similarity values.

[0037] As is widely known in the field of CAD manufacturing, segmentation, such as the segmentation produced by this segmentation method or any other segmentation method using neural network output, can be used in a variety of applications or processes that require or demand segmentation at some stage. Segmentation improves the representation of the outer surface of a 3D modeled object and can be useful in many applications. Furthermore, segments become parameterized and editable in CAD tools, and after segmentation, these segments can be manipulated (edited, etc.), which can also be used in many applications. "Parameterization" means that each segment can be fitted to exactly one 3D geometric object represented by a parameterized equation or parameterized function, and thus each segment can contain one or more parameters, each of which can take values ​​within a certain continuous range. Parameterized 3D geometric objects, in contrast to non-parameterized 3D geometric objects such as discrete representations (point clouds, meshes, dexel representations, etc.), allow for easy manipulation and / or editing and / or efficient storage to memory. For example, segments may be fitted to canonical primitives (e.g., planes, spheres, or cylinders) or parameterized with other fitting geometry tools (e.g., parameterized freeform surfaces such as NURBS or extruded surfaces). In any application of segmentation, including those described below, the 3D modeled object may be a measured 3D modeled object, and therefore, when a 3D modeled object is segmented, a measured 3D modeled object is processed, and ultimately, the measured 3D modeled object (when segmented) becomes editable. Thus, segmentation may generally be used to segment a measured 3D modeled object and then process it into an editable data structure.

[0038] In the first application, segmentation, such as the segmentation output by this segmentation method or any other segmentation method that uses neural network output, is used to construct B-reps. B-rep construction using segments is described in the references "P. Benko et. al., Algorithm for reverse engineering boundary representation models, Computer-Aided Design 33 (2001), 839-851", "A. Tumanin, Polygonal Mesh to B-Rep Solid Conversion: Algorithm Details and C++ Code Samples" published on Habr.com on September 4, 2019, and "Beniere et. al., Recovering Primitives in 3D CAD meshes, Proceedings of SPIE, 2011", all of which are incorporated herein by reference. This first application involves using segmentation of 3D modeled objects to convert them into boundary representations (i.e., B-reps, which are sets of connected boundary surface elements, as is well known, for example in the STEP file format). This conversion may involve fitting the surface to each segment and using segmentation data to boundary the surface (i.e., determining the B-rep's phase data, i.e., the "- is bounded by -" relationships). Thus, this segmentation method or any other segmentation method using neural network output may be included in a computer implementation process for converting 3D modeled objects representing mechanical parts into B-reps, and this process is -By performing segmentation, the segmentation of the 3D modeled object is provided, and - This includes fitting the surface to each segment of the segmentation and defining the surface based on the segmentation, thereby converting the 3D modeled object to a B-rep according to a known method suitable for such a conversion.

[0039] In a second application, segmentation, such as the segmentation output by this segmentation method or any other segmentation method that uses neural network output, is used to construct feature trees. Segment-based feature tree construction (e.g., fitting primitives to segments) is described in the references "T. Du et. al., InverseCSG: Automatic Conversion of 3D Models to CSG Trees, ACM SIGGRAPH ASIA 2018," "Wu. Q. et. al., Constructing 3D CSG Models from 3D Raw Point Clouds, Computer Graphics Forum, 2018," and "Shapiro V. et. al., Separation for Boundary to CSG Conversion, ACM Transactions on Graphics, Vol. 12, No. 1, January 1993, Pages 35-55," all of which are incorporated herein by reference. This second application involves using segmentation to construct a feature tree representation of a 3D modeled object. This construction involves first using segmentation to fit primitives into each segment to build a pool of CAD features (also known as a "feature list," from which the feature tree is then constructed). Therefore, this segmentation method, or any other segmentation method using neural network output, may be included in a computer implementation process for constructing a feature tree from a 3D modeled object representing a mechanical part, and this process is: - Provides segmentation of 3D modeled objects by performing segmentation. - Constructing a pool of CAD features (i.e., a feature list) by fitting surfaces to each segment of the segmentation according to known methods, and This includes constructing a feature tree from the fitted surface according to known methods.

[0040] In the third application, segmentation, such as the segmentation output by this segmentation method or any other segmentation method that uses the neural network output, is used for remeshing (if the provided 3D modeled object is a 3D mesh) or resampling (if the provided 3D modeled object is a 3D point cloud). The segments obtained by segmentation may be parameterized by canonical primitives and / or NURBS, so that the surface definition of each segment can be used to remeshing (if the provided 3D modeled object is a 3D mesh) or resampling (if the provided 3D modeled object is a 3D point cloud) the 3D modeled object. This remeshing / resampling may be used to remove noise from the 3D modeled object (for example, removing outlier points, especially in the case of a 3D point cloud, or smoothing the outer surface of the 3D modeled object, especially in the case of a 3D mesh). Additionally or alternatively, the 3D mesh may be efficiently tessellated using remeshing / resampling, that is, the number of faces may be minimized to optimize the weight (in storage) of the mesh, while the size of the mesh faces may be adapted to the curvature of the corresponding surface to ensure the maximum discretization distance to the accurate surface. Thus, this segmentation method or any other segmentation method using neural network output may be included in a computer implementation process for remeshing (resampling) 3D model objects of 3D meshes (3D point clouds) representing mechanical parts, and this process is - Provides segmentation of 3D modeled objects by performing segmentation. -Parameterizing each segment with canonical primitives and / or NURBS, thereby creating surface definitions for all segments within the segmentation, and - Remeshing (resampling) the 3D mesh (3D point cloud) using the surface definition of each segment according to known appropriate methods. -Optional, • Removing noise from a 3D mesh (3D point cloud) using remeshing (resampling), and / or This includes one or more of the following: optimizing the weight of the mesh (in storage) while ensuring the maximum discretization distance to the correct surface by efficiently tessellating the 3D mesh by matching the size of the mesh faces to the curvature of the corresponding surface.

[0041] In the fourth application, segmentation, such as the segmentation output by this segmentation method or any other segmentation method that uses the neural network output, is used to detect anomalies (e.g., manufacturing defects) in 3D modeled objects. In this application, available knowledge about the characteristics that a machine part should have may be compared with the segments of the segmentation, thereby providing information about defects or anomalies in the machine part. For example, available knowledge may include information such as "the cubic portion of a machine part should have six large planes" or "a normal machine part should not have small face portions." If the 3D modeled object portion representing a cube is segmented into seven faces, or if some segments represent non-planar faces, this application may include estimating anomalies in the 3D modeled object, such as indicating that the machine part is anomaly (i.e., has a manufacturing defect).

[0042] The segmentation output by this segmentation method or any other segmentation method that uses neural network output may be used in other applications of 3D shape segmentation, such as 3D deformation, 3D rendering (calculation of geometry / material attributes, occlusion culling, shadow determination), 3D animation, and / or shape compression. These applications are discussed in the previously cited reference, "Kaiser A. et. al., A survey of Simple Geometric Primitives Detection Methods for Captured 3D data, Computer Graphics Forum, 2018," which is incorporated herein by reference.

[0043] Figures 1-6 show the segmentation of machine parts, with each figure representing a segmented machine part.

[0044] The learning method will be explained further below.

[0045] As is well known in the field of machine learning, processing input by a neural network involves applying an operation to the input, where the operation is defined by data containing weight values. Therefore, training a neural network involves the step of determining weight values ​​based on a dataset configured for such training, such a dataset may be called a training dataset or learning dataset. Thus, a dataset contains data pieces, each forming a training sample. Training samples represent a variety of situations in which the neural network will be used after training. Any training dataset in this specification may contain more than 1,000, 10,000, 1,000,000, or 1,000,000 training samples. In the context of this disclosure, “training a neural network based on a dataset” means that the dataset is the training dataset for the neural network and that the weight values ​​(also called “parameters”) are set based on it.

[0046] In terms of the learning method, the training dataset is a dataset of provided training samples, each training sample containing pairs of 3D modeled object parts, each labeled with a value indicating whether the two parts belong to the same segment of the 3D modeled object. Before explaining how the dataset is provided, let's describe the data structure included in the dataset.

[0047] Each training sample contains pairs of 3D modeled object parts, i.e., pairs of parts, each representing a part of a 3D modeled object. A part of a 3D modeled object is a surface of the 3D modeled object, i.e., a part of the outer surface of the 3D modeled object. Points of a 3D modeled object are also considered parts in this disclosure, meaning that a neural network can accept parts that are points and parts that are geometric surfaces as input. In the context of segmentation, which involves iteratively integrating surfaces / parts of 3D modeled objects, some parts may be the result of integrating other parts. Any part of any pair in the training dataset may also be the result of integrating other parts, making the dataset particularly suitable for training a robust neural network that can be used for 3D segmentation.

[0048] Therefore, this learning method manipulates modeled objects. A modeled object is any object defined by data stored, for example, in a database. Thus, the expression "modeled object" refers to the data itself. Depending on the type of system, modeled objects may be defined by various types of data. A system can actually be any combination of CAD systems, CAE systems, CAM systems, PDM systems, and / or PLM systems. In these various systems, modeled objects are defined by corresponding data. Therefore, CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, and CAE data can be discussed. However, since modeled objects can be defined by data corresponding to any combination of these systems, these systems are not exclusive to each other. Therefore, a system can be both a CAD and a PLM system, as will be evident from the definition of such a system described below.

[0049] In this specification, any 3D modeled object may represent a product, such as a part or an assembly of parts, or in some cases, an assembly of a product. “3D modeled object” means an object modeled by data that enables a 3D representation. 3D representation allows parts to be viewed from all viewpoints (e.g., from various angles and / or distances). For example, a 3D modeled object, when represented in 3D, may be processed and rotated based on any axis of its or any axis of the displayed screen. Thus, 2D icons, in particular, that are not modeled in 3D are excluded. A 3D modeled object may represent the geometry of a product after its virtual design has been completed, for example, by a CAD software solution or CAD system, and the product may be a (e.g., mechanical) part or an assembly of parts (parts and assemblies of parts are equivalent, as an assembly of parts may be considered the part itself from the perspective of this method), or more generally, any rigid body assembly (e.g., a movable mechanism). CAD software solutions enable the design of products in a wide range of industrial sectors, including aerospace, architecture, construction, consumer goods, high-tech equipment, industrial equipment, transportation, marine, and / or offshore oil / gas production or transportation.Therefore, the 3D modeled object may be any machine part or industrial product, such as parts for land vehicles (e.g., automobile and light truck equipment, racing cars, motorcycles, truck and motor equipment, trucks and buses, trains, etc.), parts for aircraft vehicles (e.g., airframe equipment, aerospace equipment, propulsion equipment, defense products, aircraft equipment, space equipment, etc.), parts for marine vehicles (e.g., naval equipment, commercial ships, offshore equipment, yachts and workboats, marine equipment, etc.), general machine parts (e.g., industrial manufacturing machinery, heavy machinery or equipment, installation equipment, industrial equipment products, metalworking products, tire manufacturing products, etc.), electrical machinery or electronic components (e.g., home appliances, security and / or control and / or measurement products, computing and communication equipment, semiconductors, medical devices and equipment, etc.), consumer goods (e.g., furniture, home and garden products, leisure goods, fashion products, products of durable goods retailers, products of textile retailers, etc.), and packaging (e.g., food and beverages and tobacco, beauty and personal care, household goods packaging, etc.).

[0050] Any 3D modeled object described herein can form a discrete geometric representation of a 3D real-world object, such as a machine part. A discrete geometric representation, as described herein, is a data structure containing a discrete set of data fragments. Each data fragment represents a geometric entity located in 3D space. Each geometric entity represents a location on a 3D object (in other words, a part of the material that makes up the solid represented by the 3D object). A collection of geometric entities (i.e., a combination or juxtaposition) as a whole represents at least a portion of a 3D object. In this specification, any discrete geometric representation may contain, for example, more than 100, 1000, or 10000 data fragments.

[0051] Any discrete geometric representation in this specification may be, for example, a 3D point cloud where each geometric entity is a point. Any discrete geometric representation in this specification may be, for example, a 3D mesh where each geometric entity is a tile or face of a mesh. A 3D mesh in this specification may be regular or irregular (i.e., composed of or not composed of homogeneous faces). A 3D mesh in this specification may be a polygonal mesh, for example, a triangular mesh. A 3D mesh in this specification may be a B-Rep. A 3D mesh in this specification may be obtained from a 3D point cloud, for example, by triangulating the 3D point cloud (for example, by Delaunay triangulation). A 3D point cloud in this specification may be determined by physically measuring a real object, for example, in a 3D reconstruction process. A 3D reconstruction process may include the steps of providing a real object, providing one or more physical sensors, each configured to acquire a different physical signal, and operating the one or more physical sensors on the real object to acquire one or more physical signals, respectively (i.e., scanning the real object with each sensor). The 3D reconstruction may then automatically determine a 3D point cloud and / or 3D mesh based on measurements, according to any known technique. One or more sensors may include multiple (e.g., RGB, and / or image or video) cameras, and the determination may include structure-from-motion analysis. One or more sensors may optionally or additionally include one or more depth sensors (e.g., on an RGB-depth camera), and the determination may include 3D reconstruction from depth data. One or more depth sensors may include, for example, a laser (e.g., LiDAR) or an ultrasonic emitter-receiver. Alternatively, any discrete geometry representation in this specification may be any kind of CAD model.

[0052] Alternatively, the 3D point cloud or 3D mesh described herein may be obtained from a 3D modeling object representing the skin (i.e., outer surface) of a solid (for example, a B-Rep model representing the skin, i.e., corresponding to an exact surface) by, for example, raycasting to the 3D modeling object or tessellating the 3D modeling object. Tessellation may be performed according to any rendering process of the 3D modeling object. Such a rendering process may be coded on any CAD system to display a graphic representation of the 3D modeling object. The 3D modeling object may be designed by or created by a CAD system user.

[0053] Providing a dataset may include forming the dataset. Forming the dataset may include the steps of providing an initial dataset of 3D modeled objects or initial datasets of 3D modeled object parts, and labeling pairs of 3D modeled object parts with their respective values. Alternatively, forming the dataset may include the step of directly forming training samples, which are pairs of parts that have already been labeled with their respective values ​​(for example, by randomly creating pairs of labeled parts from a CAD model).

[0054] In the example, providing a dataset includes the step of providing an initial dataset of segmented 3D modeled objects. This may include forming an initial dataset of segmented 3D modeled objects, for example, by providing a dataset of unsegmented 3D modeled objects and segmenting that dataset, or by accessing a database (e.g., remotely) and downloading an initial dataset from the database for already segmented 3D modeled objects. Providing an initial dataset may include the step of selecting segmented 3D modeled objects within a dataset of CAD models with ground truth data (e.g., parameterized), such as an ABC dataset. That is, each segment may have ground truth data representing the characteristics of the segment, such as a label indicating the segment.

[0055] Next, providing the dataset includes the step of creating pairs of parts of the segmented 3D model object for each segmented 3D model object in the initial dataset. For each pair, the paired parts belong to either the same segment or different segments of the segmented 3D model object. In other words, the pair is created such that the two parts of the pair belong to either the same segment or different segments of the segmented 3D model object. For each 3D model object, the pair creation may include the step of iteratively and randomly selecting parts of the 3D model object and forming pairs with these parts. This may include the step of merging parts, meaning that some pairs may each contain one or more parts resulting from the merging of other parts. Thus, pairs in the dataset may contain parts of different sizes between pairs or even within pairs. Therefore, pairs of parts corresponding to different levels of segmentation of the 3D model object can be included in the training dataset, improving the robustness of the neural network after training. This allows the neural network to process pairs with parts of different sizes (corresponding to different segmentation levels), ensuring that the neural network functions equally well at any level / stage of segmentation. In fact, such pairs arise in segmentation algorithms that repeatedly merge parts of a 3D modeled object to form segments, resulting in adjacent parts that can be of different sizes.

[0056] The creation is performed such that, for one or more pairs (e.g., most pairs, e.g., all pairs), each region of the pair belongs to each single segment of the segmented 3D modeled object, that is, the region completely belongs to each single segment. The two regions of a pair may both belong to the same segments or different segments. With respect to one or more segmented objects in the initial dataset, the creation of pairs may include the step of creating pairs of parts where at least one part does not completely belong to a single segment. In other words, for each of one or more (e.g., only some) segmented 3D modeled objects, at least one pair of parts of the 3D modeled object is created such that it includes at least one part that does not completely belong to a single segment of the 3D modeled object (i.e., the part includes a first subpart belonging to a first segment and a second subpart belonging to a second segment, where the first and second segments are different segments). Parts that do not completely belong to a single segment are sometimes called "impure" parts. Training a neural network with pairs of parts that may exhibit some degree of impurity allows it to learn to adapt to potential errors that may occur during segmentation (e.g., by the neural network itself). In other words, a neural network is fed a pair of parts, and even if at least one of them contains impurities, the neural network can still output satisfactory values ​​for these parts (i.e., the neural network is trained to do its best despite the errors it may encounter).

[0057] Each pair contains a value indicating whether the two parts belong to the same segment of the 3D modeled object. In other words, creating a pair may include the step of labeling each pair with a value indicating whether the two parts of the pair belong to the same segment. Labeling may be based on ground truth data. In other words, ground truth data may be used to determine whether the two parts of a pair belong to the same segment. For example, labeling may include the step of using the labels of each segment to which the two parts belong in order to label the pair, for example, labeling it "1" if the two parts belong to the same segment and "0" otherwise.

[0058] In either case, each training sample contains a pair of parts of a 3D modeled object, each labeled with a value indicating whether the two parts belong to the same segment of the 3D modeled object. If the two parts belong to the same segment, the value may be "1"; otherwise, it may be "0". Next, we train a neural network with these training samples, the training process and neural networks will be explained in more detail below.

[0059] The neural network is configured to take two parts of a 3D modeled object representing a machine part as input and output distinct values ​​indicating the extent to which the two parts belong to the same segment of the 3D modeled object. The output distinct values ​​are of the same kind as the respective values ​​of the pairs in the training dataset. The values ​​may be confidence scores or probabilities (i.e., numbers belonging to [0,1]), with higher values ​​indicating a higher confidence or probability that the two parts belong to the same segment. The neural network may have any architecture suitable for taking two parts as input and outputting distinct values.

[0060] In particular, the neural network may include an encoding neural network configured to take a portion of a 3D modeled object as input and encode it into a latent vector. A latent vector is an intermediate feature that encodes the input portion. The encoding may have meaning in terms of segments, as the encoding neural network learns to encode two portions belonging to the same segment in a relatively similar manner. The encoding neural network is sometimes called an "encoder". In addition to a portion of the 3D modeled object, the encoding neural network may also take the index of a point in that portion (for example, the index of a point in the point cloud if the 3D modeled object is a point cloud, or the index of a mesh vertex if the 3D modeled object is a mesh) as input. The encoding neural network may be configured to take a portion of the 3D modeled object, as well as adjacent portions of its input portion, as input. In other words, the encoding neural network may take the adjacent portions of its input portion as well as its input portion. For example, an encoding neural network may have a point-by-point neighbor pooling architecture that creates a latent vector / intermediate feature that takes neighbors into account, meaning the latent vector also encodes the neighboring data of the input portion, thereby further improving the segmental meaning of the encoding.The point-by-point neighbor pooling architecture is PointNet++ (described in the reference “C. Qi, L. Yi, H. Su, L. Guibas, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, in NIPS, 2017” incorporated herein by reference), or if the 3D modeled object is a 3D point cloud, DGCNN (described in the reference “Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. Bronstein, J. Solomon, Dynamic Graph CNN for Learning on Point Clouds, in ACM, 2019” incorporated herein by reference), or if the 3D modeled object is a 3D mesh, MeshCNN (described in the reference “R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. lFleishman, D. Cohen-Or, MeshCNN: A Network with an Edge, in ACM,” incorporated herein by reference). This architecture may be any architecture, such as the one described in "2019". This architecture may be combined with a Point-Net-like architecture that creates a single latent vector representing the input portion (described in the reference "C. Qi, H. Su, K. Mo, L. Guibas. PointNet, Deep Learning on Point Sets for 3D Classification and Segmentation, in CVPR, 2017", which is incorporated herein by reference).

[0061] Furthermore, the neural network may also include a predictive neural network configured to take pairs of latent vectors output by an encoding neural network as input and output distinct values ​​indicating the extent to which the two parts encoded by the latent vectors belong to the same segment of the 3D modeled object. These distinct values ​​are of the same kind as the values ​​of each training sample (for example, confidence scores or probabilities, as described above).

[0062] Training a neural network may include a step to minimize loss. The loss may be any loss suitable for training the neural network to output distinct values ​​indicating the extent to which the input pairs belong to the same segment. The loss may include a term that penalizes the error between the value indicating whether the two parts of a pair belong to the same segment of a 3D modeled object and the distinct value that the neural network outputs for those two parts, for a pair of datasets (e.g., a pair of minibatches). In other words, this term may be a function of the distinct value that the neural network outputs for the two parts of a pair and the value that labels the pair to indicate whether the two parts belong to the same segment, and this term may tend to be large if the value indicates that the two parts belong to (or do not belong to) the same segment, while the distinct value output by the neural network tends to indicate that the two parts do not belong to (or do belong to) the same segment.

[0063] The neural network may convert the two input parts into corresponding intermediate features. For each part, the corresponding intermediate feature encodes that part, and the neural network may output distinct values ​​for the two parts based on the corresponding encoding (i.e., the intermediate feature) of that part. For example, in the above case where the neural network includes an encoding neural network and a predicting neural network, the intermediate feature is a latent vector that encodes the corresponding part. In either case, the loss may further include another term that penalizes the error between the approximation between the respective values ​​indicating whether the two parts of a given pair belong to the same segment of the 3D modeled object and the intermediate features corresponding to those two parts, for each pair of the dataset (i.e., the same pair included in the loss term above). Specifically, approximation is an arbitrary function that quantifies the closeness of intermediate features (e.g., latent vectors) (e.g., closeness in terms of the distance between latent vectors, and approximation is based on such distances), and the other term is a function that tends to have large values ​​when approximation indicates that the two parts have similar (or different) intermediate features, but each value tends to indicate that the two parts do not (or do) belong to the same segment. Thus, the other term contributes to training the neural network (e.g., its encoding neural network) to encode the input parts into intermediate features (e.g., latent vectors) that are already meaningful in terms of segments, that is, intermediate features (e.g., latent vectors) that are close (e.g., in terms of the distance between latent vectors) if the two input parts belong to the same segment, and not close otherwise. This further improves the robustness of the neural network.

[0064] The terms may be as follows:

number

Number

Number

[0065] If training is performed per mini-batch, ((X1 (1) , X2 (2) ), , , (X N (1) , X N (2) )) may be pairs forming a minibatch, where N is the size of the minibatch. If the neural network includes an encoding neural network and a predicting neural network, f w is an encoding neural network, g w is a predictive neural network, f w (X i (1) ) and f w (X i (2) ) are each part X i (1) and X i (2) This is the latent vector that encodes the result.

[0066] d CE This can be a loss that penalizes the neural network output value by comparing the input pairs to the respective labeling values, for example, a binary cross-entropy loss. CE Therefore, the neural network can predict / output individual values ​​according to the knowledge belonging to the correct part. For example, d CE The following may be the case:

number

number

[0067] The loss value may be given by the following formula:

number

[0068] The loss may include other terms designed to cause the neural network to predict one or more meaningful quantities (such as normals or curvature), such as one or more losses along with each trade-off parameter. This trains the neural network to encode features that contain explicitly meaningful information for partial comparison.

[0069] Neural network training may be performed on a mini-batch basis. Training may use any mini-batch stochastic optimization algorithm (such as stochastic gradient descent or Adam optimization). In such cases, providing the dataset further involves creating paired minibatches. In other words, minibatches in mini-batch training are created beforehand, i.e., before training. Thus, mini-batch training relies solely on minibatches that perfectly reflect the correct answers for the segmented 3D modeled objects contained in the training dataset. Creating minibatches may include the step of selecting pairs to form paired minibatches. This may include the step of selecting pairs formed with parts of different sizes, and / or pairs having various parts of different sizes between pairs, and / or pairs containing at least one impure part (for example, with respect to each mini-batch or at least a portion thereof). This provides minibatches that have parts corresponding to various levels of clustering and, as mentioned above, also account for errors that occurred during training if there are pairs containing at least one impure part.

[0070] Next, I will explain how to implement the learning method.

[0071] This implementation uses a trained neural network to segment any 3D shape into meaningful parts, leveraging deep learning techniques to achieve both robustness to noise and the concept of meaningful parts directly derived from ground truth segmentation of the training dataset. This implementation introduces an improved neural network that learns to predict whether two parts of a 3D shape (also called "subparts") belong to the same segment (also called "parts" or "related parts"), enabling iterative segmentation of 3D shapes.

[0072] The implementation of the learning method first includes the step of providing a training dataset, which includes the step of creating unsegmented intermediate clusterings of these shapes at various completion levels, so that it takes a dataset of 3D shapes labeled with ground truth decomposition parts as input. Next, the implementation includes taking two subparts of a given shape as input and training a neural network to predict the probability that those two subparts belong to the same part. To train the network, a mini-batch training plan such as the following is used: 1) Create sets of pairs of multiple subparts of various shapes and clustering levels (create minibatches). 2) For these pairs of subparts, along with the intermediate features that encode each subpart, the network's prediction is computed. 3) To optimize the prediction probability of belonging to the same part, the neural network weights are designed to minimize both the loss in prediction and the loss in intermediate feature distance for matching ground truth subparts.

[0073] Neural networks can ultimately be used in combination with other terms in segmentation algorithms such as hierarchical clustering algorithms to represent the distance between multiple subparts (also called a "similarity index"). This, along with a threshold for network prediction, enables 3D shape segmentation. This implementation makes the following possible: - By combining this with distance-based segmentation algorithms and learning the probability that two subparts belong to the same part, it becomes possible to not fix the number of segmented parts, unlike end-to-end segmentation learning methods. - No explicit definition of the part (e.g., a primitive expression) is required. - The network predicts the probability of pairing multiple subparts based only on multiple subparts and their local conditions, without knowledge of the entire object. Therefore, because the network does not depend on the overall class-specific features, it facilitates generalization to shapes never seen before. - The learned integrated probability can be easily combined with other relevant metrics of the clustering algorithm (purity score, curvature distance, etc.). -By predicting the combined probability, that value can be directly used as a reliability score, and may, for example, be used as a stopping criterion for clustering algorithms. - Training consistent metrics on encoded features increases the robustness of network predictions by forcing meaningful features. - No initialization of subparts is required. - The desired types of parts to be segmented can be easily adjusted via the training dataset.

[0074] Mini-batch creation The inputs used in implementing this learning method are an initial dataset of 3D shapes, either a point cloud or a 3D mesh, and the labels of the ground truth parts to which each point of each 3D shape belongs. Providing the training dataset includes centering each shape and scaling it to a unit sphere so that the maximum distance to the center is 1. Providing the training dataset also includes discarding ambiguous points, i.e., points that lie on the intersection of at least two parts and do not have a unique label as possible.

[0075] Next, providing the training dataset involves creating pairs of multiple subparts, which form input minibatches for the neural network. The creation focuses on predicting the clustering of adjacent subparts (from the points with the greatest connectivity in the mesh or the point cloud) so that the neural network is trained to predict whether pairs of subparts belong to the same meaningful part, and then used as the clustering criterion in the clustering algorithm. Furthermore, to ensure that the neural network functions equally well at every stage of the clustering algorithm, the creation step creates adjacent subparts of different sizes, representing different levels of clustering.

[0076] Therefore, for each input shape, the creation step creates a corresponding graph, where the nodes of the graph initially correspond to points, and the edges correspond to mesh edges if it is a 3D mesh, or the maximum pairwise distance if it is a point cloud. To create different levels of clustering, the creation step repeatedly and randomly merges connected nodes of the same part's label. At some point, the graph nodes correspond to a set of points belonging to a subpart of a labeled part of the 3D shape, and the edges correspond to two adjacent subparts. Next, the creation step selects a random set of edges from the various levels of graph clustering and provides them as input to the neural network. Thus, a ground truth clustering criterion for the edges becomes available: 1 if both nodes represent a subpart of the same labeled part, and 0 if the corresponding parts are different. To efficiently train the neural network, this implementation may include sampling edges to be clustered and edges not to be clustered in predetermined proportions.

[0077] To make the neural network's predictions more robust when applied to real-world data, the creation step may involve creating multiple impure subparts. When creating clusterings of different levels of a graph corresponding to the input 3D shape, this may include merging nodes that do not represent the same part with a given probability. Beyond a certain level of impurity (measured as the number of points in a node that do not belong to the most represented part), the node is treated as if it belonged to the most represented part (with respect to the ground truth clustering criteria and subsequent graph clustering). By training the neural network with multiple subparts that can exhibit some degree of impurity, the neural network can be trained to adapt to errors that may occur at some point during the clustering of real-world 3D shapes.

[0078] Network Architecture A neural network consists of a subpart encoder followed by a predictive network.

[0079] Subpart encoder f wIt accepts a 3D shape (point cloud or mesh) and the indices of points belonging to a subpart as input. The subpart encoder first uses a point-by-point neighbor pooling encoder architecture to create a neighborhood-aware feature for each point in the subpart. Such architectures may be based on, for example, PointNet++ (described in the aforementioned reference "C. Qi, L. Yi, H. Su, L. Guibas, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, in NIPS, 2017"), or, in the case of point clouds, DGCNN (described in the aforementioned reference "Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. Bronstein, J. Solomon, Dynamic Graph CNN for Learning on Point Clouds, in ACM, 2019"), or in the case of meshes, MeshCNN (described in the aforementioned reference "R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, D. Cohen-Or, MeshCNN: A Network with an Edge, in ACM, 2019"). Once these point-by-point features are acquired, they are concatenated into subpart equivalents and passed through a PointNet-like architecture (described in the aforementioned reference "C. Qi, H. Su, K. Mo, L. Guibas. PointNet, Deep Learning on Point Sets for 3D Classification and Segmentation, in CVPR, 2017") that creates a single feature representing the subpart. Using a neighborhood pooling architecture improves training because, when clustering the mesh, multiple initial subparts are transformed into a single point, requiring its local neighborhood to compute meaningful features.

[0080] Multiple input subparts X1 and X2 both have two feature vectors fw (X1) and f w When encoded into (X2), these vectors become the predictive network g w This is given as input. Given two feature vectors h1 and h2, this network will determine the corresponding clustering probability g w The network predicts (h1, h2) ∈ [0, 1] and indicates the certainty that these features represent multiple subparts belonging to the same part. The network first concatenates both features, applies a series of multilayer perceptrons (MLPs) to this concatenation, and then a final sigmoid activation follows to limit the output to "0, 1".

[0081] training The learning process involves N pairs of mini-batches of multiple 3D shape subparts (S1 (1) S2 (2) ), , , (S N (1) S N (2) By minimizing the loss L(w) calculated for )), minibatch stochastic optimization algorithms (such as stochastic gradient descent and Adam optimization algorithms) can optimize the network f w and g w Train both of them. ((X1 (1) , X2 (2) ), , , (X N (1) , X N (2) )) to subpart encoder f w The corresponding inputs, i.e., the point cloud of the shape (or, in the case of a mesh, the point cloud to which the phase is ultimately attached) and the points corresponding to the subparts, are indexes related to the shape point cloud, p1, ..., p N Let be the correct binary variable, which is 1 if the corresponding pair of subparts belong to the same part and need to be merged, and 0 otherwise. The loss is given by the following formula:

number

number

[0082] The neural network may be configured to predict quantities such as normals and curvature, along with corresponding loss and trade-off parameters for L(w). This trains the network to encode features that contain explicitly meaningful information for subpart comparison.

[0083] Figure 7 shows the neural network architecture used in this implementation and the quantities identified by the training loss (metrics and combined probability).

[0084] Usage in segmentation algorithms Once training is complete, the neural network predicts the probability that any pair of subparts belong to the same part. This information can be used in distance-based segmentation algorithms, as described above, to segment the input shape into meaningful parts. As described above, such algorithms may be, for example, hierarchical clustering or region growth algorithms. For example, using a hierarchical clustering algorithm, the distance between each pair of clusters may be set to the inverse of the network prediction when applied to the pair, that is, the pairs of clusters are merged in descending order of the network prediction, and the stopping criterion can be set as the prediction threshold.

[0085] The network may be used in combination with other distances within such a segmentation algorithm. For example, the interchangeable distance of normals and / or curvatures may be combined with the network's integrated probability prediction to obtain another formula for subpart distance. Another neural network may be trained to predict the purity of subparts of a shape, i.e., the probability that a subpart occupies a single meaningful part. Such a neural network may be used to predict the purity of a combination of two subparts, and may also be combined with the integrated prediction of the neural network trained in this implementation.

[0086] This method is implemented on a computer. That is, the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of this method are performed by a computer, possibly fully automatically or semi-automatically. In the example, the initiation of at least some steps of this method may be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the need to implement the user's wishes. In the example, this level may be user-defined and / or predefined.

[0087] A typical example of a computer implementing this method is running it on a system adapted for this purpose. The system may include a processor coupled with memory and a graphical user interface (GUI), where memory stores computer programs containing instructions for performing this method. Memory may also store a database. The memory is hardware adapted for such storage and may include several physically distinct components (e.g., one for the program and possibly one for the database).

[0088] Figure 8 shows an example of a system, which is a client computer system, such as a user's workstation.

[0089] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The client computer further includes a video random access memory 1100 and associated graphics processing unit (GPU) 1110 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices like EPROMs, EEPROMs, and flash memory devices, magnetic disks like internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the above may be complemented by or incorporated into specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to the network 1060. The client computer may also include haptic devices 1090, such as a cursor control device and a keyboard. A cursor control device is used in a client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the mouse buttons are used to generate signals. Alternatively or additionally, the client computer system may include a pressure-sensitive pad and / or a pressure-sensitive screen.

[0090] The computer program may include instructions that can be executed by the computer, and the instructions include means for causing the system to perform the Method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented, for example, as a device such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The steps of the Method may be performed by a programmable processor that executes a program of instructions to perform the functions of the Method by acting on input data and producing output. Thus, the processor may be programmable or coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. The application program may be implemented in a high-level procedural programming language or an object-oriented programming language, or in assembly language or machine language, as necessary. In any case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In any case, the application of the program on the system results in instructions for performing the Method.

Images