Machine learning for 3D segmentation

Abstract

The present disclosure relates to a computer-implemented method of machine learning, the method comprising providing a dataset of training samples. Each training sample comprises a pair of 3D modeled object parts, which pair is labeled with a respective value. The respective value indicates whether the two parts belong to the same segment of a 3D modeled object. The method further comprises learning a neural network based on the dataset. The neural network is configured for taking as input two parts of a 3D modeled object representing a mechanical component, and for outputting a respective value. The respective value indicates a degree to which the two parts belong to the same segment of the 3D modeled object. The neural network is thereby usable for 3D segmentation. The method constitutes an improved solution for 3D segmentation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer programs and systems, and more specifically, to a method, system and program for machine learning for 3D segmentation. BACKGROUND

[0002] Many systems and programs for the design, engineering and manufacturing of objects are available on the market. CAD is an acronym for Computer-Aided Design, e.g. CAD relates to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, e.g. CAE relates to software solutions for simulating the physical behavior of future products. CAM is an acronym for Computer-Aided Manufacturing, e.g. CAM relates to software solutions for defining manufacturing processes and operations. In such computer-aided design systems, the graphical user interface plays an important role regarding the efficiency of the technology. These technologies can be embedded in a Product Lifecycle Management (PLM) system. PLM refers to a business strategy that enables companies to share product data, apply common processes, and connect with the extended enterprise in order to gain a product-related enterprise knowledge. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA and DELMIA) provide an Engineering hub that organizes product engineering knowledge, a Manufacturing hub that manages manufacturing engineering knowledge and an Enterprise hub that enables enterprise integration and connections to both the Engineering and Manufacturing hubs. The overall system provides an open object model that links products, processes, and resources to enable a dynamic, knowledge-based product creation and decision-making support that drives optimized product definition, manufacturing preparation, production and service.

[0003] Within this and other contexts, segmenting 3D modeled objects (which can also be simply referred to as “3D segmentation” or “segmentation”) is becoming increasingly important.

[0004] Given a 3D modeled object (e.g. a mesh), 3D segmentation generally consists in splitting / separating the geometric elements (e.g. faces, and possibly other elements such as vertices or edges) of the 3D modeled object into several connected clusters (called segments) where each segment maximizes its internal consistency while minimizing its similarity with respect to other segments. The result of the segmentation process is to separate the faces into several sets (segments) by making each segment represent a semantically consistent set and have a clear boundary with respect to other segments.

[0005] Segmentation can be used in the shape abstraction process, where 3D geometrical shapes and / or features thereof are retrieved from captured 3D data, as discussed in e.g. the reference Kaiser A. et al. A survey of Simple Geometric Primitives Detection Methods for Captured 3D data (Computer Graphics Forum, 2018). Segmentation can also be used in a process of constructing a CSG (Constructive Solid Geometry) model (e.g. a CSG tree) from 3D raw data, as discussed in e.g. the references Wu. Q. et al. Constructing 3D CSG Models from 3D Raw Point Clouds (Computer Graphics Forum, 2018) and Shapiro V. et al. Separation for Boundary to CSG Conversion (ACM Transactions on Graphics, Vol. 12, No. 1, January 1993, pages 35-55). Both references in particular use a RANSAC (Random Sample Consensus) based method to detect an initial feature pool from which a feature tree is constructed.

[0006] However, there is still a need for improved solutions for 3D segmentation. SUMMARY

[0007] Hence, a computer-implemented method of machine learning is provided. The method comprises providing a dataset of training samples. Each training sample comprises a pair of parts of a 3D modeled object, which pair is labeled with a respective value. The respective value indicates whether the two parts belong to the same segment of the 3D modeled object. The method further comprises learning a neural network based on the dataset. The neural network is configured for taking as input two parts of a 3D modeled object representing a mechanical component, and for outputting a respective value. The respective value indicates a degree to which the two parts belong to the same segment of the 3D modeled object. The neural network is thereby usable for 3D segmentation.

[0008] The method can comprise one or more of the following:

[0009] The neural network comprises:

[0010] a coding neural network configured for taking as input a part of a 3D modeled object, and for encoding the part of the 3D modeled object into a latent vector; and

[0011] The predictive neural network is configured to take a pair of latent vectors encoded by the neural network as input and to output a corresponding value indicating the degree to which the two parts encoded by the latent vectors belong to the same segment of the 3D modeled object.

[0012] - The encoding neural network is configured to take a portion of the 3D modeled object and an adjacent portion of the input portion as input;

[0013] Learning a neural network involves minimizing a loss that includes a term for pairings in the dataset that penalizes the difference between the corresponding value indicating whether the two parts of the pair belong to the same fragment of a 3D modeled object and the corresponding value output by the neural network for the two parts.

[0014] - The neural network converts each of the two parts of the input into a corresponding intermediate feature, and the loss also includes another term that, for the pairings in the dataset, penalizes the difference between the corresponding value indicating whether the two parts of the pair belong to the same fragment of the 3D modeling object and the proximity between the intermediate features corresponding to the two parts.

[0015] -Items are of the following types:

[0016]

[0017] in, It is a pairing of datasets. It is a neural network for pairing The corresponding output value, It's a neural network, p i It is the indicator part and Whether it belongs to the corresponding value of the same fragment of the 3D modeled object, and d CE It is punishment With p i The loss of the difference between them

[0018] -And the other item is of the following type:

[0019]

[0020] in, and It is with pairing Part and The corresponding intermediate features, and d M It is punishment in p i and and The loss of the difference in proximity between them;

[0021] - providing the dataset comprises:

[0022]

[0023]

[0024] - generating, for one or more segmented 3D modeled objects in the initial dataset, the pairings of portions comprises generating pairings of portions at least one of which does not entirely belong to a single segment; and / or

[0025] - providing the dataset further comprises creating mini-batches for which the learning is performed in batches.

[0026] There is also provided a neural network learnable according to the method, for example a neural network that has been learned according to the method. The neural network forms a computer-implemented data structure having multiple layers of neurons having weights (also called parameters) whose values can be set by learning according to the method (for example have been set according to learning of the method).

[0027] There is also provided a computer-implemented method of using a neural network for 3D segmentation. The method of using comprises providing a 3D modeled object. The method of using further comprises applying the neural network to pairings of portions of the 3D modeled object, thereby outputting a set of values. Each value is for a respective pairing of portions of the 3D modeled object. The value is indicative of a degree to which the two portions belong to a same segment of the 3D modeled object. The method of using further comprises performing a segmentation of the 3D modeled object based on a set of one or more similarity indicators between adjacent portions of the 3D modeled object. The set of similarity indicators comprises the set of values.

[0028] Performing the segmentation comprises:

[0029] - performing a network-based segmentation of the 3D modeled object based on the set of values, and then performing another segmentation of the 3D modeled object based on at least one other similarity indicator; or performing said another segmentation, and then performing the network-based segmentation; or

[0030] - performing the segmentation of the 3D modeled object based only on the set of values; or

[0031] - performing the segmentation of the 3D modeled object based on a combination of the set of values and at least one other similarity indicator.

[0032] ​​A computer program is also provided, comprising instructions for performing and / or using the method.

[0033] A device is also provided, comprising a computer-readable data storage medium having recorded thereon a computer program and / or neural network.

[0034] The device can form or function as a non-transitory computer readable medium, e.g., on a SaaS (Software as a Service) or other server, or cloud-based platform, etc. The device can alternatively comprise a processor coupled to a data storage medium. The device can thus form, in whole or in part, a computer system (e.g., the device is a subsystem of a whole system). The system can further comprise a graphical user interface coupled to the processor. BRIEF DESCRIPTION OF DRAWINGS

[0035] Non-limiting examples will now be described with reference to the drawings, in which:

[0036] - Figures 1 to 7 A method is illustrated;

[0037] - Figure 8 Examples of a system are illustrated. DETAILED DESCRIPTION

[0038] A computer-implemented method of machine learning is provided. The method comprises providing a dataset of training samples. Each training sample comprises a pair of parts of a 3D modeled object that is labeled with a respective value (i.e., each pair is labeled with a respective value). The respective value indicates whether the two parts belong to the same segment of the 3D modeled object. The method further comprises learning a neural network based on the dataset. The neural network is configured for taking as input two parts of a 3D modeled object representing a mechanical component, and for outputting a respective value. The respective value indicates a degree to which the two parts belong to the same segment of the 3D modeled object. Thereby, the neural network can be used for 3D segmentation. The method can be referred to as a “learning method”.

[0039] The learning method constitutes an improved solution for 3D segmentation.

[0040] Note that the learning method learns a neural network that can be used for 3D segmentation. In other words, the neural network constitutes a tool for 3D segmentation, i.e., the tool can be used in a method of 3D segmentation.

[0041] The neural network is configured and learned to take as input any pair of parts of a 3D modeled object representing a mechanical component, and outputs a respective value indicative of the extent to which the two parts belong to the same segment of the 3D modeled object. In other words, the output of the neural network is indicative of the extent to which the two parts can be classified as belonging to the same segment of the 3D modeled object, e.g. the probability or confidence score that the two parts belong to the same segment. The term "segment" is to be understood as a piecewise segment, i.e. a geometrically coherent surface part of the 3D modeled object. Thus, the output of the neural network can be directly used in a segmentation algorithm or method that uses the information about the extent to which two neighboring parts of a 3D modeled object belong to the same segment for (e.g. all) pairs of neighboring parts of the 3D modeled object to be segmented.

[0042] In particular, many segmentation algorithms or methods perform a segmentation of a 3D modeled object based on a set of one or more similarity indicators between neighboring parts of the 3D modeled object, each similarity indicator comprising a set of values each indicative of the extent to which two neighboring parts of the 3D modeled object belong to the same segment, i.e. the set of values is a set of shape similarity values between neighboring parts of the 3D modeled object. The neural network can be applied to (e.g. all) pairs of neighboring parts of the 3D modeled object, the application resulting in the neural network outputting respective values (each value for one respective pair), each respective value being indicative of the extent to which the two parts of the pair belong to the same segment of the 3D modeled object. These respective values collectively form a set of values which is the similarity indicator on which the segmentation algorithm or method can be based. Thus, the output of the neural network can be directly used in the segmentation algorithm, i.e. without pre-processing (e.g. pre-segmentation). Moreover, the output of the neural network already provides a stopping criterion for such an algorithm: to stop the algorithm, it is sufficient to apply e.g. the criterion that only merge / assemble two parts in a segment if their respective values (output by the neural network) are greater than a tolerance threshold, such that the segmentation stops when only pairs of parts of the segmented 3D modeled object whose respective values are below the threshold remain.

[0043] Furthermore, using the set of values output by the neural network in a segmentation algorithm allows for a relatively fast segmentation. Indeed, the similarity indicators are obtained by only applying the neural network to pairs of parts of the 3D modeled object, which can be done relatively quickly once the neural network has been learned. In other words, by only applying the neural network, the similarity indicators can be obtained relatively quickly. Thus, once the neural network has been learned, complex computations for obtaining the similarity indicators are avoided during segmentation.

[0044] Furthermore, the learning approach provides robustness to the neural network for the following reasons. First, the learning approach relies only on the corresponding values of the pairs of the training dataset for learning, and does not rely, for example, on explicit definitions (e.g., parameterizations and / or primitive equations) of the parts in the pairs. Thereby, the neural network does not require explicit definitions of its inputs to work efficiently. Second, the neural network outputs the corresponding values of the pairs of the parts based only on the parts (in the example, based on their local context), without knowing the entire 3D modeled object to which the parts belong. This improves the generalization capability of the neural network by allowing it to generalize to unseen shapes, since this does not depend on global class-specific features. Third, the robustness of the neural network depends on the training dataset (e.g., its completeness and / or variability), such that the neural network can be used to accurately segment any type / category of 3D modeled objects, provided that the training dataset is properly selected for this purpose.

[0045] As previously discussed, the output of the neural network can be directly used in a 3D segmentation method, i.e., a method for segmenting a 3D modeled object / performing a 3D segmentation of a 3D modeled object. As known, given a 3D modeled object (e.g., a mesh), a 3D segmentation typically consists in splitting / separating the geometric elements (or faces thereof) of the 3D modeled object into a number of connected clusters (called segments), where each segment maximizes its internal consistency while minimizing its similarity with respect to other segments, the clusters / segments being the result of the segmentation and possibly referred to as “segmentation”.

[0046] The output of the neural network can be used in any method of 3D segmentation implementing this general segmentation framework. In particular, the output of the neural network can be used in any method of 3D segmentation that takes as input a 3D modeled object and outputs a set of segments of the 3D modeled object based on one or more similarity indicators.

[0047] The set of segments can be referred to as the “segmentation output / produced by the method of 3D segmentation” (i.e., the segmentation of the 3D modeled object), or simply “segmentation”. Each segment identified / output by the method of 3D segmentation is a part of the 3D modeled object that constitutes a surface forming a coherent part of the shape of the 3D modeled object. In other words, a segment can be defined as a geometrically coherent part of the shape of the 3D modeled object. Each segment identified by the method of 3D segmentation tends to maximize its internal consistency while minimizing its similarity with respect to other segments, and has a clear boundary with respect to other segments. For example, each segment can form a fundamental surface of the 3D modeled object, e.g., a fundamental surface that can be represented (e.g., fitted and / or parameterized) by a single CAD feature (e.g., a CATIA feature).

[0048] From the perspective of manufacturing of the mechanical part, each segment can form a contiguous surface. In other words, in the real world, the mechanical part can be divided into multiple portions, each having a respective geometry that is required or accommodates a respective manufacturing process (e.g., molding, additive manufacturing, or machining), each portion being represented by one or more segments in the segmentation output by the 3D segmentation method. In yet other words, the collection of segments output by the 3D segmentation method can contain, in one or more subsets of segments each, where for each subset, the one or more segments of the subset collectively represent, from the perspective of manufacturing, a surface shape of a portion of the contiguous mechanical part (i.e., the layout of material), i.e., a surface shape that is required or accommodates a particular manufacturing process (e.g., possibly chosen among several suitable manufacturing processes). For example, one or more such subsets can each represent a respective portion of the mechanical part to be manufactured by machining, and each have a shape (i.e., geometry) that is suitable as a path for a machining tool. Additionally or alternatively, one or more other subsets can each represent a respective portion of the mechanical part to be manufactured by molding, and each have a shape (i.e., geometry) that corresponds to a shape (i.e., geometry) of a contiguous portion of a mold. The mold itself can be manufactured by machining, and each such portion can exhibit a shape (i.e., geometry) that is suitable as a path for a machining tool. Additionally or alternatively, one or more other subsets can each represent a respective portion of the mechanical part to be manufactured by additive manufacturing, and each have a shape (i.e., geometry) that corresponds to a shape (i.e., geometry) of a final outer layer formed by the additive manufacturing process. Thus, the segmentation method using the neural network output allows to identify portions of the mechanical part that are contiguous in terms of manufacturing.

[0049] Any method of 3D segmentation herein can be any 3D segmentation that segments a 3D modeled object (i.e., outputs a set of segments for an input 3D modeled object) based on one or more similarity indicators. Such a method can comprise one or more segmentation steps. Each segmentation step performs at least part of the segmentation of the 3D modeled object based on one or more similarity indicators. Each segmentation step can be referred to as a segmentation per se, as it outputs segments. A segmentation step can output segments for only a part of the 3D modeled object, e.g., the segmentation step only performs a partial segmentation of the 3D modeled object, or the segmentation step can output initial or intermediate segments for the entire 3D modeled object, which can require at least one other segmentation step, which can also be referred to as “partial” (i.e., this segmentation step is an intermediate state / result). This partial segmentation can complete one or more other preceding partial segmentations, each output by another respective segmentation step, and / or can be completed by one or more other subsequent partial segmentations, each output by another respective segmentation step. Each segmentation step can be based on a single similarity indicator, i.e., the segmentation step only identifies segments based on a single similarity indicator. Alternatively, a segmentation step can be based on several similarity indicators, and in this case, the segmentation step identifies segments based on these similarity indicators. A method of 3D segmentation can only comprise one single segmentation step, thereby completely segmenting the 3D modeled object. Optionally, a method of 3D segmentation can comprise any filtering of segments, as known in the art of 3D segmentation. A method of 3D segmentation or any segmentation step thereof can use any 3D segmentation algorithm. Such an algorithm can be based on an iterative exploration and merging of pairs of parts of the 3D modeled object based on one or more similarity indicators. The iterative exploration and merging comprises updating the similarity indicators at each merging of parts, which comprises reapplying a neural network to recompute / update the respective values after merging in case the output of the neural network is used as a similarity indicator. Such an algorithm can be based on the 3D modeled object itself, or on any data structure representing the 3D modeled object, e.g., a dual graph of a mesh or a nearest neighbor graph of a point cloud. The algorithm can be any suitable algorithm, and can for example comprise a hierarchical clustering or a region growing algorithm.

[0050] Any similarity indicator herein can be a set of values each indicating a degree (e.g., a probability, a confidence score, a quantification, a measure) to which two adjacent parts (also referred to as “surfaces”) of a 3D modeled object to be segmented belong to the same segment. The set of values output by the neural network constitutes such a similarity indicator, as previously discussed. Any similarity indicator herein can alternatively be a set of distance values (i.e., values of distances) each being between two adjacent parts and each quantifying a shape similarity between the adjacent parts. Any similarity indicator herein may, for example, be a set of values of a centroid curvature distance, which penalizes the average curvature difference between surfaces (i.e., favors / encourages merging of surfaces having a low varying curvature at their boundaries), the distance being given, for example, by:

[0051]

[0052] where N1and N2denote adjacent surfaces, and denotes the average minimum or maximum curvature of N j . Any similarity indicator herein can alternatively be a set of values of a boundary curvature smoothness distance, which rewards the curvature smoothness around the interface (also referred to as “boundary”) between surfaces (i.e., favors / encourages merging of surfaces having a locally smooth curvature around their common boundary), the distance being given, for example, by:

[0053]

[0054] where N1and N2denote adjacent surfaces, (f1, f2) denote two adjacent surface parts belonging to N1and N2, respectively, and is the minimum or maximum curvature of f j , denotes clamping the value to the segment [k0, k1], and k0and k1are tuning values. Any similarity indicator herein can alternatively be a set of values of a centroid normal distance, which penalizes the average normal direction difference between surfaces (i.e., favors / encourages merging of surfaces having similar normal directions), the distance being given, for example, by:

[0055]

[0056] where N1and N2denote adjacent surfaces, and denotes the average normal of N j . Any similarity indicator herein can alternatively be a set of values of a distance, which penalizes the average curvature difference between surfaces and / or penalizes irregularities of merged surfaces. The distance can be of the following types:

[0057]

[0058] Where N1 and N2 represent adjacent surfaces, N represents j The average maximum curvature, and where:

[0059]

[0060] Where P N Let A represent the perimeter of N. N Represents the area of ​​N. Indicated to clamping, It means ∈{N1,N2} and is not N i The nodes are defined by α, and α represents a hyperparameter that defines the reduction effect of small surfaces.

[0061] As an application of neural networks for 3D segmentation, a method for using neural networks for 3D segmentation is also provided. This method includes 3D segmentation based on one or more similarity indicators, one of which is a set of values ​​output by the neural network. This method can be referred to as a "segmentation method".

[0062] The segmentation method includes providing a 3D modeling object representing a mechanical part (i.e., to be segmented). Providing the 3D modeling object may include, for example, measuring or acquiring the 3D modeling object by providing physical sensors and operating these physical sensors on the mechanical part (e.g., this could involve scanning the mechanical part), and then performing a 3D reconstruction process to obtain the 3D modeling object. Alternatively, providing the 3D modeling object may include, for example, creating the 3D modeling object by sketching it. In yet another alternative, providing the 3D modeling object may include retrieving the 3D modeling object from a (e.g., remote) database where 3D modeling objects have already been stored for further use in its creation or acquisition.

[0063] The segmentation method then involves applying a neural network to pair parts of a 3D modeled object. This outputs a set of corresponding values, each corresponding to a pair of parts (i.e., adjacent parts) of the 3D modeled object, and each corresponding value indicates the degree to which the two parts belong to the same segment of the 3D modeled object.

[0064] The segmentation method then involves performing segmentation of the 3D modeled object based on a set of one or more similarity indicators between adjacent parts of the 3D modeled object; that is, a method for applying / performing 3D segmentation to the 3D modeled object. The set of similarity indicators comprises a set of values, i.e., the output of a neural network is used when applying pairings of adjacent parts of the 3D modeled object.

[0065] Performing the segmentation can comprise performing a segmentation of the 3D modeled object based only on the set of values. In other words, the performed segmentation comprises only one segmentation step based on the set of values output by the neural network. The segmentation step can use any segmentation algorithm, e.g. a hierarchical clustering algorithm or a region growing algorithm.

[0066] Alternatively, performing the segmentation can comprise performing a network-based segmentation of the 3D modeled object (i.e. a segmentation based on the use of a neural network) based on the set of values, and then performing another segmentation of the 3D modeled object based on at least one other similarity indicator. In other words, the performed segmentation comprises two segmentation steps, the network-based segmentation and the other segmentation. The network-based segmentation is based on the set of values output by the neural network, and can use any segmentation algorithm, e.g. a hierarchical clustering algorithm or a region growing algorithm. The other segmentation is based on one or more other similarity indicators (i.e. different from the set of values output by the neural network), and can also use any segmentation algorithm, e.g. a hierarchical clustering algorithm or a region growing algorithm. The other segmentation can use the same segmentation algorithm as the one used for the network-based segmentation.

[0067] The other segmentation can be any example of hierarchical segmentation disclosed in European patent application EP21305195.6, filed on February 16, 2021. This hierarchical segmentation comprises a first segmentation, i.e. which forms one segmentation step of the hierarchical segmentation. The first segmentation comprises identifying first patches in the surface of the 3D modeled object, each of the first patches corresponding to a simple geometric surface of the 3D modeled object. A simple geometric surface is a primitive exhibiting at least one slidable motion. Then, the hierarchical segmentation comprises a second segmentation, i.e. which forms a subsequent segmentation step of the hierarchical segmentation. The second segmentation comprises identifying second patches in the un-identified surface of the 3D modeled object, each of the second patches corresponding to a freeform surface of the 3D modeled object. As described in European patent application EP21305195.6, the first segmentation can comprise three segmentation steps (referred to as “perform exploration and merging” in this European patent application), each step being based on a respective first distance, the value of which forms a similarity indicator. Also as described in European patent application EP21305195.6, the second segmentation can form a single segmentation step based on one second distance. The content of European patent application EP21305195.6, in particular the description of the hierarchical segmentation and all its features and examples, including the described implementation with hierarchical graph clustering, is incorporated herein by reference.

[0068] Alternatively, performing the segmentation can comprise performing the segmentation of the 3D modeled object based on a combination of the set of values and at least one other similarity indicator. In other words, the performed segmentation comprises a single segmentation step based on a combination of the set of values and at least one other similarity indicator (i.e., based on a similarity indicator formed by the combination of the set of values and one or more other similarity indicators). This similarity indicator may, for example, be a set of values each indicating a degree (e.g., a probability, a confidence score, a quantification, a measure) to which two neighboring parts (also referred to as “surfaces”) of the 3D modeled object belong to the same segment, each value being a combination of a respective value output by the neural network for the two parts and one or more values each of which is a similarity indicator and each of which indicates a degree to which the two parts belong to the same segment. For example, this value can be a sum of the respective value and one or more similarity values (e.g., each obtained from a respective distance) each quantifying a shape similarity between the two parts. Alternatively, this value can be a product of the respective value and one or more similarity values. Alternatively, this value can be an average of the respective value and one or more similarity values.

[0069] As known in the field of manufacturing CAD, the segmentation (e.g., the segmentation output by the segmentation method or any other segmentation method using the output of the neural network) can be used in various applications or processes that require or need the segmentation in some steps. The segmentation forms an improved representation of the outer surface of the 3D modeled object, which can be used in many applications. Moreover, the segments can be parameterized and made editable with CAD tools for further segmentation, which allows manipulating (e.g., editing) these segments, which can also be used in many applications. “Parameterization” refers to each segment can be accurately fitted with one 3D geometric object represented by a parametric equation or a parametric function, thus involving one or more parameters, which can take values each within a respective continuous range. In contrast to non-parameterized 3D geometric objects such as discrete representations (e.g., point clouds, meshes, dexel representations), 3D parameterized geometric objects allow easy manipulation and / or editability and / or efficient storage in memory. For example, the segments can be fitted with canonical primitives (e.g., planes, spheres, or cylinders) or can be parameterized with other applicable geometric tools (e.g., with parametric surfaces of free-form shapes such as NURBS or T-splines). In any application of the segmentation, including those discussed below, the 3D modeled object can be a measured 3D modeled object, thus the segmentation process of the 3D modeled object is performed on the measured 3D modeled object and allows for the final editing of the measured 3D modeled object (i.e., once segmented). Thereby, the segmentation can generally be used to segment a measured 3D modeled object, which is then processed into an editable data structure.

[0070] In a first application, the segmentation (e.g. the segmentation output by the segmentation method or any other segmentation method using the output of the neural network) is used for B-rep construction. B-rep construction using patches is discussed in the reference P. Benko et al., Algorithm for reverse engineering boundary representation models (Computer-Aided Design 33 (2001), 835-851), in A. Tumanin, Polygonal Mesh to B-Rep Solid Conversion: Algorithm Details and C++ Code Samples, published on the website habr.com on September 4, 2019, and in Beniere et al., Recovering Primitives in 3D CAD meshes (Proceedings of SPIE, 2011), all incorporated by reference herein. This first application includes using the segmentation of the 3D modeling object to convert the 3D modeling object into a boundary representation (i.e. a B-rep which is a collection of surface elements connected as boundaries, e.g. under the well-known STEP file format). The conversion can include fitting a surface to each patch and using the data about the patch to form the boundary of the surface (i.e. determining the topological data of the B-Rep, that is the “bounded by” relationships). Thus, the segmentation method or any other segmentation method using the output of the neural network can be included in a computer-implemented process for converting a 3D modeling object representing a mechanical part into a B-rep, the process comprising:

[0071] - providing a segmentation of the 3D modeling object by performing the segmentation; and

[0072] - converting the 3D modeling object into a B-rep by fitting a surface to each patch of the segmentation and forming the boundary of the surface based on the segmentation according to any known method suitable for the conversion.

[0073] In a second application, the segmentation (e.g. the segmentation output by the segmentation method or any other segmentation method using the output of the neural network) is used for feature tree construction. Fragment-based feature tree construction (e.g. by fitting primitives to the fragments) is discussed in the references T. Du et al., Inverse CSG: Automatic Conversion of 3D Models to CSG Trees, ACM SIGGRAPH ASIA 2018, Wu. Q. et al., Constructing 3D CSG Models from 3D Raw Point Clouds, Computer Graphics Forum, 2018, and Shapiro V. et al., Separation for Boundary to CSG Conversion, ACM Transactions on Graphics, Vol. 12, No. 1, January 1993, pages 35-55, all incorporated herein by reference. This second application includes the use of the segmentation to construct a feature tree representation of a 3D modeled object. This construction includes first using the segmentation and fitting primitives to each fragment to build a CAD feature pool (also called a “feature list”), from which a feature tree is then built. Thus, the segmentation method or any other segmentation method using the output of the neural network can be included in a computer-implemented process for constructing a feature tree from a 3D modeled object representing a mechanical component, the process comprising:

[0074] - providing a segmentation of the 3D modeled object by performing the segmentation;

[0075] - building a CAD feature pool (i.e. a feature list) from each fragment of the segmentation by fitting surfaces to each fragment of the segmentation, according to any known method;

[0076] and

[0077] - building a feature tree from the fitted surfaces, according to any known method.

[0078] In a third application, the segmentation (e.g. output by the segmentation method or any other segmentation method using the output of the neural network) is used for re-meshing (i.e. if the provided 3D modeling object is a 3D mesh) or resampling (i.e. if the provided 3D modeling object is a 3D point cloud). The segments of the segmentation can be parameterized by canonical primitives and / or NURBS, which allows to re-mesh (i.e. if the provided 3D modeling object is a 3D mesh) or resample (i.e. if the provided 3D modeling object is a 3D point cloud) the 3D modeling object by using the surface definitions of each segment. This re-meshing / resampling can be used for denoising the 3D modeling object (e.g. removing outliers, in particular for 3D point clouds, or smoothing the outer surface of the 3D modeling object, in particular for 3D meshes). Additionally or alternatively, this can be used for efficiently tessellating a 3D mesh, i.e. adapting the size of the faces of the mesh to the curvature of the corresponding surface, in order to minimize the number of faces, thereby optimizing the weight (i.e. storage aspect) of the mesh, while ensuring a maximum discrete distance to the exact surface. Thus, the segmentation method or any other segmentation method using the output of the neural network can be comprised in a computer-implemented process for re-meshing (or respectively resampling) a 3D modeling object representing a mechanical component as a 3D mesh (or respectively 3D point cloud), the process comprising:

[0079] - providing a segmentation of the 3D modeling object by performing the segmentation;

[0080] - parameterizing each segment with canonical primitives or NURBS, thereby generating surface definitions for all segments in the segmentation;

[0081] and

[0082] - using the surface definitions of each segment to re-mesh (or respectively resample) the 3D mesh (or respectively 3D point cloud) according to any known suitable method;

[0083]

[0084] - optionally one or more of:

[0085] o denoising the 3D mesh (or respectively 3D point cloud) using the re-meshing (or respectively resampling); and / or

[0086] o more efficiently tessellating the 3D mesh by adapting the size of the faces of the mesh to the curvature of the corresponding surface, in order to optimize the weight (i.e. storage aspect) of the mesh, while ensuring a maximum discrete distance to the exact surface.

[0087] ​In a fourth application, the segmentation (e.g., the segmentation output by the segmentation method or any other segmentation method using the output of the neural network) is used to detect anomalies (e.g., manufacturing anomalies) in the 3D modeled object. This application can include comparing available knowledge of properties that a mechanical part should have with the segments of the segmentation, thereby producing information about defects or anomalies of the mechanical part. For example, the available knowledge can include information such as “a cuboid portion of a mechanical part should have six large faces” or “a usual mechanical part should not have any facet portion”. If a 3D modeled object portion representing a cuboid is segmented into seven faces, or if some segments represent non-planar faces, this application can include inferring that the 3D modeled object feature is abnormal, so that the mechanical portion feature is abnormal (i.e., a manufacturing defect).

[0088] The segmentation (e.g., the segmentation output by the segmentation method or any other segmentation method using the output of the neural network) can be used for other applications of 3D shape segmentation, e.g., 3D morphing, 3D rendering (geometry / material property computation, occlusion culling, shadow determination), 3D animation, and / or shape compression. These applications are discussed in the previously cited reference Kaiser A. et al., A survey of Simple Geometric Primitives Detection Methods for Captured 3D data, Computer Graphics Forum, 2018, which is incorporated herein by reference.

[0089] Figures 1 to 6 Segmentations of mechanical parts are shown, each figure showing the respective segmented mechanical part.

[0090] The learning method is now further discussed.

[0091] As known per se in the field of machine learning, the processing of an input by a neural network comprises applying to the input an operation, which operation is defined by data comprising weight values. Learning a neural network thus comprises determining the values of the weights based on a data set configured for such learning, which data set can be referred to as learning data set or training data set. To this end, the data set comprises data pieces, each data piece forming a respective training sample. The training samples represent the diversity of situations for which the neural network is to be used after being learned. Any training data set herein can comprise more than 1000, 10000, 100000, or 1000000 training samples. In the context of the present disclosure, “learning a neural network based on a data set” means that this data set is the learning / training data set of the neural network, based on which the values of the weights (also referred to as “parameters”) are set.

[0092] In the context of the learning method, the training dataset is a dataset of training samples provided, each training sample comprising a pair of 3D modeled object parts, the pair of 3D modeled object parts being labeled with respective values indicating whether the two parts belong to the same segment of the 3D modeled object. Before discussing the provided dataset, the data structures involved therein are now discussed.

[0093] Each training sample comprises a pair of 3D modeled object parts, i.e. in the pair of parts, each part is a part of the 3D modeled object. A part of the 3D modeled object is a surface of the 3D modeled object, i.e. a portion of the outer surface of the 3D modeled object. A point of the 3D modeled object is also considered as a part in the present disclosure, i.e. the neural network is able to take as input parts that are points as well as parts that are geometric surfaces. In the context of iteratively merging segmentations of the surface / parts of the 3D modeled object, a part can be the result of a merge between other parts. Any part in any pair of the training dataset can also be the result of a merge between other parts, which makes the dataset particularly suitable for learning a robust neural network that can be used for 3D segmentation.

[0094] The learning method thus manipulates modeled objects. A modeled object is any object defined by data, for example stored in a database. By extension, the expression "modeled object" designates the data itself. Depending on the type of system, the modeled object can be defined by different kinds of data. The system can actually be any combination of a CAD system, a CAE system, a CAM system, a PDM system and / or a PLM system. In these different systems, the modeled object is defined by corresponding data. Thus, one can speak of CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, CAE data. However, these systems are not mutually exclusive from each other, since a modeled object can be defined by data corresponding to any combination of these systems. Thus, it is quite possible for a system to be both a CAD system and a PLM system, as will be apparent from the definitions of such systems provided below.

[0095] Any 3D modeled object herein can represent a product, for example, a part or an assembly of parts, or possibly an assembly of products. A "3D modeled object" refers to any object modeled by data allowing its 3D representation. A 3D representation allows viewing the part from all perspectives, for example, from various angles and / or distances. For example, when a 3D modeled object is in 3D representation, it can be processed and rotated around any of its axes or around any of the axes of the screen on which the representation is displayed. In particular, this does not include 2D icons that are not 3D modeled. A 3D modeled object can represent the geometry of a product, for example, a (e.g., mechanical) part or an assembly of parts (or an equivalent of an assembly of parts, as an assembly of parts can be seen as a part itself from a method point of view), after its virtual design is done with, for example, a CAD software solution or a CAD system, to be manufactured in the real world, or more generally any rigid body assembly (e.g., a mobile mechanism). CAD software solutions allow the design of products in different and unlimited industrial fields, including: aerospace, architecture, construction, consumer goods, high-tech equipment, industrial plants, transportation, marine, and / or offshore oil / gas production or transportation. Thus, a 3D modeled object can represent an industrial product, which can be any mechanical part, for example, a part of a land vehicle (including, for example, automotive and light truck equipment, racing car, motorcycle, truck and motor car equipment, truck and bus, train), a part of an aircraft (including, for example, fuselage equipment, aerospace equipment, propulsion equipment, defense product, aviation equipment, space equipment), a part of a naval vessel (including, for example, naval equipment, commercial ship, offshore equipment, yacht and work boat, marine equipment), a part of a general purpose machinery (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment product, metal structure product, tire manufacturing product), an electromechanical or electronic part (including, for example, consumer electronics, security and / or control and / or instrumentation product, computing and communication equipment, semiconductor, medical equipment and supplies), a consumer good (including, for example, furniture, home and garden product, leisure good, fashion product, hard goods retailer's product, soft goods retailer's product), a packaging (including, for example, food and beverage and tobacco, beauty and personal care, home product packaging).

[0096] Any 3D modeled object herein can form a discrete geometric representation of a 3D real-world object, e.g., representing an object from the real world, e.g., a mechanical part. A discrete geometric representation is a data structure herein comprising a discrete set of data pieces. Each piece of data represents a respective geometric entity positioned in 3D space. Each geometric entity represents a respective position of the 3D object (in other words, a respective portion of material constituting the entity represented by the 3D object). The aggregate (i.e., union or juxtaposition) of the geometric entities together represents the 3D object. Any discrete geometric representation herein may, in examples, comprise multiple such data pieces numbering more than 100, 1000, or 10000.

[0097] Any discrete geometric representation herein may, for example, be a 3D point cloud, each geometric entity being a point. Any discrete geometric representation herein may, alternatively, be a 3D mesh, each geometric entity being a mesh tile or face. Any 3D mesh herein can be regular or irregular (i.e., composed of or not composed of faces of the same type). Any 3D mesh herein can be a polygonal mesh, e.g., a triangular mesh. Any 3D mesh herein may, alternatively, be a B-Rep. Any 3D mesh herein can be obtained from a 3D point cloud, e.g., by triangulating the 3D point cloud (e.g., with Delaunay triangulation). Any 3D point cloud herein may, for example, be determined from physical measurements of a real object in a 3D reconstruction process. The 3D reconstruction process can comprise providing a real object; providing one or more physical sensors, each physical sensor being configured for acquiring a respective physical signal; and acquiring one or more respective physical signals by operating the one or more physical sensors on the real object (i.e., scanning the real object with each sensor). The 3D reconstruction can then automatically determine a 3D point cloud and / or a 3D mesh based on the measurements according to any known technique. The one or more sensors can comprise multiple (e.g., RGB and / or image or video) cameras, and the determination can comprise according to a motion analysis structure. The one or more sensors may, alternatively or additionally, comprise one or more depth sensors (e.g., on an RGB depth camera), and the determination can comprise 3D reconstruction from depth data. The one or more depth sensors may, for example, comprise a laser (e.g., lidar) or an ultrasound transmitter-receiver. Any discrete geometric representation herein may, alternatively, be any type of CAD model.

[0098] Any 3D point cloud or 3D mesh herein can alternatively be obtained from a 3D modeling object representing a skin (i.e. outer surface) of an entity (e.g. which corresponds to a B-Rep model representing a skin (i.e. exact surface)), e.g. by ray casting on the 3D modeling object or by surface subdivision on the 3D modeling object. Surface subdivision can be performed according to any 3D modeling object rendering process. Such rendering process can be coded on any CAD system in order to display a graphical representation of the 3D modeling object. The 3D modeling object can be designed by a user with the CAD system, or already designed by a user with the CAD system.

[0099] Providing the dataset can comprise forming the dataset. Forming the dataset can comprise providing an initial dataset of 3D modeling objects or an initial dataset of portions of 3D modeling objects, and labeling pairs of portions of 3D modeling objects with respective values. Alternatively, forming the dataset can comprise directly forming training samples, each training sample being a pair of portions of 3D modeling objects that have been labeled with their respective values (e.g. by randomly generating pairs of labeled portions from a CAD model).

[0100] In an example, providing the dataset comprises providing an initial dataset of segmented 3D modeling objects. This can comprise forming an initial dataset of segmented 3D modeling objects, e.g. by providing a dataset of unsegmented 3D modeling objects and segmenting them, or retrieving segmented 3D modeling objects from a (e.g. remote) database (e.g. by accessing the database and downloading the initial dataset). For example, providing the initial dataset can comprise selecting segmented 3D modeling objects within any dataset of CAD models, each dataset being equipped with ground truth data (e.g. parameterization), e.g. the ABC dataset. In other words, each segment can be equipped with ground truth data characteristics of the segment, e.g. a label indicating the segment.

[0101] Then, providing the dataset comprises, for each segmented 3D modeled object of the initial dataset, generating pairs of portions of the segmented 3D modeled object. For each pair, the portions of the pair either belong to the same segment of the segmented 3D modeled object or to different segments. In other words, the pairs are generated such that the two portions of a pair either belong to the same segment of the segmented 3D modeled object or to different segments. Generating the pairs can comprise, for each 3D modeled object, iteratively and randomly selecting portions of the 3D modeled object and forming pairs with these portions. This can comprise merging portions, i.e. some pairs can comprise one or more portions, each portion being resulting from a merging of other portions. Thus, the pairs of the dataset can comprise portions having varying sizes from one pair to another pair or even within one pair. This allows having pairs of portions in the training dataset corresponding to different levels of segmentation of the 3D modeled objects, which improves the robustness of the neural network after training: this helps ensuring that the neural network performs equally well at any level / phase of segmentation as the neural network processes pairs of portions having different sizes (corresponding to different levels of segmentation). Such pairs do occur in segmentation algorithms that iteratively merge portions of 3D modeled objects to form segments, resulting in adjacent portions that can have different sizes.

[0102] Generating such that, for one or more pairs (e.g. for most of the pairs, e.g. all pairs), each region of the pair belongs to a single respective segment of the segmented 3D modeled object, i.e. the region belongs entirely to a single respective segment. The two regions of a pair can both belong to the same respective segment or to different segments. Generating the pairs for one or more segmented objects of the initial dataset can comprise generating pairs of portions, at least one portion of the pairs of portions not belonging entirely to a single segment. In other words, for each respective one of the one or more segmented 3D modeled objects (e.g. only a portion thereof), generating such that the pairs of portions of the 3D modeled object comprise at least one portion that does not belong entirely to a single segment of the 3D modeled object (i.e. the portion comprises a first sub-portion belonging to a first segment and a second sub-portion belonging to a second segment, the first segment and the second segment being different segments). The portion that does not belong entirely to a single segment can be referred to as a “contaminated” portion. Training the neural network on pairs of portions for which there can be some degree of contamination allows it to learn how to adapt to potential errors that occur during segmentation (e.g. due to the neural network). In other words, if the neural network is fed pairs of portions for which at least one of the portions has a contamination, the neural network can still output satisfactory values for these portions (i.e. the neural network is trained to do the best it can in case of errors that can be encountered).

[0103] Each pair includes a respective value that indicates whether the two parts belong to the same segment of the 3D modeled object. In other words, generating the pairs can include, for each pair, labeling the pair with a respective value that indicates whether the two parts of the pair belong to the same segment. The labeling can be based on ground truth data. In other words, the ground truth data can be used to determine whether the two parts of a pair belong to the same segment. For example, the labeling can include labeling the pair using a label for each segment that the two parts belong to, e.g., using a label of “1” if the two parts belong to the same segment, and a label of “0” if the two parts do not belong to the same segment.

[0104] In any case, each training sample includes a pair of 3D modeled object parts that is labeled with a respective value that indicates whether the two parts belong to the same segment of the 3D modeled object. The value can be “1” if the two parts belong to the same segment, and “0” if the two parts do not belong to the same segment. The neural network is then learned on these training samples, which is now further discussed.

[0105] The neural network is configured to take as input two parts of a 3D modeled object that represents a mechanical component, and to output a respective value that indicates how much the two parts belong to the same segment of the 3D modeled object. The outputted respective value is of the same type as the respective values of the pairs of the training dataset. The value can be a confidence score or a probability (i.e., a number that belongs to [0, 1]), the higher the value, the higher the confidence or probability that the two parts belong to the same segment. The neural network can have any architecture that is suitable to take as input two parts and output a respective value.

[0106] Note that the neural network can comprise an encoding neural network configured for taking a part of the 3D modeled object as input and for encoding the part of the 3D modeled object into a latent vector. The latent vector is an intermediate feature that encodes the input part. The encoding can be meaningful in terms of segments, because the encoding neural network is learned this way, i.e. the encoding neural network can encode two parts belonging to the same segment relatively similarly. The encoding neural network can be referred to as an “encoder”. In addition to the part, the encoding neural network can take as input an index of a point of the part (e.g. an index of a point cloud point if the 3D modeled object is a point cloud, or an index of a mesh vertex if the 3D modeled object is a mesh). The encoding neural network can also be configured for taking as input a neighboring part of the 3D modeled object and the input part. In other words, in addition to the part, the encoding neural network can take as input a neighboring part of the input part. For example, the encoding neural network can have a per-point neighborhood pooling architecture that produces a neighborhood-aware latent vector / intermediate feature, i.e. the latent vector also encodes data on the input part neighborhood, which further improves the encoding to be meaningful in terms of segments. The per-point neighborhood pooling architecture can be any such architecture, e.g. if the 3D modeled object is a 3D point cloud, the architecture can be PointNet++ (discussed in C. Qi, L. Yi, H. Su, L. Guibas, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, NIPS, 2017, which is incorporated by reference herein), or DGCNN (discussed in Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. Bronstein, J. Solomon, Dynamic Graph CNN for Learning on Point Clouds, ACM, 2019, which is incorporated by reference herein), or if the 3D modeled object is a 3D mesh, the architecture can be MeshCNN (discussed in R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, D. Cohen-Or, MeshCNN: A Network with an Edge, ACM, 2019, which is incorporated by reference herein).The architecture can be combined with a Point-Net-like architecture (discussed in C. Qi, H. Su, K. Mo, L. Guibas. PointNet, Deep Learning on Point Sets for 3D Classification and Segmentation (CVPR, 2017), which is incorporated by reference herein) that produces a single latent vector representing the input parts.

[0107] Additionally, the neural network can further include a prediction neural network configured to take as input pairs of latent vectors output by the encoding neural network and to output respective values indicative of how well the two parts encoded by the latent vectors belong to the same segment of the 3D modeled object. The respective values are of the same type (e.g., confidence scores or probabilities, as previously discussed) as the respective values of the training samples.

[0108] Learning of the neural network can include minimizing a loss. The loss can be any loss suitable for training the neural network to output respective values indicative of how well the parts in pairs of input parts belong to the same segment. The loss can include a term that penalizes, for pairs of the dataset (e.g., for pairs of a mini-batch), a difference between the respective value indicating whether the two parts of the pair belong to the same segment of the 3D modeled object and the respective value output by the neural network for the two parts. In other words, the term can be a function of the respective value output by the neural network for the two parts of the pair and the respective value that labels the pair to indicate whether the two parts belong to the same segment, and the term can tend to have a high value if the respective value indicates that the two parts belong to (or respectively do not belong to) the same segment, while the respective value output by the neural network tends to indicate that the two parts do not belong to (or respectively belong to) the same segment.

[0109] A neural network can transform each of the two parts of the input into a corresponding intermediate feature. For each part, each corresponding intermediate feature encodes that part, and the neural network can output corresponding values ​​for both parts based on the corresponding encodings (i.e., intermediate features) of the parts. For example, in the case of the previously discussed neural network including an encoding neural network and a predictive neural network, for a corresponding part, the intermediate feature is the latent vector encoding that part. In any case, the loss may also include another term that, for the pairings in the dataset (i.e., the same pairings involved in the previously discussed loss term), penalizes the difference between the corresponding value indicating whether the two parts of the pair belong to the same fragment of a 3D modeled object and the proximity between the intermediate features corresponding to the two parts. Specifically, proximity is any function that quantifies whether intermediate features (e.g., latent vectors) are close (e.g., proximity is based on such distance, for example, in terms of the distance between latent vectors), and this other term tends to have a high value when proximity tends to indicate that the two parts have similar (or different) intermediate features respectively, while the corresponding value indicates that the two parts do not belong to (or belong to) the same fragment. Thus, this other aspect helps train neural networks (e.g., encoding neural networks) to encode input parts as intermediate features (e.g., latent vectors) that are already meaningful in terms of the segments; that is, to encode intermediate features (e.g., latent vectors) that are close if two input parts belong to the same segment (e.g., in terms of the distance between latent vectors), and not close otherwise. This further improves the robustness of the neural network.

[0110] This item could be of the following type:

[0111]

[0112] in, It is a pairing of datasets. It is a neural network for pairing The corresponding output value, It's a neural network, p i It is the indicator part and Whether it belongs to the corresponding value of the same fragment of the 3D modeled object, and d CE It is punishment With p i The loss of the difference between them, and another item is of the following type:

[0113]

[0114] in, and It corresponds to pairing Part and intermediate features, and d M is a loss penalizing the proximity between p i and and the difference between the proximity and

[0115] In the case where the training is performed in mini-batches, can be a pair forming a mini-batch, N is the size of the mini-batch. In the case where the neural network comprises an encoding neural network and a prediction neural network, f w is the encoding neural network, and g w is the prediction neural network, and are latent vectors encoding parts and respectively.

[0116] d CE can be a loss penalizing the neural network output values with respect to the respective values of the labeled input pairs, e.g., binary cross-entropy loss. d CE helps the neural network to predict / output the respective values according to the ground truth parts belong to knowledge. For example, d CE may be of the following types:

[0117]

[0118] If the parts of the input do not belong to the same segment, d M is a loss penalizing the proximity of the intermediate features, or boosting the feature proximity if the parts of the input belong to the same segment. In other words, d M is a metric learning loss of the intermediate features (e.g., latent vectors) corresponding to the parts of the input. For example, d M may be of the following types:

[0119]

[0120] where a is a hinge loss margin parameter, which represents the expected minimum distance between the parts not belonging to the same segment. In this case, the loss is designed such that the encoding neural network learns features such that the Euclidean distance on these features distinguishes the parts not belonging to the same segment with a margin large enough. d M helps the encoding network to learn meaningful distinguishing features.

[0121] The loss can be given by the following formula:

[0122]

[0123] where l is a trade-off parameter. One can choose l such that d CE and ldM have the same order of magnitude, so that both are trained with the same importance. Lambda can for example be equal to 0.6.

[0124] The loss can include other terms, for example one or more losses with respective trade-off parameters, which are designed for the neural network to predict one or more meaningful quantities (e.g. normals and / or curvatures). This trains the neural network to encode features that include explicit meaningful information, in order to compare the parts.

[0125] The learning of the neural network can be performed in mini-batches. The learning can use any mini-batch stochastic optimization algorithm (e.g. stochastic gradient descent or Adam optimization algorithm). In this case, providing the dataset also includes creating the pairs of mini-batches. In other words, the mini-batches for the mini-batch-wise learning are created beforehand, i.e. before the learning. Thus, the mini-batch-wise learning is based only on mini-batches that fully reflect the ground truth of the 3D modeled objects involved in the segmentation of the training dataset. Creating the mini-batches can include selecting pairs of pairs to form the mini-batches. This can include selecting (e.g. for each mini-batch or for at least a portion thereof) pairs formed by parts having different sizes and / or pairs having different parts between them with different sizes and / or pairs including at least one impurity part. This provides the mini-batches with parts corresponding to different levels of clustering and, if there are pairs including at least one impurity part, accounts for errors occurring during the learning, as discussed previously.

[0126] An implementation of the learning method is now discussed.

[0127] The neural network learned by this implementation can be used to segment any 3D shape into meaningful parts, thus leveraging both the robustness to noise and the concept of meaningful parts directly given by the ground truth segmentation of the training dataset, using deep learning techniques. This implementation introduces an improved neural network that learns to predict whether two parts (also called “sub-parts”) of a 3D shape belong to the same segment (also called “part” or “relevant part”), thus allowing an iterative segmentation of the 3D shape.

[0128] Taking as input a dataset of 3D shapes labeled with their ground truth part decomposition, the implementation includes first providing a training dataset, which includes generating intermediate unsegmented clusters of different completion levels for these shapes. The implementation then includes learning a neural network that takes as input two sub-parts of a given shape and predicts the probability that these two sub-parts belong to the same part. To learn this neural network, a mini-batch training scheme is used as follows:

[0129] 1) generating a set of pairs of sub-parts from different shapes and cluster levels, the set constituting a mini-batch;

[0130] 2) Compute the network's prediction of the pairing of these sub-parts, and encode the intermediate features of each sub-part;

[0131] 3) Minimize the loss of both the prediction and the ground truth sub-parts' intermediate features' distance with respect to the matching neural network's weights, to optimize the probability of the prediction belonging to the same part.

[0132] The neural network can then be used in a segmentation algorithm (e.g. hierarchical clustering algorithm) as the distance between sub-parts (also called "similarity indicator"), eventually in conjunction with other terms. This, together with the threshold of the network's prediction, allows the segmentation of 3D shapes. This implementation allows the following operations:

[0133] - Learning the probability of two sub-parts belonging to the same part, in conjunction with a distance-based segmentation algorithm, allows the number of segmented parts to be not fixed, contrary to end-to-end segmentation learning methods.

[0134] - No explicit definition of the required parts (e.g. primitive equations) is needed.

[0135] - The network predicts the merging probability of sub-parts' pairing based only on the sub-parts and their local context, without needing to understand the whole object.

[0136] This simplifies the network's generalization to unseen shapes, as it does not rely on global class-specific features.

[0137] - The learned merging probability can easily be combined with any other related measure in a clustering algorithm (e.g. purity score, curvature distance).

[0138] - Predicting the merging probability allows directly leveraging its value as a confidence score, which can for example be used as a stopping criterion in a clustering algorithm.

[0139] - Training on consistently measured features increases the robustness of the network's prediction by enforcing meaningful features.

[0140] - No sub-part initialization is required.

[0141] - Desired part types to segment are easily controlled via the training dataset.

[0142] Mini-batch generation

[0143] The input to the implementation is an initial dataset of 3D shapes (point clouds or 3D meshes), and a label of the ground truth part each point of each 3D shape belongs to. Providing the training dataset comprises centering and scaling each shape into a unit sphere, i.e. so that its maximum distance to the center is 1. Providing the training dataset comprises discarding ambiguous points, i.e. points that lie on the intersection of at least two parts, and thus do not have a unique possible label.

[0144] Then, providing the training dataset comprises generating pairs of sub-parts that will constitute the input mini-batches of the neural network. In order to train the neural network to predict whether a pair of sub-parts belongs to the same meaningful part, then used as a clustering criterion in a clustering algorithm, the generation focuses on the clustering prediction of neighboring (w.r.t. connectivity in a mesh or maximum distance in a point cloud) sub-parts. Moreover, in order to make the neural network perform as well at any stage of the clustering algorithm, the generation generates neighboring sub-parts at different levels of clustering, specifically representing different sizes.

[0145] In order to do so, for each input shape, the generation generates a corresponding graph, where nodes initially correspond to points, and edges correspond to mesh edges for a 3D mesh, or maximum pairwise distances for a point cloud. In order to generate different levels of clustering, the generation then iteratively and randomly merges connected nodes with the same part label together. At any given point, a graph node then corresponds to a set of points belonging to a sub-part of a labeled part of the 3D shape, and an edge corresponds to two neighboring sub-parts. The generation then selects a random set of edges from the various levels of graph clustering as input to the neural network. This allows access to the ground truth clustering criterion for edges: 1 if both nodes represent sub-parts of the same labeled part, 0 if the corresponding parts are different. In order to train the neural network efficiently, the implementation can comprise sampling a given proportion of edges that correspond to clusters and edges that should not be clustered.

[0146] When applied to real-life data, in order to increase the robustness of the neural network prediction, the generation can also generate impurity sub-parts. When generating different levels of clustering of the graph corresponding to an input 3D shape, this also comprises merging nodes that do not represent the same part together with a given probability. At a certain level of impurity (measured as the number of points in a node that do not belong to the most represented part), the nodes are then considered to belong to the most represented part (w.r.t. the ground truth clustering criterion and the subsequent graph clustering). Training the neural network on sub-parts that can have a certain degree of impurity allows it to learn how to adapt to errors that eventually occur during the clustering of real-world 3D shapes.

[0147] Network architecture

[0148] The neural network is composed of a sub-part encoder and a prediction network.

[0149] Sub-part encoder fw As input, one takes the 3D shape (point cloud or mesh) and the indices of the points that belong to a subpart. One first uses a per-point neighborhood-pooling encoder architecture in order to produce neighborhood-aware features for each point in the subpart. This architecture can for example be based on PointNet++ for point clouds (discussed in the previously cited reference C. Qi, L. Yi, H. Su, L. Guibas, PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, NIPS, 2017) or DGCNN (discussed in the previously cited reference Y. Wang, Y. Sun, Z. Liu, S. Sarma, M. Bronstein, J. Solomon, Dynamic Graph CNN for Learning on Point Clouds, ACM, 2019), or MeshCNN for meshes (discussed in the previously cited reference R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, D. Cohen-Or, MeshCNN: A Network with an Edge, ACM, 2019). Once these per-point features are obtained, they are concatenated to the subpart coordinates and passed through a PointNet-like architecture (discussed in the previously cited reference C. Qi, H. Su, K. Mo, L. Guibas, PointNet, Deep Learning on Point Sets for 3D Classification and Segmentation, CVPR, 2017) that produces a single feature representing the subpart. The use of a neighborhood-pooling architecture improves training, as when clustering the mesh, the initial subparts will reduce to a single point, so their local neighborhood is necessary to compute meaningful features.

[0150] Once the input subparts X1 and X2 are both encoded into two feature vectors f w (X1) and f w (X2), these vectors will be the input to the prediction network gw 的 . Given the two feature vectors h1 and h2, this network predicts the clustering probability g w (h1,h2) e [0,1], i.e., these features represent the degree of certainty that the subparts belong to the same part. One first concatenates the two features, then applies a sequence of multi-layer perceptrons (MLP) to this concatenation, followed by a final sigmoid activation in order to limit the output to [0,1].

[0151] training

[0152] The network f is trained by minimizing the computational loss L(w) of small batches of N pairs of 3D shape subparts through small batch stochastic optimization algorithms (e.g., stochastic gradient descent or Adam optimization algorithm) w and g w Let be the corresponding input to the subpart encoder f w , i.e., the shape (point cloud with final accompanying topology in case of a mesh) and the indices of the shape point cloud relative to the points corresponding to the subpart, let p1,..., p N be the ground truth binary variable, which is 1 if the corresponding pair of subparts belongs to the same part and thus should be merged, and 0 otherwise. The loss is given by:

[0153]

[0154] d CE is a loss that penalizes the network merge prediction with respect to the ground truth binary merge value, e.g., binary cross-entropy loss. d CE helps the neural network to predict clustering probabilities according to the ground truth part membership knowledge. If the input subparts do not belong to the same segment, d M is a loss that penalizes feature proximity, or boosts feature proximity if the input subparts belong to the same segment. In other words, d M is a metric learning loss on the encoded features of the subparts. For example, d M may be of the following type:

[0155]

[0156] where a is a hinge loss margin parameter that represents the minimum distance between subparts that do not belong to the same part. In this example, d M allows the encoder to be trained to learn features such that the Euclidean distance on these features discriminates between subparts that do not belong to the same part with a margin that is large enough. d M helps the encoding network to learn meaningful distinguishing features. l is a trade-off parameter.

[0157] The neural network can also be configured to predict quantities such as normals or curvatures, e.g., by adding a corresponding loss and trade-off parameter to L(w). This trains the neural network to encode features that include explicit meaningful information for comparing subparts.

[0158] Figure 7A neural network architecture according to an implementation is shown, as well as quantities distinguished in the training loss (metrics and merge probabilities).

[0159] Use in segmentation algorithms

[0160] Once trained, the neural network predicts for any pair of sub-parts the probability that they belong to the same part. As discussed previously, this information can be used in a variety of distance-based segmentation algorithms in order to segment the input shape into meaningful parts. As discussed previously, such algorithms can be, for example, a hierarchical clustering or a region growing algorithm. For example, using a hierarchical clustering algorithm, one can set the distance between each pair of clusters to be inversely proportional to the network prediction when applied to that pair, meaning that pairs of clusters will be merged in decreasing order of the network prediction, and one can set a stopping criterion to be a threshold on the prediction.

[0161] The network can also incorporate other distances for such segmentation algorithms. For example, a compatible distance on normals and / or curvature can be combined with the merge probability prediction of the network to obtain another expression of the sub-part distance. Another neural network can also be trained to predict the purity of a sub-part of a shape, i.e. the likelihood that the sub-part covers a single meaningful part. Such a neural network can be used to predict the purity of a combination of two sub-parts, as well as to combine the merge predictions of the neural networks learned by the implementation.

[0162] These methods are computer-implemented. This means that the steps (or substantially all the steps) of the method are performed by at least one computer or any system similar. Thus, the steps of the method are performed by the computer possibly fully automatically or semi-automatically. In an example, the triggering of at least some of the steps of the method can be performed by user-computer interaction. The level of user-computer interaction required can depend on the level of automation foreseen and be balanced with the need to implement the user’s will. In an example, the level can be user-defined and / or pre-defined.

[0163] A typical example of computer-implemented of a method is to execute the method with a system adapted to this purpose. The system can comprise a processor coupled to a memory and a graphical user interface (GUI), the memory having recorded thereon a computer program comprising instructions for executing the method. The memory can also store a database. The memory is any hardware suitable for such storage, possibly comprising several physically distinct parts (for example, one part for the program and possibly one part for the database).

[0164] Figure 8 An example of a system is shown, wherein the system is a client computer system, for example, a workstation of a user.

[0165] The client computer of this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, a random access memory (RAM) 1070 also connected to the bus. The client computer is also provided with a graphics processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass memory device controller 1020 manages access to a mass memory device (e.g., a hard disk drive 1030). Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks 1040. Any of the foregoing can be supplemented by, or incorporated in, a specially- designed ASIC (application-specific integrated circuit). A network adapter 1050 manages access to a network 1060. The client computer can also include haptic devices 1090, e.g., a cursor control device, a keyboard, etc. A cursor control device is used in the client computer to allow the user to selectively position a cursor on the display 1080 at any desired location. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, the buttons of which are used to generate signals. Alternatively or additionally, the client computer system can include a sensitive mat and / or a sensitive screen.

[0166] A computer program can include instructions executable by a computer, which instructions include means for causing the system above to perform the method. The program can be recordable on any data storage medium including a storage device of the system. The program can be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program can be implemented as apparatus, for example, a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps can be performed by the programmable processor executing a program of instructions to perform functions of the method by operating on input data and generating output. The processor can be programmable and coupled for receiving data and instructions from, and sending data and instructions to, a data storage system, at least one input device, and at least one output device. The application program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired. In any case, the language can be a compiled or interpreted language. The program can be a complete installation program or an update program. In any case, application of the program on the system results in instructions for performing the method.

Claims

1. A computer-implemented method of machine learning, the method comprising: - providing a dataset of training samples, each training sample comprising a pair of parts of a 3D modeled object, the pair of parts of the 3D modeled object being labeled with a respective value indicative of whether the two parts belong to a same segment of the 3D modeled object; and - learning a neural network based on the dataset, the neural network being configured for taking as input two parts of a 3D modeled object representing a mechanical component and for outputting a respective value indicative of a degree to which the two parts belong to a same segment of the 3D modeled object, the neural network thereby being usable for 3D segmentation, wherein the learning of the neural network comprises minimizing a loss comprising a term penalizing, for pairs of the dataset, a difference between the respective value indicative of whether the two parts of a pair belong to a same segment of the 3D modeled object and a respective value output by the neural network for the two parts, wherein the neural network transforms each of the two parts of input into a corresponding intermediate feature, and the loss further comprises another term penalizing, for the pairs of the dataset, a difference between the respective value indicative of whether the two parts of a pair belong to a same segment of the 3D modeled object and a proximity between the intermediate features corresponding to the two parts, wherein the term is of the following type: wherein, is a pair of the dataset, is a respective value output by the neural network for a pair , is the neural network, is a respective value indicating whether a part and belongs to the same segment of the 3D modeled object, and is a loss penalizing the difference between and , the other term is of the following type: wherein, and are intermediate features corresponding to the parts of the pair and and is a loss penalizing the difference between the proximity of and and .

2. The method of claim 1, wherein, the neural network comprises: - an encoding neural network configured for taking as input a part of a 3D modeled object and for encoding the part of the 3D modeled object into a latent vector; and - a prediction neural network configured for taking as input pairs of latent vectors output by the encoding neural network and for outputting a respective value indicative of a degree to which the two parts encoded by the latent vectors belong to a same segment of the 3D modeled object.

3. The method of claim 2, wherein, the encoding neural network is configured for taking as input a part of a 3D modeled object and an adjacent part of an input part.

4. The method of any one of claims 1 to 3, wherein, providing the dataset comprises: - providing an initial dataset of segmented 3D modeled objects; and - for each segmented 3D modeled object in the initial dataset, generating pairs of parts of the segmented 3D modeled object, the parts in a pair belonging to a same segment or to different segments of the segmented 3D modeled object, each pair comprising a respective value indicative of whether the two parts belong to a same segment of the 3D modeled object.

5. The method of claim 4, wherein, generating the pairs comprises, for one or more segmented 3D modeled objects in the initial dataset, generating pairs of parts, at least one part of the pairs of parts not belonging entirely to a single segment.

6. The method of claim 4, wherein, providing the dataset further comprises creating mini-batches of pairs, the learning being performed mini-batch-wise.

7. A neural network learnable according to the method of any one of claims 1 to 6.

8. A computer-implemented method for 3D segmentation using the neural network of claim 7, the method comprising: - providing a 3D modeled object representing a mechanical component; - applying the neural network to pairs of portions of the 3D modeled object, thereby outputting a set of values, each value being for a respective pair of portions of the 3D modeled object, each value being indicative of a degree to which the two portions belong to a same segment of the 3D modeled object, and - performing a segmentation of the 3D modeled object based on a set of one or more similarity indicators between adjacent portions of the 3D modeled object, the set of similarity indicators comprising the set of values.

9. The method of claim 8, wherein, Performing the segmentation comprises: - performing a network-based segmentation of the 3D modeled object based on the set of values, then performing another segmentation of the 3D modeled object based on at least one other similarity indicator; or performing the other segmentation, then performing the network-based segmentation; or - performing the segmentation of the 3D modeled object based only on a set of values; or - performing the segmentation of the 3D modeled object based on a combination of the set of values and at least one other similarity indicator.

10. A computer program product comprising instructions for performing the method according to any one of claims 1 to 6 and / or the method according to any one of claims 8 to 9.

11. An apparatus comprising a computer-readable data storage medium having recorded thereon instructions for performing the method according to any one of claims 1 to 6 and / or the method according to any one of claims 8 to 9 and / or the neural network according to claim 7.

12. The apparatus according to claim 11, further comprising a processor coupled to the data storage medium.

Images