Data-driven mapping function for visual effects applications using mesh segmentation

Abstract

A volumetric based three-dimensional (3D) object is segmented. A training dataset of 3D training objects is acquired. A neural network is defined by multiple layers including a linear layer and a graph layer. The linear layer operates on 2D objects and provides inputs to the graph layer that performs convolution operations on vertices of the 3D objects. A model, that approximates a shape diameter function (SDF) determines neighborhood diameters including a distance from a vertex of the 3D objects to an antipodal vertex. The model is generated by iterating through the 3D objects using the neural network to converge on weights of SDF features. An input mesh is acquired and the converged weights are used to approximate SDF values. The SDF values are input to a graph cut algorithm that generates vertex clusters defining a segmented part of the input mesh. The segmented part is visually displayed or provided.

Description

Data-Driven Mapping Function for Visual Effects ApplicationsUsing Mesh SegmentationCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit under 35 U.S.C. Section 119(e) of the following co-pending and commonly-assigned U.S. provisional patent application(s), which is / are incorporated by reference herein:

[0002] Provisional Application Serial No. 63 / 481,075, filed on January 23, 2023, with inventor(s) Bruno Roy, entitled “Data-Driven Mapping Function for Visual Effects Applications Using Mesh Segmentation,” attorneys’ docket number 30566.0605USP1.BACKGROUND OF THE INVENTION1. Field of the Invention.

[0003] The present invention relates generally to three-dimensional (3D) image processing and segmenting a polygonal mesh, and in particular, to a method, apparatus, system, and article of manufacture for generating a mapping between a 3D surface and its underlying volume that is then used as input to efficiently segment a polygonal mesh.2. Description of the Related Art.

[0004] (Note: This application references a number of different publications asindicated throughout the specification by names enclosed in brackets, e.g., [x], A list of these different publications ordered according to these reference names can be found below in the section entitled “References.” Each of these publications is incorporated by reference herein.)

[0005] A fundamental topic in geometry processing and computer graphics is mesh segmentation. Mesh segmentation finds many applications in 3D shape analysis such as: (i) Texture Mapping; (ii) 3D Shape Modeling; (iii) 3D Shape Retrieval; (iv) MultiResolution and Mesh Compression; and (v) Animation.

[0006] Mesh segmentation is the process of decomposing a polygonal mesh into meaningful parts. These meaningful parts consist of subsets of vertices or faces that are semantically relevant and informative for further applications. For various reasons, a user may wish to segment a mesh into parts for further applications. For instance, when animating a character, the user may prefer to first identify semantic parts of the body before creating the underlying animation structure to move the vertices in a realistic manner. Although some methods offer ways to decompose a polygonal mesh, these are computationally expensive since they rely on the mesh complexity to carry out their task. These state-of-the-art methods can provide detailed segmentation of the mesh at the cost of many hours of computation on a high- resolution model used for production. Additionally, the user may need to adjust the resulting segmentation throughout the tedious linear process of performing the decomposition using different sets of parameters.

[0007] For example, decomposing mesh structures is usually performed by definingedge loops (i.e., pairs of vertices) acting as boundaries between these subsets of elements. Although performing this task on a closed manifold mesh seems inherently intuitive, generalizing is still challenging considering that multiple valid solutions exist for any mesh. Moreover, most of the state-of-the-art methods are rather time consuming.

[0008] The problem with prior art mesh segmentation methods is that they mostly rely on constraints - making them highly time consuming as they require iterative solvers to jointly optimize them. The constraint types are generally cardinality, geometry and topological, respectively guiding the number of segmented parts, biases towards specific primitive shapes, and sub-mesh connected components. Some of these methods also require the user to select initial seeds to reduce the number of iterations before satisfying the constraints, adding on the total time required to accomplish the mesh segmentation task.

[0009] In recent years, machine learning has revived the problem of segmentation on meshes by proposing much more generalizable approaches and by offering a better understanding of the intrinsic semantics of shapes. Although these approaches have considerably improved this challenging task, the time required to train and cover a wide spectrum of different topologies remains problematic for uses on dense meshes.

[0010] To better understand the problems of the prior art, a description of mesh segmentation and prior art research may be useful.

[0011] FIGs. 1 A and IB illustrate the processing of a mesh in accordance with the prior art. To process the mesh, the neighborhood at each point in the original mesh102 / 106 is approximated. The goal is to determine if each point is part of a different neighborhood in order to segment the mesh and generate / construct the end result 104 / 108. The prior art performs this approximation / segmentation by shooting rays to approximate the diameter at each point and determining if each point (based on the diameter) is part of the same neighborhood. The points are then divided into clusters to construct a segmented mesh as illustrated at 104 / 108. However, such a segmentation / processing is time intensive (e.g., the example may take -2-2.5 hours) as the original mesh 102 / 106 may comprise a complex model (e.g., a few million polygons).

[0012] There are ways to approximate the segmentation via data driven methods using machine learning where the goal is to make the original mesh 102 / 106 parameterizable and fast at predicting classes (referred to herein as “part segmentation” or “segmenting parts” using machine learning). Such part segmentation using machine learning may be satisfactory (e.g., parameterizable and fast) to classify different parts and regions of an object. However, using prior art methods, it is difficult to generalize a mesh and make the processing work for any type of topologies / geometries. For example, monsters and shapes that are out of this world and different from standard animals / shapes / geometries may be hard to predict for.

[0013] The description below provides a more detailed description of such prior art processing in order to facilitate a better understanding of the limitations of the prior art.

[0014] Over the last two decades, mesh segmentation has been used for variousapplications in computer graphics. This task has proven to benefit many applications in 3D shape analysis such as texture mapping [Sander et al. 2003], 3D shape modeling [Ji et al. 2006], 3D shape retrieval [Ferreira et al. 2010], multi-resolution and mesh compression [Maglo et al. 2011], and animation [Yuan et al. 2016], [Rodrigues et al. 2018] provides a survey on part-based mesh segmentation. Historically, the traditional problem of mesh segmentation has been approached in many ways.

[0015] Region growing. One of the most intuitive and simple approaches for segmentation is the region growing technique. The criterion that determines whether or not an element should be added to a cluster is what mainly differentiates the variations of the available region growing algorithms. Among other criteria: representative planes [Kalvin and Taylor 1996], curvature [Lavoue et al. 2005], and convexity [Chazelle et al. 1995; Sheffer 2 007] were used as conditioners for clustering. Another common variation of the region growing method uses multiple source seeds to initiate the growing process [Eck et al. 1995; Levy et al. 2002;Sorkine et al. 2002],

[0016] Clustering. Merge operations on clusters may be crucial when it comes to segmentation tasks [Attene et al. 2006; Garland et al. 2001; Gelfand and Guibas 2004; Sander et al. 2001], Although hierarchical clustering algorithms are similar to growing region algorithms, hierarchical clustering algorithms prioritize set operations between existing clusters in a structured way. Iterative clustering algorithms are stated as parametric as the number of clusters is given a-priori. As opposed to previous methods, iterative clustering methods are focused on converging towards optimalsegmentation given a number of clusters [Cohen- Steiner et al. 2004; Hart et al. 2000; Lloyd 1982; Shlafman et al. 2002; Wu and Kobbelt 2005],

[0017] Implicit methods. The implicit methods for mesh segmentation focus on boundaries and correspondences between subsets of elements of the object to segment. Again, the main difference between these algorithms is how they define the boundaries and underlying structures. To mention a few, the most common ways pass through curvature / contours [Lee et al. 2005; Levy et al. 2002; Mitani and Suzuki 2004], subdivision [Katz and Tai 2003; Podolak et al. 2006] (similar to hierarchical clustering), and underlying structures connecting the shape with intrinsic surface properties [Li et al. 2001; Lien et al. 2006; Raab et al. 2004], As a matter of fact, the Shape Diameter Function (SDF) algorithm [Shapira et al. 2008] (generally described above with respect to FIG. 1 A and IB) is at the intersection of using subdivision and an underlying structure. The intuiting behind the Shape Diameter Function is to produce a diameter measure throughout vertex neighborhoods of the mesh. The resulting measures (i.e., SDF values) relate to the medial axis transform and provide a volume correspondence of the shape at any given point on the surface. These per- vertex measures are then used as a threshold for a graph-cut algorithm.

[0018] Embodiments of the invention take advantage of the generalization power of neural networks to estimate per-vertex properties as input to a graph-cut algorithm - making the latter highly adaptive for mesh segmentation.

[0019] Data-driven. Neural networks have been widely used for the segmentation problem on images [Lai 2015], point clouds [Qi et al. 2017a], and more recently,meshes [Hanocka et al. 2019], Several interesting approaches were proposed to tackle the mesh segmentation problem using deep neural network algorithms such as using convolution operators on edges [Hanocka et al. 2019], converting 3D shapes into voxel-based representations [Graham et al. 2018; Wang et al. 2017; Wu et al. 2015], and leveraging local features of point clouds [Qi et al. 2017a, b]. Nevertheless, these state-of-the-art data-driven methods remain hardly generalizable and often require dividing the learned model into several specific classes of objects to avoid overfitting. In contrast to these methods and [Kovacic et al. 2010], embodiments of the invention avoid the computationally expensive part by using predicted SDF values A to generate the final mesh segmentation. Moreover, learning a mapping function as opposed to directly learning to classify mesh elements [Kalogerakis et al. 2010] (e.g., to cluster vertices) makes embodiments of the invention more robust when used in unknown settings.

[0020] In view of the above, prior art systems process / segment a mesh using two primary steps: (1) using a time consuming machine learning process where a function measures the neighborhood diameter of an object at each vertex to produce clusters; followed by (2) segmenting the mesh based on the clusters. During step (1), the examination of each vertex for distance to neighbors is expensive. In particular, the complexity of the prior art algorithms is O( r) where v is the number of vertices and n is the number maximum of neighbors per vertex. Based on the clusters generated in step (1), the mesh segmentation in step (2) can be relatively fast and trivial (depending on the algorithm utilized). However, the prior art bottleneck is that of step(1).

[0021] In view of the above, it is desirable to provide a system and method that quickly and efficiently determines Shape Diameter Function (SDF) scalar values that define a per-vertex diameter based on their neighborhood (that can then be used to perform mesh segmentation). Further, the selection and use of an efficient mesh segmentation algorithm that takes advantage of the generated scalar values is also advantageous.SUMMARY OF THE INVENTION

[0022] Embodiments of the invention provide the following: (1) generating segmented parts from predictions; (2) refining the generated segmentation; (3) applications and use cases; and (4) usage and update of a learned model. Each of these capabilities are summarized as follows:(1) Generating Segmented Parts from PredictionsContext:

[0023] Embodiments of the invention generate segmented parts from predictions. More specifically, the outputs of the neural network (of embodiments of the invention) are Shape Diameter Function (SDF) scalar values defining a per-vertex diameter based on their neighborhood.Steps:

[0024] The main steps for segmentation are as follows:1. Create vertex clusters based on the SDF values (i.e., using predicted SDF values as input); and2. Find a smooth partitioning using a graph-cut algorithm.Generate SDF clusters:

[0025] In the first step, a Gaussian mixture model (GMM) is used to fit k Gaussians to the histogram of the SDF vertex values. The number of Gaussians (k) is used to discretize the SDF values into a hierarchy of k clusters (i.e., using an expectationmaximization algorithm).Partition clusters using graph-cut:

[0026] Once the clusters are computed in step 1, a graph-cut algorithm is used to smooth the boundaries between clusters (mesh parts). Basically, the graph-cut algorithm minimizes an energy term allowing the transitions to smooth in between mesh parts (i.e., equivalent to applying a threshold determining if a vertex should remain with a cluster or one of its adjacent ones).(2) Refinement StepsContext:

[0027] The generated segmentation requires a few refinement steps to provide aproper solution. As users may be mostly working with polygonal meshes for various applications, it is desirable to provide a clean segmentation matching the object’s discretization (i.e., defined by the polygons).Steps:

[0028] Inspired by best practices when manually segmenting a mesh, embodiments of the invention may apply the three (3) following steps on the generated segmentation:1. Detect symmetry;2. Merge small regions; and3. Proper boundaries.Detecting a symmetry plane:

[0029] Objects with symmetry usually simplify work, halving an object for further processing. Embodiments of the invention use a voting-based approach to identify feature vertices based on their local curvature values. Once these feature vertices are computed, multiple plane candidates are compared to find the one in which combined curvature values are equivalent on both sides.

[0030] A custom symmetry plane can also be provided by the user to speed things up.Merging isolated clusters:

[0031] In this step, embodiments of the invention pass through every cluster to determine if they should be merged with other clusters. The objective of this processing is to remove isolated or small clusters of vertices from the final solution (e.g., a single polygon or a small cluster in the middle of another one).Proper boundaries:

[0032] In this last step, the goal is to smooth the polygonal boundaries between the segmentation parts. As an artist would do, embodiments of the invention detect and apply a smoother path between segmented parts. For example, embodiments may smooth out noisy paths formed by polygonal edges between segmented parts (i.e., unnecessary irregular lines dividing them).(3) Applications and Use Cases

[0033] There are several applications using the approach of embodiments of the invention. Applications may be divided into two (2) categories: (1) General Use Case; and (2) Specific Use Cases.General Use Case:

[0034] As segmenting and labeling meshes is a prerequisite for supervised learning, such an approach would be useful with the data preparation of training sets. Embodiments of the invention can be used to decompose meshes in preparation forclassification and manual labeling.Specific Use Cases:1. Selection

[0035] By identifying parts of complex meshes, embodiments of the invention have the inherent ability to facilitate the selection of meaningful parts for further processing.2. Modeling

[0036] When combined with the ability to select parts of meshes, embodiments of the invention provide ways of altering portions of meshes with traditional modeling tools (e.g., deleting / scaling / replacing selected parts of an input mesh).3. Material assignment

[0037] Assigning material (i.e., defining interaction between surfaces and light sources) for an object can also be augmented with a data-driven segmentation approach of embodiments of the invention. Using the meaningful semantic parts identified, embodiments of the invention can facilitate the material assignment task. For example, with an axe decomposed into a handle and a blade, proper materials such as wood and metal can be easily assigned to these parts.4. Animation

[0038] Animation often requires dividing an object into semantic parts. Afterward, these parts can be used to assign a proper hierarchical structure such as an animation skeleton to deform the mesh accordingly. Using the predicted segmented parts, traditional animation methods can leverage a mesh decomposition (of embodiments of the invention) to optimize the underlying animation structure for control (e.g., decomposing arms into shoulders, upper arms, lower arms, wrists, and hands to integrate animation bones).5. UV mapping

[0039] The UV mapping process can also benefit from the segmentation approach of embodiments of the invention. As part of the UV mapping process, UV unwrapping which is the process of cutting and unwrapping a 3D shape into a 2D space is still today a tedious process for artists. By decomposing a complex mesh into semantic parts, embodiments of the invention simplify the task of unwrapping by processing a set of simpler mesh parts (as opposed to processing a single and more complex mesh).4, Learned Model: Usage and UpdateCore Model:

[0040] Embodiments of the invention provide what may be referred to as a “core model” as a basis for the approach for predicting SDF values on input meshes. Thecore model may be composed of pre-trained weights obtained from computationally expensive training jobs (weeks of training) using training and testing datasets. The provided core model can generalize on various topologies as the model is trained using volumetric terms to learn on local mesh features to predict the SDF values (i.e., a model of embodiments of the invention is resilient to noise and rigid transformations on input objects, making it self-consi stent). In other words, the model may be leamed / updated via machine learning (ML) such that after each iteration the updated model is utilized to recognize new parts and a feedback loop feeds the new parts back to the model to further update the model.In-house Model:

[0041] Naturally, as with most learning techniques, the core model can be updated to match specific needs. The core provides a basic knowledge to generalize on most topologies but can be updated using an in-house dataset to better fit a user’s need.

[0042] Since a goal may be to provide speed-up factors (compared with the state of the art) and more accuracy on mesh segmentation solutions, embodiments of the invention may provide “supervised” ways to update the provided core model. That way, the resulting model can integrate in-house knowledge on multiple shapes while preserving accuracy and avoiding overfitting. In this regard, using cloud services to do so may be suitable to secure users’ data and control how the trained weights are updated (i.e., using the hyperparameters used to generate models: batch size, learning rate, and of course, using the network architecture described herein).BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Referring now to the drawings in which like reference numbers represent corresponding parts throughout:

[0044] FIGs. 1 A and IB illustrate the processing of a mesh in accordance with the prior art;

[0045] FIG. 2 illustrates a hybrid network architecture used to predict SDV values in accordance with one or more embodiments of the invention;

[0046] FIG. 3 illustrates a neural shape diameter approach workflow in accordance with one or more embodiments of the invention;

[0047] FIG. 4A illustrates the result of the generation of a self-consistent dataset in accordance with one or more embodiments of the invention;

[0048] FIG. 4B illustrates the result of the extraction of appropriate features for learning in accordance with one or more embodiments of the invention;

[0049] FIG. 5 illustrates the bash mesh and a subset of blend shapes used to generate a dataset in accordance with one or more embodiments of the invention;

[0050] FIG. 6A illustrates generated SDF values that are used to segment the mesh in accordance with one or more embodiments of the invention;

[0051] FIG. 6B illustrates a close-up of the mesh of FIG. 6 A in accordance with one or more embodiments of the invention;

[0052] FIG. 7 illustrates a simple application example of altering an engraved logo in a 3D modeling application in accordance with one or more embodiments of theinvention;

[0053] FIG. 8 illustrates utilizing the approach of embodiments of the invention to facilitate a 3D unfolding task;

[0054] FIG. 9 illustrates the logical flow for segmenting a volumetric based three- dimensional (3D) object in accordance with one or more embodiments of the invention;

[0055] FIG. 10 is an exemplary hardware and software environment used to implement one or more embodiments of the invention; and

[0056] FIG. 11 schematically illustrates a typical distributed / cloud-based computer system in accordance with one or more embodiments of the invention.DETAILED DESCRIPTION OF THE INVENTION

[0057] In the following description, reference is made to the accompanying drawings which form a part hereof, and which is shown, by way of illustration, several embodiments of the present invention. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.Overview

[0058] Embodiments of the invention provide a data-driven method leveraging a correspondence between the surface and its underlying volume to efficiently segment a polygonal mesh. This correspondence is generated by using a measure based on the medial axis transform from the Shape Diameter Function (SDF). The neural SDF values are then used as inputs to a graph-cut algorithm providing an efficient and controllable workflow for mesh segmentation. By combining an SDF of embodiments of the invention with a state-of-the-art graph-cut algorithm, a high-quality segmentation may be generated for any polygonal mesh at a fraction of the cost. In addition, embodiments of the invention provide an intuitive way to generate various solutions for mesh segmentation by reusing the neural SDF values with different sets of parameters throughout the graph-cut steps.

[0059] Some features of embodiments of the invention include: the introduction and use of a neural Shape Diameter Function improving generalization of local features for mesh segmentation;• a novel approach that reuses neural SDF values to generate various and unique outcomes - making it efficient and highly adaptive;• a resolution-agnostic approach by down-sampling the input mesh while querying the full-resolution mesh for neighborhoods;• utilizing a mesh segmentation algorithm that (i) over-samples the clusters into primitive patches; (ii) uses a probability matrix to encode each patch presence within a cluster; and (iii) uses a graph cut algorithm to perform the segmentation; and• experimental results on various applications of mesh segmentation for computer graphics workflows.Method

[0060] Given a polygonal mesh M, a deep neural network D predicts a mapping function clustering vertices into meaningful groups G. The mapping function learned by the network D is the Shape Diameter Function (SDF) expressing a measure of the diameter of the object’s volume in the neighborhood of each point on the surface. Using the SDF, embodiments of the invention are able to process and manipulate groups of vertices that contain similarities by consistently partitioning and creating skeletons S within meshes. In contrast to [Kovacic et al. 2010], embodiments of the invention skip the computationally expensive part of their method by using predicted SDF values to generate the final mesh segmentation. Learning a mapping function as opposed to directly learning to classify mesh elements [Kalogerakis et al. 2010] (e.g.,to cluster vertices or edges) makes embodiments of the invention more robust when used in unknown settings.

[0061] Once the SDF values are predicted, embodiments of the invention may follow an approach that is highly efficient and parameterizable. A graph cut algorithm may be used to partition the resulting skeleton S associated with an input mesh M and obtained by network D. From those cuts, a set of meaningful segmented parts P are produced. In addition, since the SDF is a volume-based function, the robustness of embodiments of the invention can be improved by directly including the implicit surface in the loss function used with the network D, allowing embodiments to remain resilient to animation pose changes of the same objects in the training set.

[0062] In view of the above, as a formal definition, the traditional mesh segmentation task is described as follows: given a closed manifold mesh M, and E the set of mesh elements (i.e., vertices, edges, or faces). The segmentation S of mesh M is a set of sub-meshes S = {Mo, ..., Mn-i }, where Mt is defined as a subset of elements e e E.Neural SDF

[0063] In the traditional Shape Diameter Function method, the set of sub- meshes S is obtained by subdividing a graph using the diameter measures di from per-vertex neighborhoods Vi as thresholds for the graph-cut algorithm. As embodiments of the invention aim to estimate these SDF values Ai through a graph neural network EMD, the problem of finding the sub-meshes Mi can be expressed as partitioning S such thatthe constraint criteria {Co, ..., Cn} are minimized. The predicted SDF values Xi are then used as threshold criteria to a graph-cut algorithm to generate the sub-meshes S = {Mo, ..., Mn-1 }.

[0064] The constraint criteria {Co, ..., Cn} are defined as two terms: the similarities between the reference SDF values A and the predicted SDF values Xi, and the neighborhood densities pi of vertices Vi. The Lz similarities between the reference SDF values and the predicted ones may be expressed as follows:where n is the number of down-sampled surface vertices (i.e., used to query the neighborhoods). Along with the input vertices, embodiments of the invention provide the network EMD with an additional term to weigh in the local mesh density for adaptive resolution meshes. Embodiments may use the Poisson sampling disk method at the surface of meshes to select the down-sampled points and compute their neighborhood density. The resulting densities pi is then used as a scaling factor during the messaging stage within the network to properly propagate the attributes in the current neighborhood.Model Architecture

[0065] FIG. 2 illustrates a hybrid network architecture used to predict SDV values (e.g., in the training phase) in accordance with one or more embodiments of the invention. Similar to [Verma et al. 2018], the hybrid network D is composed of both linear 202 and graph layers 204. The linear layers 202 are used to warm up thenetwork with typical padding operations, as performed with images 200, reducing the limitations related to varying input mesh sizes. The graph layers 204 are then used with convolution operations to establish correspondences between filter weights and graph neighborhoods with arbitrary connectivity. Finally, a last dense regression layer 206 projects the predictions back into an image 208.

[0066] FIG. 3 illustrates a neural shape diameter approach workflow. Training 302 (also referred to as an encoder-messenger-decoder network [and / or a neural network]) is conducted using an input 304 that includes meshes (e.g., both full-resolution meshes [for neighbors when updating nodes by the messenger] and coarse-resolution meshes [down-sampled for training]). In one or more embodiments, the encoder- messenger-decoder (EMD) network 302 includes the encoder 302 A, messenger 302B, and decoder 302C). The EMD network 302 generates the predicted SDF values 306 used by a segmenting algorithm 308 (e.g., a k-way graph-cut algorithm 308 (i.e., a gaussian mixture and clustering on the graphics processing unit [GPU]) to partition / segment the final segmentation (i.e., generating output 310).

[0067] In view of the above, the network EMD architecture 300 is essentially based on the Encode-Message-Decode model. The network 302 is composed of two hidden MLP layers on both Encoder 302 A and Messenger 302B with an output size of 128. The resulting Decoder 302C output size matches the down-sampled points provided as input 304 during training and inference. Embodiments of the invention may be trained and loaded for inference using a A6000 GPU with the Adam optimizer for 5M training steps with an exponential learning rate decay from 10'3to 10'5when passingthe threshold of 3M steps. Further, the training / prediction 302 may include a feedback loop and may be implemented via machine learning (ML) where new parts are recognized and fed back for further updating of the model.

[0068] Resolution-Agnostic Graph Network. Embodiments of the invention handle varying resolutions using two mechanisms: down-sampling the input mesh 304 while keeping the full-resolution neighborhoods and using the neighborhood densities for each node as scaling factors when updating the nodes during the messaging stage. The idea to remain resolution-agnostic is to pass the full-resolution neighborhoods along the down-sampled mesh. In this regard, embodiments of the invention can query the vertex neighborhoods by solely using the down-sampled mesh vertices to compute the SDF values A. Moreover, as the network architecture may require a known input size, a fixed radius may be used with the Poisson disk sampling algorithm to compute the neighborhood densities.

[0069] Graph Cut / Segmenting Algorithm 308. Once the predicted SDF valuesA are obtained, they may be used as inputs to a fast fc-way graph-cut algorithm 308 to offer an efficient and flexible way for mesh segmentation. Such an approach can leverage the predicted SDF values 306 using a grid-search method to find the optimal parameters for the mesh segmentation. Embodiments may also use a GPU implementation of the fc-way graph partitioning so the graph-cut algorithm 308 can leverage the nvGRAPH library from NVIDIA.

[0070] The partitioning algorithm of embodiments of the invention may be composed of two steps. The first step uses soft-clustering of the mesh elements (faces)to compute k clusters based on their SDF values, and the second step finds the actual partitioning using fc-way graph-cut to include local mesh geometric properties. Note that k, the number of clusters chosen, is more naturally related to the number of levels in the hierarchy and not to the number of parts.

[0071] As an optional post-processing step, embodiments may recursively use such an approach as a refinement process to improve the segmentation of detailed parts.Dataset and Features

[0072] The dataset used to train the network D I EMD 302 may be generated using the AUTODESK CHARACTER GENERATOR ™ tool producing multiple variants of the same base mesh by applying blend shape operations on it (as shown in FIGs. 4A and 4B). In this regard, FIG. 4A illustrates the result of the generation of a self- consistent dataset, and FIG. 4B illustrates the result of the extraction of appropriate features for learning in accordance with one or more embodiments of the invention.

[0073] The generated dataset of FIG. 4 A is self-consi stent since it is exclusively composed of meshes having the same number of vertices. To train the network 302 to encode resolution-agnostic features, a tessellation technique may be used during training to provide multiple versions of the same mesh. A remeshing method may also be used to make the network 302 resilient to consistent input samples 304 (i.e., by changing the positions of selected downsampled vertices).

[0074] FIG. 5 illustrates the bash mesh and a subset of blend shapes used to generate a dataset in accordance with one or more embodiments of the invention. Asshown in FIG. 5, several blend shapes B 502 may be used to alter a base mesh into many variants to grow the dataset. The blend shapes B 502 are divided into two groups: facial Bf 502A (e.g., traits such as ears, mouth, chin, nose, cheeks, etc. and / or features such as elvish, bulk, gorn, or, standard, etc.) and body B6502B (e.g., traits such as arms, shoulders, chest, stomach, legs, etc. and / or features such as muscle, skinny, bulk, cloth, posture, etc.) (collectively referred to as blend shapes 502). For both groups, the blend shapes used to augment the dataset are described as either traits Bt or features B . By permuting these blend shapes, a large dataset can be generated using solely a single base mesh. Additionally, embodiments of the invention may use an animation skeleton to randomly generate different poses as it may generate different SDF values for the same mesh. Further, an anisotropic smoothing method may be used on the SDF values to overcome these differences during training. Moreover, embodiments of the invention may build an additional custom dataset using TURBOS QUID (TS)™ assets to evaluate the approach with production-ready content.

[0075] Further and / or in addition to the above, to improve generalization, embodiments of the invention may transpose the input features used during training to local space, making these motions invariant. In addition to making them resilient to rigid transformations, learning these features locally allows us to better generalize on complex inputs. The objective function of embodiments of the invention may be composed of two loss terms to minimize: LSDF and L<j> We first compare predicted values SDF (M) with the ground truth.where the reference SDF values are obtained from the method proposed by [Kovacic et al. 2010], Then, the volume-based formulation of the method is leveraged to include another Sh loss term <2® focusing on minimizing the similarities in topology of the final solution. The loss function may use values from the underlying implicit surface and its associated gradient (approximate normals) and is expressed as follows:where V is the set of vertices of the input mesh M. Lastly, both loss terms are combined providing a more resilient update during epochs for various samples.Experiments and Results

[0076] Embodiments of the invention were compared against a variety of state-of- the-art methods to determine efficiency and precision while remaining controllable for the users. In the following sections, the approach of embodiments of the invention are compared with a few baselines using known segmentation datasets and datasets of embodiments of the invention. Computation times are also presented on several scenarios. A particular case is also described where embodiments of the invention perform better than the state-of-the-art on dense meshes. Lastly, a few applications are presented that are useful to improve traditional graphics workflows.Performance Analysis

[0077] As embodiments of the invention may solely focus on mesh segmentation, performance analysis may be limited to recent methods performing well in that area. Four datasets are focused on to evaluate precision: COSEG, HUMAN BODY SEGMENTATION (HBS)™, AUTODESK CHARACTER GENERATOR™, and TURBOSQUID™. The precision of embodiments of the invention on the segmentation task were compared with POINTNET, POINTNET++, and MESHCNN.TABLE 1

[0078] As presented in Table 1, embodiments of the invention perform similarly to MESHCNN and slightly outperform POINTNET and POINTNET++ on the COSEG (only the Vases set) and HBS datasets. Also highlighted in that same table, embodiments of the invention are way more accurate when used on the ACG and TS datasets. This is not surprising as embodiments of the invention are trained on multiple-resolution samples. By querying the full resolution neighborhoods and their corresponding downsampled mesh, embodiments of the invention have proven to be less sensitive to adaptive meshing, which is the case for most of the 3D models contained in the TS dataset. Lastly, as embodiments of the invention aim to speed up the whole traditional mesh segmentation process, a few computation-time results arecompiled on the presented assets.

[0079] As shown in Table 2, the neural -based approach of embodiments of the invention shows a speed-up factor of up to lOx compared to the original Shape Diameter Function method in most of the presented assets. Finally, to improve the partitioning part, embodiments of the invention implemented the gaussian mixture and the clustering to GPU. With such an implementation, embodiments of the invention almost halved the computation times required by these steps.Dense Meshses

[0080] It may also be demonstrated that the refinement and post-processing steps of embodiments of the invention improve the results by a few points on dense meshes using the refinement and post-processing steps. With an error rate of below 2% compared to the ground truth, the neural SDF approach (of embodiments of the invention) performs well on dense meshes.

[0081] Table 3 illustrates precision metrics obtained on several 3D models with performances broken down into four parts: generating SDF values, partitioning (Part.), refinement (Ref.), and post processing (PP). As shown in Table 3, the approach of embodiments of the invention produces high-precision SDF values on both the SAMURAI and GLADIATOR HULK meshes. One can provide even more precise values using the refinement and post-processing steps. For example, with the GLADIATOR HULK mesh (FIGs. 6A and 6B), small additional details were extracted using the refinement step on segmented parts such as the shoulder pad presented in the close-up (FIG. 6B). In this regard, FIG. 6A illustrates generated SDF values that are used to segment the mesh with a close-up of the mesh illustrated in FIG. 6B in accordance with one or more embodiments of the invention.

[0082] On a SAMURAI mesh, better results of the segmentation were obtained after post-processing the boundaries (between each segmented part) to smooth them out using unwrapping heuristics (such as loop cuts). Refinement and post-processing steps were not used when the resulting segmentation seemed adequate.Applications

[0083] As previously stated, many tasks intuitively require decomposing meshes before further processing. Mesh segmentation finds many applications in 3D shapeanalysis such as: texture mapping [Sander et al. 2003], 3D shape modeling [Ji et al. 2006], 3D shape retrieval [Ferreira et al. 2010], multi-resolution and mesh compression [Maglo et al. 2011]; and animation [Yuan et al. 2016], [Rodrigues et al. 2018] provides a survey on part-based mesh segmentation.

[0084] Embodiments of the invention introduce an efficient way to facilitate these various applications through an efficient inferred mapping while reducing the computational times required by a factor of 100 x. For example, FIG. 7 illustrates a simple application example of altering an engraved logo in a 3D modeling application in accordance with one or more embodiments of the invention. Specifically, FIG. 7 shows a typical example in which a cube 702 with an engraved logo is decomposed into meaningful parts 704 using a neural SDF approach. Once segmented 706, the parts can then be selected in 708 (indicated by 1 and 2) for further manipulations such as: replacing with a different logo, deforming the selected logo, or as shown at 710, scaling up and removing parts of the logo.

[0085] Embodiments of the invention have also experimented to see how effective the approach would be in the case of UV mapping. The unwrapping of parts is known to be very unintuitive and requires an expert knowledge on shape topology to properly unwrap 3D models into 2D while preserving the semantic meaning of the parts. There has been attempts to tackle this directly by predicting UV seams using a graph attention network [Teimury et al. 2020], but such approaches were very limited to the training set and were hardly generalizable for production uses. Embodiments of the invention are able to provide semantically meaningful parts on complex models topartially alleviate a fair portion of the pain with respect to the UV mapping workflow. That way, each simpler part can be unwrapped using only a few cuts and simple projection methods.

[0086] FIG. 8 illustrates utilizing the approach of embodiments of the invention to facilitate a 3D unfolding task. As shown, the gas can 802 is first segmented into meaningful parts 804 before running an automatic unwrapping on the whole mesh based on the prominent shape features (to generate the unwrapped mesh 806), and then performing the automatic UV mapping recursively on the segmented parts to generate the image 808. From the segmented object 806, embodiments of the invention clearly outperformed the one generated using an automatic method in the MAYA™ application.Logical Flow

[0087] FIG. 9 illustrates the logical flow for segmenting a volumetric based three- dimensional (3D) object in accordance with one or more embodiments of the invention.

[0088] At step 902, a training dataset is acquired. The training dataset includes / consists of multiple volumetric based three-dimensional (3D) training objects.

[0089] At step 904, a neural network is obtained. The neural network is defined by multiple layers. The multiple layers include at least one linear layer and at least one graph layer. The linear layer operates on 2D objects. An output of the linear layerprovides an input to the graph layer. The graph layer performs convolution operations on vertices of the 3D training objects.

[0090] At step 906, a model is generated based on the training dataset. The model approximates a shape diameter function. The shape diameter function determines neighborhood diameters. The neighborhood diameters include a distance from a first vertex of one or more of the 3D training objects to an antipodal vertex (antipodal is the opposite point) of the one or more 3D training objects. The model is generated by iterating through the one or more 3D training objects using the neural network to converge on weights of each feature of the shape diameter function. Further, during each iteration, an error based on the weights is minimized.

[0091] The error based on the weights may be minimized for: (a) a predicted shape diameter function (SDF) value for a single model compared to a ground truth SDF value; (b) a predicted signed distance function compared to a ground truth signed distance function; and / or (c) a gradient of the predicted signed distance function compared to a gradient of the ground truth signed distance function.

[0092] The model may also be updated (after each iteration) via machine learning (ML). The updated model may then be utilized to recognize new parts and then fed back into the ML for further updating of the model.

[0093] Further, the model may support varying input sizes by downsampling one or more of the 3D training objects or other 3D input objects to a coarser structure while keeping full-resolution neighborhoods. Such a downsampling may include: (a) making a graph network resolution-agnostic by using a fixed input size; (b) queryingthe full-resolution neighborhoods when converging on the weights; and (c) updating the weights during the converging based on contributions of the full-resolution neighborhoods.

[0094] At step 908, an input mesh for a first new volumetric based 3D object is acquired.

[0095] At step 910, the converged weights are used to approximate shape diameter function (SDF) values. The approximate shape diameter function may then be utilized as input to enable user control.

[0096] At step 912, the approximate shape diameter function values are used as an input to a graph cut algorithm. The graph cut algorithm generates vertex clusters that define a segmented part of the input mesh.

[0097] At step 914, the segmented part is visually displayed or provided.Hardware Environment

[0098] FIG. 10 is an exemplary hardware and software environment 1000 (referred to as a computer-implemented system and / or computer-implemented method) used to implement one or more embodiments of the invention. The hardware and software environment includes a computer 1002 and may include peripherals. Computer 1002 may be a user / client computer, server computer, or may be a database computer. The computer 1002 comprises a hardware processor 1004A and / or a special purpose hardware processor 1004B (hereinafter alternatively collectively referred to as processor 1004) and a memory 1006, such as random access memory (RAM). Thecomputer 1002 may be coupled to, and / or integrated with, other devices, including input / output (I / O) devices such as a keyboard 1014, a cursor control device 1016 (e.g., a mouse, a pointing device, pen and tablet, touch screen, multi-touch device, etc.) and a printer 1028. In one or more embodiments, computer 1002 may be coupled to, or may comprise, a portable or media viewing / listening device 1032 (e.g., an MP3 player, IPOD, NOOK, portable digital video player, cellular device, personal digital assistant, etc.). In yet another embodiment, the computer 1002 may comprise a multitouch device, mobile phone, gaming system, internet enabled television, television set top box, or other internet enabled device executing on various platforms and operating systems.

[0099] In one embodiment, the computer 1002 operates by the hardware processor 1004A performing instructions defined by the computer program 1010 (e.g., a computer-aided design [CAD] application) under control of an operating system 1008. The computer program 1010 and / or the operating system 1008 may be stored in the memory 1006 and may interface with the user and / or other devices to accept input and commands and, based on such input and commands and the instructions defined by the computer program 1010 and operating system 1008, to provide output and results.

[0100] Output / results may be presented on the display 1022 or provided to another device for presentation or further processing or action. In one embodiment, the display 1022 comprises a liquid crystal display (LCD) having a plurality of separately addressable liquid crystals. Alternatively, the display 1022 may comprise a lightemitting diode (LED) display having clusters of red, green and blue diodes driven together to form full-color pixels. Each liquid crystal or pixel of the display 1022 changes to an opaque or translucent state to form a part of the image on the display in response to the data or information generated by the processor 1004 from the application of the instructions of the computer program 1010 and / or operating system 1008 to the input and commands. The image may be provided through a graphical user interface (GUI) module 1018. Although the GUI module 1018 is depicted as a separate module, the instructions performing the GUI functions can be resident or distributed in the operating system 1008, the computer program 1010, or implemented with special purpose memory and processors.

[0101] In one or more embodiments, the display 1022 is integrated with / into the computer 1002 and comprises a multi-touch device having a touch sensing surface (e.g., track pod or touch screen) with the ability to recognize the presence of two or more points of contact with the surface. Examples of multi-touch devices include mobile devices (e.g., IPHONE, NEXUS S, DROID devices, etc.), tablet computers (e g., IPAD, HP TOUCHPAD, SURFACE Devices, etc ), portable / handheld game / music / video player / console devices (e.g., IPOD TOUCH, MP3 players, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc ), touch tables, and walls (e.g., where an image is projected through acrylic and / or glass, and the image is then backlit with LEDs).

[0102] Some or all of the operations performed by the computer 1002 according to the computer program 1010 instructions may be implemented in a special purposeprocessor 1004B. In this embodiment, some or all of the computer program 1010 instructions may be implemented via firmware instructions stored in a read only memory (ROM), a programmable read only memory (PROM) or flash memory within the special purpose processor 1004B or in memory 1006. The special purpose processor 1004B may also be hardwired through circuit design to perform some or all of the operations to implement the present invention. Further, the special purpose processor 1004B may be a hybrid processor, which includes dedicated circuitry for performing a subset of functions, and other circuits for performing more general functions such as responding to computer program 1010 instructions. In one embodiment, the special purpose processor 1004B is an application specific integrated circuit (ASIC).

[0103] The computer 1002 may also implement a compiler 1012 that allows an application or computer program 1010 written in a programming language such as C, C++, Assembly, SQL, PYTHON, PROLOG, MATLAB, RUBY, RAILS, HASKELL, or other language to be translated into processor 1004 readable code. Alternatively, the compiler 1012 may be an interpreter that executes instruct! ons / source code directly, translates source code into an intermediate representation that is executed, or that executes stored precompiled code. Such source code may be written in a variety of programming languages such as JAVA, JAVASCRIPT, PERL, BASIC, etc. After completion, the application or computer program 1010 accesses and manipulates data accepted from I / O devices and stored in the memory 1006 of the computer 1002 using the relationships and logic that were generated using the compiler 1012.

[0104] The computer 1002 also optionally comprises an external communication device such as a modem, satellite link, Ethernet card, or other device for accepting input from, and providing output to, other computers 1002.

[0105] In one embodiment, instructions implementing the operating system 1008, the computer program 1010, and the compiler 1012 are tangibly embodied in a non- transitory computer-readable medium, e.g., data storage device 1020, which could include one or more fixed or removable data storage devices, such as a zip drive, floppy disc drive 1024, hard drive, CD-ROM drive, tape drive, etc. Further, the operating system 1008 and the computer program 1010 are comprised of computer program 1010 instructions which, when accessed, read and executed by the computer 1002, cause the computer 1002 to perform the steps necessary to implement and / or use the present invention or to load the program of instructions into a memory 1006, thus creating a special purpose data structure causing the computer 1002 to operate as a specially programmed computer executing the method steps described herein.Computer program 1010 and / or operating instructions may also be tangibly embodied in memory 1006 and / or data communications devices 1030, thereby making a computer program product or article of manufacture according to the invention. As such, the terms “article of manufacture,” “program storage device,” and “computer program product,” as used herein, are intended to encompass a computer program accessible from any computer readable device or media.

[0106] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and otherdevices, may be used with the computer 1002.

[0107] FIG. 11 schematically illustrates a typical distributed / cloud-based computer system 1100 using a network 1104 to connect client computers 1102 to server computers 1106. A typical combination of resources may include a network 1104 comprising the Internet, LANs (local area networks), WANs (wide area networks), SNA (systems network architecture) networks, or the like, clients 1102 that are personal computers or workstations (as set forth in FIG. 10), and servers 1106 that are personal computers, workstations, minicomputers, or mainframes (as set forth in FIG. 10). However, it may be noted that different networks such as a cellular network (e.g., GSM [global system for mobile communications] or otherwise), a satellite based network, or any other type of network may be used to connect clients 1102 and servers 1106 in accordance with embodiments of the invention.

[0108] A network 1104 such as the Internet connects clients 1102 to server computers 1106. Network 1104 may utilize ethemet, coaxial cable, wireless communications, radio frequency (RF), etc. to connect and provide the communication between clients 1102 and servers 1106. Further, in a cloud-based computing system, resources (e.g., storage, processors, applications, memory, infrastructure, etc.) in clients 1102 and server computers 1106 may be shared by clients 1102, server computers 1106, and users across one or more networks. Resources may be shared by multiple users and can be dynamically reallocated per demand. In this regard, cloud computing may be referred to as a model for enabling access to a shared pool of configurable computing resources.

[0109] Clients 1102 may execute a client application or web browser and communicate with server computers 1106 executing web servers 1110. Such a web browser is typically a program such as MICROSOFT INTERNET EXPLORER / EDGE, MOZILLA FIREFOX, OPERA, APPLE SAFARI, GOOGLE CHROME, etc. Further, the software executing on clients 1102 may be downloaded from server computer 1106 to client computers 1102 and installed as a plug-in or ACTIVEX control of a web browser. Accordingly, clients 1102 may utilize ACTIVEX components / component object model (COM) or distributed COM (DCOM) components to provide a user interface on a display of client 1102. The web server 1110 is typically a program such as MICROSOFT’S INTERNET INFORMATION SERVER.

[0110] Web server 1110 may host an Active Server Page (ASP) or Internet Server Application Programming Interface (ISAPI) application 1112, which may be executing scripts. The scripts invoke objects that execute business logic (referred to as business objects). The business objects then manipulate data in database 1116 through a database management system (DBMS) 1114. Alternatively, database 1116 may be part of, or connected directly to, client 1102 instead of communicating / obtaining the information from database 1116 across network 1104. When a developer encapsulates the business functionality into objects, the system may be referred to as a component object model (COM) system. Accordingly, the scripts executing on web server 1110 (and / or application 1112) invoke COM objects that implement the business logic. Further, server 1106 may utilize MICROSOFT’S TRANSACTION SERVER (MTS)to access required data stored in database 1116 via an interface such as ADO (Active Data Objects), OLE DB (Object Linking and Embedding DataBase), or ODBC (Open DataBase Connectivity).

[0111] Generally, these components 1100-1116 all comprise logic and / or data that is embodied in / or retrievable from device, medium, signal, or carrier, e.g., a data storage device, a data communications device, a remote computer or device coupled to the computer via a network or via another data communications device, etc. Moreover, this logic and / or data, when read, executed, and / or interpreted, results in the steps necessary to implement and / or use the present invention being performed.

[0112] Although the terms “user computer”, “client computer”, and / or “server computer” are referred to herein, it is understood that such computers 1102 and 1106 may be interchangeable and may further include thin client devices with limited or full processing capabilities, portable devices such as cell phones, notebook computers, pocket computers, multi-touch devices, and / or any other devices with suitable processing, communication, and input / output capability.

[0113] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used with computers 1102 and 1106. Embodiments of the invention are implemented as a software / CAD application on a client 1102 or server computer 1106. Further, as described above, the client 1102 or server computer 1106 may comprise a thin client device or a portable device that has a multi-touch-based display.Conclusion

[0114] This concludes the description of the preferred embodiment of the invention. The following describes some alternative embodiments for accomplishing the present invention. For example, any type of computer, such as a mainframe, minicomputer, or personal computer, or computer configuration, such as a timesharing mainframe, local area network, or standalone personal computer, could be used with the present invention.

[0115] In summary, embodiments of the invention provide a Neural Shape Diameter Function approach that considerably accelerates high-quality mesh segmentation while remaining adaptive and controllable. Using the generated SDF values, embodiments of the invention can generate multiple alternative segmentation solutions of the same mesh by adjusting the partitioning parameters for a k-way graph-cut algorithm.

[0116] Given the fact that production meshes often use adaptive tessellation to optimize memory on a GPU based on the local complexity of the topology, graph-cut algorithms have proven to be less effective than deep neural networks to segment such irregular structures with highly varying numbers of neighbors per vertex (as opposed to images). While embodiments of the invention partially alleviates that part by querying the full-resolution mesh, using stacked networks, the current network output may be leveraged as input to a single non-binary classifier.

[0117] The foregoing description of the preferred embodiment of the invention has been presented for the purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.References

[0118] [Attene et. al. 2006] Marco Attene, Bianca Falcidieno, and Michela Spagnuolo. 2006. Hierarchical mesh segmentation based on fitting primitives. The Visual Computer 22, 3 (2006), 181-193.

[0119] [Chazelle et. al. 1995] Bernard Chazelle, David P Dobkin, Nadia Shouraboura, and Ayellet Tai. 1995. Strategies for polyhedral surface decomposition: An experimental study. In Proceedings of the eleventh annual symposium on Computational geometry. 297-305.

[0120] [Cohen- Steiner et. al. 2004] David Cohen- Steiner, Pierre Alliez, andMathieu Desbrun. 2004. Variational shape approximation. In ACM SIGGRAPH 2004 Papers. 905-914.

[0121] [Eck et. al. 1995] Matthias Eck, Tony DeRose, Tom Duchamp, Hugues Hoppe, Michael Lounsbery, and Werner Stuetzle. 1995. Multiresolution analysis of arbitrary meshes. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 173-182.

[0122] [Ferreira et. al. 2010] Alfredo Ferreira, Simone Marini, Marco Attene, Manuel J Fonseca, Michela Spagnuolo, Joaquim A Jorge, and Bianca Falcidieno.2010. Thesaurus-based 3D object retrieval with part-in-whole matching. International Journal of Computer Vision 89, 2 (2010), 327-347.

[0123] [Garland et. al. 2001] Michael Garland, Andrew Willmott, and Paul S Heckbert. 2001. Hierarchical face clustering on polygonal surfaces. In Proceedings of the 2001 symposium on Interactive 3D graphics. 49-58.

[0124] [Gelfand et. al. 2004] Natasha Gelfand and Leonidas J Guibas. 2004. Shape segmentation using local slippage analysis. In Proceedings of the 2004 Eurographics / ACM SIGGRAPH symposium on Geometry processing. 214-223.

[0125] [Graham et. al. 2018] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 2018. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9224-9232.

[0126] [Hanocka et. al. 2019] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. Meshcnn: a network with an edge. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1-12.

[0127] [Hart et. al. 2000] Peter E Hart, David G Stork, and Richard O Duda. 2000. Pattern classification. Wiley Hoboken.

[0128] [Zhongping et. al. 2006] Zhongping Ji, Ligang Liu, Zhonggui Chen, and Guojin Wang. 2006. Easy mesh cutting. In Computer Graphics Forum, Vol. 25. Wiley Online Library, 283-291.

[0129] [Kalogerakis et. al. 2010] Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D mesh segmentation and labeling. In ACMSIGGRAPH 2010 papers. 1-12.

[0130] [Kalvin et. al. 1996] Alan D Kalvin and Russell H Taylor. 1996. Superfaces: Polygonal mesh simplification with bounded error. IEEE Computer Graphics and Applications 16, 3 (1996), 64-77.

[0131] [Katz et. al. 2003] Sagi Katz and Ayellet Tai. 2003. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM transactions on graphics (TOG) 22, 3 (2003), 954-961.

[0132] [Kovacic et. al. 2010] Maurizio Kovacic, Fabio Guggeri, Stefano Marras, and Riccardo Scateni. 2010. Fast approximation of the shape diameter function. In Proceedings Workshop on Computer Graphics, Computer Vision and Mathematics (GraVisMa), Vol. 5.

[0133] [Lai 2015] Matthew Lai. 2015. Deep learning for medical image segmentation. arXiv preprint arXiv: 1505.02000 (2015).

[0134] [Lavoue et. al. 2005] Guillaume Lavoue, Florent Dupont, and Atilla Baskurt. 2005. A new CAD mesh segmentation method, based on curvature tensor analysis.Computer-Aided Design 37, 10 (2005), 975-987.

[0135] [Lee et. al. 2005] Yunjin Lee, Seungyong Lee, Ariel Shamir, Daniel Cohen- Or, and Hans-Peter Seidel. 2005. Mesh scissoring with minima rule and part salience. Computer Aided Geometric Design 22, 5 (2005), 444-465.

[0136] [Levy et. al. 2002] Bruno Levy, Sylvain Petitjean, Nicolas Ray, and Jerome Maillot. 2002. Least squares conformal maps for automatic texture atlas generation. ACM transactions on graphics (TOG) 21, 3 (2002), 362-371.

[0137] [Li et. al. 2001] Xuetao Li, Tong Wing Woon, Tiow Seng Tan, and ZhiyongHuang. 2001. Decomposing polygon meshes for interactive applications. In Proceedings of the 2001 symposium on Interactive 3D graphics. 35-42.

[0138] [Lien et. al. 2006] Jyh-Ming Lien, John Keyser, and Nancy M Amato. 2006.Simultaneous shape decomposition and skeletonization. In Proceedings of the 2006 ACM symposium on Solid and physical modeling. 219-228.

[0139] [Lloyd 1982] Stuart Lloyd. 1982. Least squares quantization in PCM. IEEE transactions on information theory 28, 2 (1982), 129-137.

[0140] [Maglo et. al. 2011] Adrien Maglo, Ian Grimstead, and Celine Hudelot.2011. Cluster-based random accessible and progressive lossless compression of colored triangular meshes for interactive visualization. In Proceedings of computer graphics international.

[0141] [Mitani et. al. 2004] Jun Mitani and Hiromasa Suzuki. 2004. Making papercraft toys from meshes using strip-based approximate unfolding. ACM transactions on graphics (TOG) 23, 3 (2004), 259-263.

[0142] [Podolak et. al. 2006] Joshua Podolak, Philip Shilane, Aleksey Golovinskiy,Szymon Rusinkiewicz, and Thomas Funkhouser. 2006. A planar-reflective symmetry transform for 3D shapes. In ACM SIGGRAPH 2006 Papers. 549-559.

[0143] [Qi et. al. 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.2017a. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652- 660.

[0144] [Qi et. al. 2017] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017).

[0145] [Raab et. al. 2004] R Raab, C Gotsman, and A Sheffer. 2004. Virtual woodwork: Generating bead figures from 3d models. International Journal on Shape Modeling 10, 1 (2004), 1-30.

[0146] [Rodrigues et. al. 2018] Rui SV Rodrigues, Jose FM Morgado, and Abel JP Gomes. 2018. Part-based mesh segmentation: a survey. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 235-274.

[0147] [Sander et. al. 2001] Pedro V Sander, John Snyder, Steven J Gortler, and Hugues Hoppe. 2001. Texture mapping progressive meshes. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 409-416.

[0148] [Sander et. al. 2003] Pedro V Sander, Zoe J Wood, Steven Gortler, John Snyder, and Hugues Hoppe. 2003. Multi-chart geometry images. (2003).

[0149] [Shapira et. al. 2008] Lior Shapira, Ariel Shamir, and Daniel Cohen-Or. 2008. Consistent mesh partitioning and skeletonisation using the shape diameter function. The Visual Computer 24, 4 (2008), 249-259.

[0150] [Kraevoy et. al. 2007] Vladislav Kraevoy Dan Julius Alla Sheffer. 2007.Shuffler: Modeling with interchangeable parts. Visual Computer journal (2007).

[0151] [Shlafman et. al. 2002] Shymon Shlafman, Ayellet Tai, and Sagi Katz. 2002. Metamorphosis of polyhedral surfaces using decomposition. In Computer graphics forum, Vol. 21. Wiley Online Library, 219-228.

[0152] [Sorkine et. al. 2002] Olga Sorkine, Daniel Cohen-Or, Rony Goldenthal, and Dani Lischinski. 2002. Boundeddistortion piecewise mesh parameterization. In IEEE Visualization, 2002. VIS 2002. IEEE, 355-362.

[0153] [Teimury et. al. 2020] Fatemeh Teimury, Bruno Roy, Juan Sebastian Casallas, David MacDonald, and Mark Coates. 2020. GraphSeam: Supervised Graph Learning Framework for Semantic UV Mapping. arXiv preprint arXiv:2011.13748 (2020).

[0154] [Wang et. al. 2017] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. 2017. O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACMTransactions On Graphics (TOG) 36, 4 (2017), 1-11.

[0155] [Wu et. al. 2005] Jianhua Wu and Leif Kobbelt. 2005. Structure Recovery via Hybrid Variational Surface Approximation.. In Comput. Graph. Forum, Vol. 24. 277-284.

[0156] [Wu 2 et. al. 015] ZhirongWu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1912-1920.

[0157] [Yuan et. al. 2016] Qing Yuan, Guiqing Li, Kai Xu, Xudong Chen, and Hui Huang. 2016. Space-time cosegmentation of articulated point cloud sequences. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 419-429.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for segmenting a volumetric based three-dimensional (3D) object comprising:(a) acquiring a training dataset, wherein the training dataset comprises multiple volumetric based three-dimensional (3D) training objects;(b) obtaining a neural network, wherein:(i) the neural network is defined by multiple layers;(ii) the multiple layers comprise at least one linear layer and at least one graph layer;(iii) the linear layer operates on 2D objects;(iv) an output of the linear layer comprises an input to the graph layer;(v) the graph layer is performing convolution operations on vertices of the 3D training objects;(c) generating a model based on the training dataset, wherein:(i) the model approximates a shape diameter function;(ii) the shape diameter function determines neighborhood diameters;(iii) the neighborhood diameters comprise a distance from a first vertex of one or more of the 3D training objects to an antipodal vertex of the one or more 3D training objects;(iv) the model is generated by iterating through the one or more 3Dtraining objects using the neural network to converge on weights of each feature of the shape diameter function, wherein during each iteration an error based on the weights is minimized;(d) acquiring an input mesh for a first new volumetric based 3D object;(e) using the converged weights to approximate shape diameter function values;(f) using the approximate shape diameter function values as an input to a graph cut algorithm, wherein the graph cut algorithm generates vertex clusters that define a segmented part of the input mesh; and(g) visually displaying or providing the segmented part.

2. The computer-implemented method of claim 1, wherein the error based on the weights is minimized for:(a) a predicted shape diameter function (SDF) value for a single model compared to a ground truth SDF value.

3. The computer-implemented method of claim 1, wherein the error based on the weights is minimized for:(a) a predicted signed distance function compared to a ground truth signed distance function.

4. The computer-implemented method of claim 1, wherein the error basedon the weights is minimized for:(a) a gradient of the predicted signed distance function compared to a gradient of the ground truth signed distance function.

5. The computer-implemented method of claim 1, further comprising:(a) updating the model via machine learning (ML) wherein the model is updated after each iteration; and(b) the updated model is utilized to recognize new parts and then fed back into the ML for further updating of the model.

6. The computer-implemented method of claim 1, wherein the approximate shape diameter function utilized as input enables user control.

7. The computer-implemented method of claim 1, wherein the model supports varying input size by downsampling one or more of the 3D training objects or other 3D input objects to a coarser structure while keeping full-resolution neighborhoods.

8. The computer-implemented method of claim 7, wherein the downsampling comprises:(a) making a graph network resolution-agnostic by using a fixed input size;(b) querying the full-resolution neighborhoods when converging on the weights; and(c) updating the weights during the converging based on contributions of the full-resolution neighborhoods.

9. A computer-implemented system for segmenting a volumetric based three-dimensional (3D) object, comprising:(a) a computer having a memory;(b) a processor executing on the computer;(c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor cause the processor to perform operations comprising:(i) acquiring a training dataset, wherein the training dataset comprises multiple volumetric based three-dimensional (3D) training objects;(ii) obtaining a neural network, wherein:(1) the neural network is defined by multiple layers;(2) the multiple layers comprise at least one linear layer and at least one graph layer;(3) the linear layer operates on 2D objects;(4) an output of the linear layer comprises an input to the graph layer;(5) the graph layer is performing convolution operations onvertices of the 3D training objects;(iii) generating a model based on the training dataset, wherein:(1) the model approximates a shape diameter function;(2) the shape diameter function determines neighborhood diameters;(3) the neighborhood diameters comprise a distance from a first vertex of one or more of the 3D training objects to an antipodal vertex of the one or more 3D training objects;(4) the model is generated by iterating through the one or more 3D training objects using the neural network to converge on weights of each feature of the shape diameter function, wherein during each iteration an error based on the weights is minimized;(iv) acquiring an input mesh for a first new volumetric based 3D object;(v) using the converged weights to approximate shape diameter function values;(vi) using the approximate shape diameter function values as an input to a graph cut algorithm, wherein the graph cut algorithm generates vertex clusters that define a segmented part of the input mesh; and(vii) visually displaying or providing the segmented part.

10. The computer-implemented system of claim 10, wherein the errorbased on the weights is minimized for:(a) a predicted shape diameter function (SDF) value for a single model compared to a ground truth SDF value.

11. The computer-implemented system of claim 10, wherein the error based on the weights is minimized for:(a) a predicted signed distance function compared to a ground truth signed distance function.

12. The computer-implemented system of claim 10, wherein the error based on the weights is minimized for:(a) a gradient of the predicted signed distance function compared to a gradient of the ground truth signed distance function.

13. The computer-implemented system of claim 10, wherein the operations further comprise:(a) updating the model via machine learning (ML) wherein the model is updated after each iteration; and(b) the updated model is utilized to recognize new parts and then fed back into the ML for further updating of the model.

14. The computer-implemented system of claim 10, wherein theapproximate shape diameter function utilized as input enables user control.

15. The computer-implemented system of claim 10, wherein the model supports varying input size by downsampling one or more of the 3D training objects or other 3D input objects to a coarser structure while keeping full-resolution neighborhoods.

16. The computer-implemented system of claim 15, wherein the downsampling comprises: (a) making a graph network resolution-agnostic by using a fixed input size;(b) querying the full-resolution neighborhoods when converging on the weights; and(c) updating the weights during the converging based on contributions of the full-resolution neighborhoods.

Images