Data-driven mapping capabilities using mesh segmentation for visual effects applications

Abstract

A volumetric-based three-dimensional (3D) object is segmented. A training dataset of 3D training objects is obtained. The neural network is defined by multiple layers, including a linear layer and a graph layer. The linear layer operates on the 2D object and provides input to the graph layer, which performs convolution operations on the vertices of the 3D object. A model approximating the shape diameter function (SDF) determines a neighborhood diameter, which includes the distance from a vertex of the 3D object to the opposite vertex. The model is generated by iterating the 3D object with the neural network so that the weights of the SDF features converge. An input mesh is obtained, and the SDF values ​​are approximated using the converged weights.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit under 35 U.S.C. §119(e) of the following co-pending and commonly assigned U.S. provisional patent applications, which are incorporated herein by reference:

[0002] Provisional Application No. 63 / 481,075, filed January 23, 2023, inventor Bruno Roy, titled "Data-Driven Mapping Function for Visual Effects Applications Using Mesh Segmentation," attorney docket number 30566.0605USP1.

[0003] 1. Field of the Invention The present invention relates generally to three-dimensional (3D) image processing and polygon mesh segmentation, and more particularly to methods, apparatus, systems, and products that generate a mapping between a 3D surface and its underlying volume, which is then used as input to efficiently segment a polygon mesh. [Background technology]

[0004] 2. Description of Related Technology (Note: This application makes reference to several different documents throughout the specification, indicated by names enclosed in brackets, e.g., [x]. A listing of these various documents, ordered according to their reference names, is provided below in the section entitled "References." Each of these documents is incorporated herein by reference.)

[0005] A fundamental subject in geometry processing and computer graphics is mesh segmentation, which has many applications in 3D shape analysis, such as (i) texture mapping, (ii) 3D shape modeling, (iii) 3D shape retrieval, (iv) multiresolution and mesh compression, and (v) animation.

[0006] Mesh segmentation is the process of decomposing a polygon mesh into meaningful parts. These meaningful parts consist of subsets of vertices or faces that are semantically related and useful for further use. Users may want to segment a mesh into parts for further use for various reasons. For example, when animating a character, a user may want to first identify semantic body parts before creating the underlying animation structure in order to move vertices realistically. While several methods exist for decomposing polygon meshes, they are computationally expensive because they rely on the complexity of the mesh to perform their task. These state-of-the-art methods can provide sophisticated mesh segmentation for high-resolution models used in production, but at the expense of hours of computational time. Furthermore, users may have to adjust the resulting segmentation through a tedious and linear process of performing the decomposition with different sets of parameters.

[0007] For example, decomposition of mesh structures is typically performed by defining edge loops (i.e., pairs of vertices) that serve as boundaries between subsets of elements. While performing this task for closed-manifold meshes seems intuitive in nature, it remains difficult to generalize, considering that multiple valid solutions exist for any mesh. Furthermore, most state-of-the-art methods require a significant amount of time.

[0008] A problem with prior art mesh segmentation methods is that they are very time-consuming because they rely primarily on constraints and require an iterative solver to simultaneously optimize them. The constraint types are typically the number of elements, geometry, and topology, which guide the number of parts to be segmented, the bias toward a particular primitive shape, and the connected components of the sub-meshes, respectively. Also, some of these methods require the user to select an initial seed to reduce the number of iterations until the constraints are satisfied, increasing the total time required to accomplish the mesh segmentation task.

[0009] In recent years, machine learning has revitalized the problem of mesh segmentation by proposing more versatile approaches and providing a deeper understanding of the intrinsic semantics of shapes. While these approaches have significantly improved this challenging task, the time required to train and accommodate a wide range of different topologies remains a challenge for use with dense meshes.

[0010] To better understand the problems of the prior art, a discussion of mesh segmentation and prior art work may be useful.

[0011] 1A and 1B illustrate mesh processing according to the prior art. To process the mesh, the neighborhood at each point in the original mesh 102 / 106 is approximated. The goal is to determine whether each point is part of a different neighborhood in order to segment the mesh and generate / construct the final result 104 / 108. In the prior art, this approximation / segmentation is performed by casting rays to approximate the diameter at each point and determining whether each point is part of the same neighborhood (based on the diameter). The points are then divided into clusters, and a segmented mesh such as that shown in 104 / 108 is constructed. However, because the original mesh 102 / 106 can contain complex models (e.g., millions of polygons), such segmentation / processing can be time-consuming (e.g., it can take approximately 2-2.5 hours in this example).

[0012] There are methods that approximate segmentation using data-driven methods using machine learning, with the goal of making the original mesh 102 / 106 parameterizable and fast for class prediction (referred to herein as "part segmentation" or "part segmenting") using machine learning. Such part segmentation using machine learning can be satisfactory (e.g., parameterizable and fast) for classifying different parts and regions of an object. However, using prior art methods, it is difficult to generalize the mesh and apply the process to all types of topology / geometry. For example, monsters and shapes that are unrealistic and different from standard animals / shapes / geometry may be difficult to predict.

[0013] The following discussion will provide a more detailed description of such prior art processes to provide a better understanding of the limitations of the prior art.

[0014] Over the past two decades, mesh segmentation has been used for a variety of applications in computer graphics. This task has proven beneficial for many applications in 3D shape analysis, such as texture mapping [Sander et al. 2003], 3D shape modeling [Ji et al. 2006], 3D shape retrieval [Ferreira et al. 2010], multi-resolution and mesh compression [Maglo et al. 2011], and animation [Yuan et al. 2016]. [Rodrigues et al. 2018] provides a survey on part-based mesh segmentation. Historically, the traditional problem of mesh segmentation has been approached in many ways.

[0015] Region Growing. One of the most intuitive and straightforward approaches to segmentation is the region growing technique. The criteria used to determine whether an element should be added to a cluster primarily distinguish the variations in available region growing algorithms. Among other criteria, representative planes [Kalvin and Taylor 1996], curvature [Lavoue et al. 2005], and convexity [Chazelle et al. 1995, Sheffer 2007] have been used as criteria for clustering. Another common variant of region growing uses multiple source seeds to initiate the growing process [Eck et al. 1995, Levy et al. 2002, Sorkine et al. 2002].

[0016] Clustering. For segmentation tasks, merging operations on clusters can be essential [Attene et al. 2006, Garland et al. 2001, Gelfand and Guibas 2004, Sander et al. 2001]. Hierarchical clustering algorithms are similar to region-growing algorithms, but structurally they prioritize aggregation operations between existing clusters. Iterative clustering algorithms are said to be parametric because the number of clusters is given in advance. In contrast to traditional methods, iterative clustering methods focus on converging to an optimal segmentation given the number of clusters [Cohen-Steiner et al. 2004, Hart et al. 2000, Lloyd 1982, Shlafman et al. 2002, Wu and Kobbelt 2005].

[0017] Implicit Methods. Implicit methods of mesh segmentation focus on boundaries and correspondences between subsets of elements of the object to be segmented. Again, the main difference between these algorithms is how they define boundaries and underlying structures. To name a few, the most common methods are via curvature / contour [Lee et al. 2005, Levy et al. 2002, Mitani and Suzuki 2004], subdivision [Katz and Tal 2003, Podolak et al. 2006] (similar to hierarchical clustering), and subdivision that links shape to intrinsic surface properties [Li et al. 2001, Lien et al. 2006, Raab et al. 2004]. In fact, the shape diameter function (SDF) algorithm [Shapira et al. 2008] (generally described above with respect to Figure 1A and B) lies at the intersection of using subdivision and underlying structures. The intuitive idea of ​​a shape diameter function is to generate diameter measurements over the entire vertex neighborhood of a mesh. The resulting measurements (i.e., SDF values) are related to the medial axis transformation to provide the volumetric correspondence of the shape at any point on the surface. These per-vertex measurements are then used as thresholds in a graph cut algorithm.

[0018] Embodiments of the present invention exploit the generalization capabilities of neural networks to estimate per-vertex properties as input to the graph cut algorithm, making it highly adaptive for mesh segmentation.

[0019] Data-driven. Neural networks have been widely used for segmentation problems of images [Lai 2015], point clouds [Qi et al. 2017a], and more recently meshes [Hanocka et al. 2019]. Several interesting approaches have been proposed to address mesh segmentation problems using deep neural network algorithms, such as using convolution operators on edges [Hanocka et al. 2019], converting 3D shapes to voxel-based representations [Graham et al. 2018, Wang et al. 2017, Wu et al. 2015], and leveraging local features of point clouds [Qi et al. 2017a, b]. However, these state-of-the-art data-driven methods still struggle with generalization and often require splitting the trained model into several specific classes of objects to avoid overfitting. In contrast to these methods and [Kovacic et al. 2010], in our embodiment, we use the predicted SDF values.

number

[0020] In view of the above, prior art systems process / segment meshes using two main steps: (1) using a time-consuming machine learning process to measure the neighborhood diameter of objects at each vertex with a function to generate clusters, followed by (2) segmenting the mesh based on the clusters. In step (1), determining the distance to neighbors for each vertex is expensive. In particular, the complexity of prior art algorithms is O(vn), where v is the number of vertices and n is the maximum number of neighbors per vertex. Based on the clusters generated in step (1), mesh segmentation in step (2) can be relatively fast and simple (depending on the algorithm employed). However, it is step (1) that represents the bottleneck in prior art techniques.

[0021] In view of the above, it would be desirable to provide a system and method for quickly and efficiently determining a shape diameter function (SDF) scalar value that defines the diameter for each vertex based on its neighborhood, which values ​​can then be used for mesh segmentation. Additionally, the selection and use of an efficient mesh segmentation algorithm that utilizes the generated scalar values ​​would also be advantageous. [Prior art documents] [Non-patent literature]

[0022] [Non-Patent Document 1] Pedro V Sander, Zoe J Wood, Steven Gortler, John Snyder, and Hugues Hoppe.2003.Multi-chart geometry images.(2003). Summary of the Invention [Means for solving the problem]

[0023] Embodiments of the present invention provide: (1) generation of segmentations from predictions, (2) refinement of the generated segmentations, (3) applications and use cases, and (4) use and update of trained models. Each of these functions is summarized as follows:

[0024] (1) Generating segmented parts from predictions background Embodiments of the present invention generate segmentations from the predictions. More specifically, the output of the neural network (in embodiments of the present invention) is a scalar value of a shape diameter function (SDF) that defines the diameter for each vertex based on its neighborhood.

[0025] Steps The main steps of the segmentation are as follows: 1. Create vertex clusters based on the SDF values ​​(i.e., using the predicted SDF values ​​as input), and 2. Use a graph cut algorithm to find a smooth division.

[0026] Generate an SDF cluster In the first step, a Gaussian mixture model (GMM) is used to fit k Gaussian distributions to the histogram of SDF vertex values. The number of Gaussians (k) is used to discretize the SDF values ​​into a hierarchy of k clusters (i.e., using the expectation-maximization algorithm).

[0027] Dividing clusters using graph cuts Once the clusters have been computed in step 1, a graph cut algorithm is used to smooth the boundaries between the clusters (mesh parts). Essentially, a graph cut algorithm allows for smooth transitions between mesh parts by minimizing an energy term (i.e., equivalent to applying a threshold that determines whether a vertex should stay in a cluster or be in one of its neighboring clusters).

[0028] (2) Refinement step background The generated segmentation requires several refinement steps to provide a suitable solution. Because users are likely to work primarily with polygon meshes for various applications, it is desirable to provide a clean segmentation that matches the object's discretization (i.e., defined by its polygons).

[0029] Steps Inspired by best practices when manually segmenting meshes, embodiments of the present invention may apply the following three steps to the generated segmentation: 1. Symmetry detection, 2. Merging small regions, and 3. Appropriate boundaries.

[0030] Symmetry plane detection Objects with symmetry are typically split into halves for further processing to simplify the task. Embodiments of the present invention use a voting-based approach to identify characteristic vertices based on their local curvature values. Once these characteristic vertices are calculated, multiple candidate faces are compared to identify faces with equal total curvature values ​​on both sides.

[0031] To speed up the process, the user can also provide a custom symmetry plane.

[0032] Merging isolated clusters In this step, embodiments of the present invention review all clusters and determine whether they should be merged with other clusters. The goal of this process is to remove isolated or small clusters of vertices (e.g., small clusters within a single polygon or another cluster) from the final solution.

[0033] Appropriate boundaries In this final step, the goal is to smooth the polygonal boundaries between the segmented portions. Embodiments of the present invention detect and apply smoother paths between the segmented portions, much like an artist would. For example, embodiments can smooth out the noisy paths formed by the polygonal edges between the segmented portions (i.e., the unwanted irregular lines that separate them).

[0034] (3) Applications and usage examples There are several applications that use the approach of embodiments of the present invention. The applications can be divided into two categories: (1) general use cases, and (2) specific use cases.

[0035] Common Use Cases Such an approach is useful in data preparation for training sets, as mesh segmentation and labeling are prerequisites for supervised learning. Embodiments of the present invention can be used to decompose meshes in preparation for classification and manual labeling.

[0036] Specific Use Cases 1. Selection By identifying portions of a complex mesh, embodiments of the present invention have the unique ability to facilitate the selection of meaningful portions for further processing.

[0037] 2. Modeling Combined with the ability to select portions of a mesh, embodiments of the present invention provide a way to modify portions of a mesh (e.g., remove / scale / replace selected portions of the input mesh) in conventional modeling tools.

[0038] 3. Assigning materials The assignment of materials to objects (i.e., the definition of interactions between surfaces and light sources) can also be extended using the data-driven segmentation approach of embodiments of the present invention. With the identified meaningful semantic parts, embodiments of the present invention can facilitate the material assignment task. For example, if an axe is divided into a handle and a blade, each part can be easily assigned an appropriate material, such as wood and metal.

[0039] 4. Animation Animation often requires dividing an object into semantic parts. These parts can then be used to assign an appropriate hierarchical structure, such as an animation skeleton, to deform the mesh accordingly. Using predicted segmented parts, traditional animation methods can take advantage of mesh decomposition (in embodiments of the present invention) to optimize the underlying animation structure for control (e.g., decomposing an arm into shoulder, upper arm, lower arm, wrist, and hand, and incorporating bones for animation).

[0040] 5. UV mapping The UV mapping process can also benefit from the segmentation approach of embodiments of the present invention. UV unwrapping, the process of cutting 3D shapes and unwrapping them into 2D space, part of the UV mapping process, remains a laborious process for artists. By decomposing complex meshes into semantic parts, embodiments of the present invention simplify the unwrapping task by working with a set of simpler mesh parts (rather than working with one complex mesh).

[0041] 4. Trained Model: Use and Update Core Model Embodiments of the present invention provide what may be referred to as a "core model" as the foundation of an approach for predicting SDF values ​​for an input mesh. The core model may be composed of pre-trained weights obtained from a computationally expensive training job (several weeks of training) using training and test datasets. The provided core model is trained using volumetric terms to learn based on local mesh features to predict SDF values, and is therefore generalizable to various topologies (i.e., the model of embodiments of the present invention is resilient to noise and rigid transformations on the input object and is self-consistent). In other words, the model may be learned / updated via machine learning (ML), and after each iteration, the updated model is utilized to recognize new features, which are then fed back into the model in a feedback loop to further update the model.

[0042] Internal Model Of course, like many other learning techniques, the core model can be updated to suit specific needs: the core provides the foundational knowledge to generalize to most topologies, but can be updated with internal datasets to better suit the user's needs.

[0043] The goal may be to achieve faster and more accurate mesh segmentation solutions (compared to the state-of-the-art), and embodiments of the present invention may provide a way to update the provided core model in a "supervised" manner. The resulting model can integrate internal knowledge about multiple shapes while maintaining accuracy and avoiding overfitting. In this regard, using a cloud service may be suitable to protect user data and control how the trained weights are updated (i.e., using the hyperparameters used to generate the model, i.e., batch size, learning rate, and of course, using the network architecture described herein).

[0044] Reference will now be made to the drawings, in which like reference numbers represent corresponding parts throughout: [Brief explanation of the drawings]

[0045] [Figure 1] A and B show the processing of the mesh according to the prior art. [Figure 2] 1 illustrates a hybrid network architecture used to predict SDV values ​​in accordance with one or more embodiments of the present invention. [Figure 3-1] 1 illustrates a workflow of a neural shape diameter approach in accordance with one or more embodiments of the present invention. [Figure 3-2] 1 illustrates a workflow of a neural shape diameter approach in accordance with one or more embodiments of the present invention. [Figure 4A] 10 illustrates the results of generating a self-consistent dataset in accordance with one or more embodiments of the present invention. [Figure 4B] 1 illustrates the results of extracting suitable features for training according to one or more embodiments of the present invention. [Figure 5] 10 illustrates a subset of bash meshes and blend shapes used to generate a dataset in accordance with one or more embodiments of the present invention. [Figure 6] 1A shows generated SDF values ​​used to segment a mesh, according to one or more embodiments of the present invention; and FIG. 1B shows a close-up of the mesh in A, according to one or more embodiments of the present invention. [Figure 7] 1 illustrates a simple example application for changing an engraved logo in a 3D modeling application in accordance with one or more embodiments of the present invention. [Figure 8] We demonstrate the use of the approach of an embodiment of the present invention to facilitate 3D unfolding tasks. [Figure 9] 1 illustrates a logic flow for volumetric-based three-dimensional (3D) object segmentation in accordance with one or more embodiments of the present invention. [Figure 10]1 illustrates an exemplary hardware and software environment that may be used to implement one or more embodiments of the present invention. [Figure 11] 1 illustrates a schematic diagram of an exemplary distributed / cloud-based computer system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0046] In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration several embodiments of the present invention. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

[0047] overview Embodiments of the present invention provide a data-driven method for efficiently segmenting polygon meshes by leveraging the correspondence between surfaces and their underlying volumes. This correspondence is generated by using metrics based on medial axis transformations from shape diameter functions (SDFs). The neural SDF values ​​are then used as input to a graph cut algorithm, which provides an efficient and controllable workflow for mesh segmentation. By combining the SDFs of embodiments of the present invention with state-of-the-art graph cut algorithms, high-quality segmentations can be generated for any polygon mesh at little cost. Furthermore, embodiments of the present invention provide an intuitive way to generate different solutions for mesh segmentation by reusing the neural SDF values ​​with different parameter sets throughout the graph cut step.

[0048] Some features of embodiments of the present invention include the following. · Introduction and use of neural shape diameter functions to improve generalization of local features for mesh segmentation. A novel approach that reuses neural SDF values ​​to generate versatile and unique results, making it efficient and highly applicable. A resolution-independent approach by downsampling the input mesh while querying nearby full-resolution meshes. Use of a mesh segmentation algorithm that (i) oversamples clusters into primitive patches, (ii) encodes the presence of each patch in the cluster using a probability matrix, and (iii) performs segmentation using a graph cut algorithm. Experimental results on various applications of mesh segmentation in computer graphics workflows.

[0049] method Given a polygon mesh M, a deep neural network D predicts a mapping function that clusters vertices into meaningful groups G. The mapping function learned by network D is a shape diameter function (SDF), which represents a measure of the diameter of the object's volume in the neighborhood of each point on the surface. Using the SDF, embodiments of the present invention can process and manipulate groups of similar vertices by consistently partitioning and creating a skeleton S within the mesh. In contrast to [Kovacic et al. 2010], embodiments of the present invention avoid the computationally expensive aspects of their method by using the predicted SDF values ​​to generate the final mesh segmentation. By learning a mapping function rather than directly learning to classify mesh elements [Kalogerakis et al. 2010] (e.g., clustering vertices or edges), embodiments of the present invention are more robust for use in unknown situations.

[0050] Once the SDF values ​​are predicted, embodiments of the present invention can follow a highly efficient and parameterizable approach. A graph cut algorithm can be used to partition the resulting skeleton S associated with the input mesh M and obtained by network D. From these partitions, a set of meaningful segmentations P is created. Furthermore, because the SDF is a volume-based function, the robustness of embodiments of the present invention can be improved by including the implicit surface directly in the loss function used by network D, allowing embodiments to remain resilient to pose changes in animations of the same object in the training set.

[0051] Considering the above, as a formal definition, the traditional mesh segmentation task is stated as follows: Given a closed manifold mesh M and a set E of mesh elements (i.e., vertices, edges, or faces), a segmentation S of mesh M is a set S={M0,...,M n-1}, where M i is defined as a subset of elements e∈E.

[0052] Neural SDF In the traditional shape diameter function method, a set of sub-meshes S is defined as a threshold for the graph cut algorithm, which is the neighborhood v i Diameter measurement d from i The embodiment of the present invention uses the graph neural network EMD to obtain these SDF values ​​λ i Since the intention is to estimate the sub-mesh M i The problem of finding the constraints {C0,...,C n} is minimized. Then, the predicted SDF value

number

[0053] Constraint criteria {C0,...,C n} is the reference SDF value λ i and predicted SDF values

number

number

[0054] Model Architecture Figure 2 illustrates a hybrid network architecture used to predict SDV values ​​(e.g., during the training phase) according to one or more embodiments of the present invention. Similar to [Verma et al. 2018], the hybrid network D is composed of both a linear layer 202 and a graph layer 204. The linear layer 202 is used to warm up the network with a typical padding process, similar to that performed for an image 200, to alleviate constraints associated with various input mesh sizes. Next, the graph layer 204 is used with convolution operations to establish correspondences between filter weights and graph neighborhoods with arbitrary connectivity. Finally, a final dense regression layer 206 reprojects the predictions to an image 208.

[0055] Figure 3 illustrates the neural shape diameter approach workflow. Training 302 (also referred to as an encoder-messenger-decoder network [and / or neural network]) is performed using input 304 including meshes (e.g., both full-resolution meshes [for use in neighboring node updates by messengers] and coarse-resolution meshes [downsampled for training]). In one or more embodiments, an encoder-messenger-decoder (EMD) network 302 includes an encoder 302A, a messenger 302B, and a decoder 302C. The EMD network 302 generates predicted SDF values ​​306 that are used by a segmentation algorithm 308 (e.g., a k-fold graph cut algorithm 308 (i.e., Gaussian mixtures and clustering on a graphics processing unit [GPU]) to divide / segment the final segmentation (i.e., generating output 310).

[0056] In view of the above, the network EMD architecture 300 is essentially based on a coding-message-decoding model. The network 302 consists of two hidden MLP layers in both the encoder 302A and messenger 302B, with an output size of 128. The resulting decoder 302C output size matches the downsampled points provided as input 304 during training and inference. An embodiment of the present invention can be trained for 5 million training steps and loaded for inference using an A6000 GPU and Adam optimizer, with a learning rate of 10 when the 3 million step threshold is exceeded. -3 From 10 -5 Additionally, training / prediction 302 may include a feedback loop, which may be implemented via machine learning (ML), where new parts are recognized and fed back for further updating of the model.

[0057] Resolution-independent graph networks. Embodiments of the present invention address varying resolutions using two mechanisms: downsampling the input mesh 304 while preserving the full-resolution neighborhood, and using each node's neighborhood density as a scaling factor when updating the nodes in the messaging stage. The idea for maintaining resolution independence is to propagate the full-resolution neighborhood along the downsampled mesh. In this regard, embodiments of the present invention use SDF values

number

[0058] Graph Cut / Segmentation Algorithm 308. Predicted SDF Values

number

[0059] The partitioning algorithm of an embodiment of the present invention can consist of two steps: the first step uses soft clustering of mesh elements (faces) to compute k clusters based on SDF values, and the second step uses k-fold graph cuts to find the actual partitioning to include geometric properties of the local mesh. Note that the number of selected clusters, k, is more naturally related to the number of levels of hierarchy rather than the number of parts.

[0060] As an optional post-processing step, embodiments may use such an approach recursively as a refinement process to improve detailed part segmentation.

[0061] Datasets and Features The dataset used to train network D / EMD 302 may be generated using the AUTODESK CHARACTER GENERATOR™ tool, which applies a blend shape process to the same base mesh to generate multiple variations of the same base mesh (as shown in FIGS. 4A and 4B). In this regard, FIG. 4A illustrates the results of generating a self-consistent dataset, and FIG. 4B illustrates the results of extracting suitable features for learning, in accordance with one or more embodiments of the present invention.

[0062] The generated dataset in Figure 4A is self-consistent because it consists only of meshes with the same number of vertices. To train the network 302 to encode resolution-independent features, tessellation techniques may be used during training to provide multiple versions of the same mesh. Remeshing methods can also be used to make the network 302 resilient to consistency in the input samples 304 (i.e., by changing the positions of selected downsampled vertices).

[0063] Figure 5 shows a subset of bash meshes and blend shapes used to generate a dataset in accordance with one or more embodiments of the present invention. As shown in Figure 5, several blend shapes B502 can be used to deform the base mesh into many variations and extend the dataset. Blend shapes B502 are used for the face, f 502A (e.g., shape of ears, mouth, chin, nose, cheeks, etc., and / or features such as Elven, Bulk, Gorn, or Standard) and B for the body b 502B (e.g., shapes of arms, shoulders, chest, abdomen, legs, etc., and / or features such as muscular, thin, fat, clothing, posture, etc.) (collectively referred to as blend shapes 502). For both groups, the blend shapes used to augment the dataset are Shape B t or Feature B f By combining these blend shapes, large datasets can be generated using only a single base mesh. Furthermore, embodiments of the present invention can randomly generate different poses using animation skeletons, which can generate different SDF values ​​for the same mesh. Furthermore, to overcome these differences, an anisotropic smoothing method can be used on the SDF values ​​during training. Furthermore, embodiments of the present invention can build additional custom datasets using TURBOSQUID (TS)™ assets to evaluate this approach with production-ready content.

[0064] Additionally and / or additionally, to improve generalization, embodiments of the present invention may permute the input features used during training in a local space, making them motion invariant. In addition to making them resilient to rigid transformations, learning these features locally can improve generalization to complex inputs. The objective function of embodiments of the present invention consists of two loss terms to be minimized: L SDF and L ΦFirst, we compare the predicted value of SDF(M) with the ground truth. L SDF =|SDF'-SDF| 2 (1) Here, the reference SDF value is obtained by the method proposed by [Kovacic et al. 2010]. Next, we utilize the volume-based formulation of this method to calculate another L2 loss term, L, which focuses on minimizing the similarity in the topology of the final solution. Φ The loss function may use values ​​from the underlying implicit surface Φ and its associated gradient (approximate normal), and is expressed as follows: LΦ=|∇Φ'-∇Φ| 2 +|Φ'-Φ| 2 (2) where V is the set of vertices of the input mesh M. Finally, both loss terms are combined to provide a more resilient update during an epoch to different samples.

[0065] Experiments and Results Embodiments of the present invention have been compared with various state-of-the-art methods to determine efficiency and accuracy while maintaining user control. In the following sections, the approach of embodiments of the present invention is compared with several baselines using known segmentation datasets and the dataset of embodiments of the present invention. Computation times for several scenarios are also shown. Specific cases where embodiments of the present invention outperform state-of-the-art techniques on dense meshes are also described. Finally, several applications useful for improving traditional graphics workflows are presented.

[0066] Performance analysis Because embodiments of the present invention may focus only on mesh segmentation, performance analysis may be limited to recent methods that have shown good results in this area. To evaluate accuracy, we focus on four datasets: COSEG, HUMAN BODY SEGMENTATION (HBS)™, AUTODESK CHARACTER GENERATOR™, and TURBOSQUID™. The accuracy of embodiments of the present invention on the segmentation task was compared with POINTNET, POINTNET++, and MESHCNN. [Table 1]

[0067] As shown in Table 1, embodiments of the present invention perform similarly to MESHCNN on the COSEG (vases only) and HBS datasets, slightly outperforming POINTNET and POINTNET++. As shown in the same table, embodiments of the present invention are significantly more accurate on the ACG and TS datasets. This is not surprising, since embodiments of the present invention are trained on multi-resolution samples. By querying full-resolution neighbors and their corresponding downsampled meshes, embodiments of the present invention prove to be less sensitive to the adaptive meshing found in most 3D models contained in the TS dataset. Finally, because embodiments of the present invention aim to speed up the entire traditional mesh segmentation process, several computational time results for the presented assets are summarized. [Table 2]

[0068] As shown in Table 2, the neural-based approach of the present invention demonstrates speedup by a factor of up to 10x compared to the traditional shape diameter function method for most of the presented assets. Finally, to improve the segmentation part, the present invention implemented Gaussian mixtures and clustering on a GPU. By doing so, the present invention reduced the computational time required for these steps by nearly half.

[0069] High-density mesh It can also be demonstrated that the refinement and post-processing steps of the present invention can improve results by several points when used on dense meshes. With an error rate of less than 2% compared to the ground truth, the neural SDF approach (of the present invention) performs well on dense meshes. [Table 3]

[0070] Table 3 shows the accuracy metrics obtained for several 3D models, with performance divided into four sections: SDF value generation, segmentation, refinement, and post-processing. As shown in Table 3, the approach of an embodiment of the present invention generates highly accurate SDF values ​​for both the SAMURAI mesh and the GLADIATOR HULK mesh. Using refinement and post-processing steps can provide even greater accuracy. For example, for the GLADIAGETOR HULK mesh (FIGS. 6A and 6B), a refinement step was used to extract small additional details from a segmented portion, such as the shoulder pad, shown in close-up (FIG. 6B). In this regard, FIG. 6A shows the generated SDF values ​​used to segment a mesh, and FIG. 6B shows a close-up of that mesh, in accordance with one embodiment of the present invention.

[0071] For the SAMURAI mesh, better segmentation results were obtained after post-processing to smooth the boundaries (between each segmentation) using unwrapping heuristics (e.g., loop cuts). If the resulting segmentation was deemed satisfactory, the refinement and post-processing steps were not used.

[0072] application As mentioned above, many tasks intuitively require mesh decomposition before further processing. Mesh segmentation has many applications in 3D shape analysis, such as texture mapping [Sander et al. 2003], 3D shape modeling [Ji et al. 2006], 3D shape retrieval [Ferreira et al. 2010], multi-resolution and mesh compression [Maglo et al. 2011], and animation [Yuan et al. 2016]. [Rodrigues et al. 2018] provides a survey on part-based mesh segmentation.

[0073] Embodiments of the present invention introduce an efficient method to facilitate these various applications through efficient inferential mapping, while reducing the required computational time by a factor of 100. For example, FIG. 7 illustrates a simple example application of modifying an engraved logo in a 3D modeling application, in accordance with one or more embodiments of the present invention. Specifically, FIG. 7 illustrates a typical example in which a cube 702 with an engraved logo is decomposed into meaningful parts 704 using a neural SDF approach. After segmentation 706, parts are selected (indicated by 1 and 2) at 708 for further manipulation, such as replacement with a different logo, deformation of the selected logo, or enlarging and removing parts of the logo as shown at 710.

[0074] Embodiments of the present invention also conduct experiments to confirm how effective this approach is for UV mapping. Unwrapping parts is highly unintuitive, requiring specialized knowledge of shape topology to properly unwrap a 3D model into 2D while preserving the semantic meaning of the parts. Attempts have been made to directly address this problem by predicting UV seams using graph attention networks [Teimury et al. 2020]. However, these approaches were very limited in their training sets and largely ungeneralizable to real-world applications. Embodiments of the present invention can provide semantically meaningful parts for complex models, thereby easing the burden of the UV mapping workflow to some extent. In this way, each simpler part can be unwrapped with only a few cuts and a simple projection.

[0075] 8 illustrates the use of an approach according to an embodiment of the present invention to facilitate 3D unwrapping. As shown, a gas can 802 is first segmented into meaningful parts 804, after which automatic unwrapping is performed on the entire mesh based on prominent shape features (generating an unwrapped mesh 806), and then automatic UV mapping is performed recursively on the segmented parts to generate image 808. Based on the segmented object 806, the embodiment of the present invention clearly outperformed that generated using the automatic method in the MAYA™ application.

[0076] Logical Flow FIG. 9 illustrates a logic flow for volumetric-based three-dimensional (3D) object segmentation in accordance with one or more embodiments of the present invention.

[0077] A training dataset is obtained at step 902. The training dataset includes / consists of a plurality of volumetric-based three-dimensional (3D) training objects.

[0078] In step 904, a neural network is obtained. The neural network is defined by multiple layers. The multiple layers include at least one linear layer and at least one graph layer. The linear layer operates on 2D objects. The output of the linear layer provides input to the graph layer. The graph layer performs convolution operations on the vertices of the 3D training object.

[0079] In step 906, a model is generated based on the training dataset. The model approximates a shape diameter function. The shape diameter function determines a neighborhood diameter. The neighborhood diameter includes the distance from a first vertex of one or more 3D training objects to an opposing vertex (a point opposite the opposing vertex) of one or more 3D training objects. The model is generated by iterating through the one or more 3D training objects using a neural network to converge on weights for each feature of the shape diameter function. Furthermore, during each iteration, an error based on the weights is minimized.

[0080] The weight-based error may be minimized with respect to (a) the predicted shape diameter function (SDF) value for a single model compared to the ground truth SDF value, (b) the predicted signed distance function compared to the ground truth signed distance function, and / or (c) the gradient of the predicted signed distance function compared to the gradient of the ground truth signed distance function.

[0081] The model may also be updated (after each iteration) via machine learning (ML), which may then be used to recognize new parts and then fed back into ML for further updating of the model.

[0082] Additionally, the model can support various input sizes by downsampling one or more 3D training objects or other 3D input objects to a coarser structure while preserving the full-resolution neighborhood. Such downsampling may include (a) making the graph network resolution-independent by using a fixed input size, (b) querying the full-resolution neighborhood when converging on weights, and (c) updating the weights during convergence based on the contribution of the full-resolution neighborhood.

[0083] In step 908, an input mesh for a first new volumetric-based 3D object is obtained.

[0084] The converged weights are used to approximate a shape diameter function (SDF) value in step 910. The approximated shape diameter function can then be used as an input to enable user control.

[0085] In step 912, the approximated shape diameter function values ​​are used as input to a graph cut algorithm, which generates vertex clusters that define segmented portions of the input mesh.

[0086] In step 914, the segmented portion is visually displayed or provided.

[0087] Hardware Environment FIG. 10 illustrates an exemplary hardware and software environment 1000 (referred to as a computer-implemented system and / or a computer-implemented method) used to implement one or more embodiments of the present invention. The hardware and software environment includes a computer 1002 and may include peripheral devices. The computer 1002 may be a user / client computer, a server computer, or a database computer. The computer 1002 includes a hardware processor 1004A and / or a dedicated hardware processor 1004B (collectively referred to as processors 1004) and memory 1006, such as random access memory (RAM). The computer 1002 may be coupled to and / or integrated with other devices, including input / output (I / O) devices, such as a keyboard 1014, a cursor control device 1016 (e.g., a mouse, pointing device, pen and tablet, touch screen, multi-touch device, etc.), and a printer 1028. In one or more embodiments, computer 1002 may be coupled to or may include a portable or media viewing / listening device 1032 (e.g., an MP3 player, an IPOD, a NOOK, a portable digital video player, a handheld device, a personal digital assistant, etc.) In yet another embodiment, computer 1002 may include a multi-touch device, a mobile phone, a gaming system, an Internet-enabled television, a television set-top box, or other Internet-enabled device operating on a variety of platforms and operating systems.

[0088] In one embodiment, computer 1002 operates by hardware processor 1004A executing instructions defined by computer programs 1010 (e.g., computer-aided design [CAD] applications) under the control of operating system 1008. Computer programs 1010 and / or operating system 1008 may be stored in memory 1006 and interface with users and / or other devices to accept inputs and commands and provide outputs and results based on these inputs and commands and the instructions defined by computer programs 1010 and operating system 1008.

[0089] The output / results may be presented on a display 1022 or provided to another device for display or further processing or action. In one embodiment, the display 1022 includes a liquid crystal display (LCD) having multiple individually addressable liquid crystals. Alternatively, the display 1022 may include a light-emitting diode (LED) display in which clusters of red, green, and blue diodes are simultaneously driven to form full-color pixels. Each liquid crystal or pixel of the display 1022 changes to an opaque or translucent state to form part of an image on the display in response to data or information generated by the processor 1004 as a result of application of instructions of the computer program 1010 and / or operating system 1008 to inputs and commands. The image may be provided through a graphical user interface (GUI) module 1018. Although the GUI module 1018 is shown as a separate module, the instructions for performing GUI functions may reside within or be distributed within the operating system 1008, the computer program 1010, or may be implemented by dedicated memory and processor.

[0090] In one or more embodiments, display 1022 is integrated with / within computer 1002 and includes a multi-touch device having a touch-sensitive surface (e.g., a track pod or touch screen) capable of recognizing contact at two or more points on the surface. Examples of multi-touch devices include mobile devices (e.g., an IPHONE®, NEXUS S, DROID device, etc.), tablet computers (e.g., an IPAD®, HP TOUCHPAD, SURFACE device, etc.), portable / handheld game / music / video player / console devices (e.g., an IPOD TOUCH®, MP3 player, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touch tables, and walls (e.g., where an image is projected through acrylic and / or glass and the image is backlit with LEDs), etc.

[0091] Some or all of the operations performed by computer 1002 pursuant to the instructions of computer program 1010 may be implemented in special purpose processor 1004B. In this embodiment, some or all of the instructions of computer program 1010 may be implemented via firmware instructions stored in read-only memory (ROM), programmable read-only memory (PROM), or flash memory within special purpose processor 1004B or within memory 1006. Special purpose processor 1004B may also be hardwired through circuit design to perform some or all of the operations for implementing the present invention. Furthermore, special purpose processor 1004B may be a hybrid processor including dedicated circuits for performing a subset of functions and other circuits for performing more general-purpose functions, such as responding to instructions of computer program 1010. In one embodiment, special purpose processor 1004B is an application-specific integrated circuit (ASIC).

[0092] The computer 1002 may implement a compiler 1012 that enables an application or computer program 1010 written in a programming language such as C, C++, assembly, SQL, PYTHON®, PROLOG, MATLB, RUBY, RAILS, HASKELL, or other language to be converted into code readable by the processor 1004. Alternatively, the compiler 1012 may be an interpreter that executes instructions / source code directly, converts source code into an intermediate representation that is executed, or executes stored pre-compiled code. Such source code may be written in a variety of programming languages, such as JAVA®, JAVASCRIPT®, PERL, BASIC, etc. Upon completion, the application or computer program 1010 accesses data received from I / O devices and stored in the memory 1006 of the computer 1002 and operates based on the relationships and logic generated using the compiler 1012.

[0093] The computer 1002 also optionally includes an external communications device such as a modem, satellite link, Ethernet card, or other device for receiving input from and providing output to other computers 1002.

[0094] In one embodiment, the instructions implementing the operating system 1008, the computer program 1010, and the compiler 1012 are tangibly embodied in a non-transitory computer-readable medium, such as, for example, a data storage device 1020, which may include one or more fixed or removable data storage devices, such as a zip drive, a floppy disk drive 1024, a hard drive, a CD-ROM drive, a tape drive, etc. Additionally, the operating system 1008 and the computer program 1010 are comprised of computer program 1010 instructions that, when accessed, read, and executed by the computer 1002, cause the computer 1002 to perform the steps necessary to implement and / or utilize the present invention, or to load the program of instructions into memory 1006, thereby generating special-purpose data structures that cause the computer 1002 to operate as a specially programmed computer that performs the method steps described herein. The computer program 1010 and / or operating instructions may also be tangibly embodied in memory 1006 and / or data communication device 1030, thereby forming a computer program product or article of manufacture in accordance with the present invention. Accordingly, as used herein, the terms "article of manufacture," "program storage device," and "computer program product" are intended to encompass a computer program accessible from any computer-readable device or media.

[0095] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used in conjunction with computer 1002 .

[0096] 11 schematically illustrates a typical distributed / cloud-based computer system 1100 connecting a client computer 1102 to a server computer 1106 using a network 1104. A typical combination of resources may include the network 1104, which may include the Internet, a LAN (local area network), a WAN (wide area network), an SNA (systems network architecture) network, etc., the client 1102 (as shown in FIG. 10), which may be a personal computer or workstation, and the server 1106 (as shown in FIG. 10), which may be a personal computer, workstation, minicomputer, or mainframe. However, it should be noted that a different network may be used to connect the client 1102 and the server 1106 according to embodiments of the present invention, such as a cellular network (e.g., GSM [global system for mobile communications]), a satellite-based network, or any other type of network.

[0097] A network 1104, such as the Internet, connects the client 1102 to the server computer 1106. The network 1104 may provide connectivity and communication between the client 1102 and the server 1106 using Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc. Furthermore, in a cloud-based computing system, resources (e.g., storage devices, processors, applications, memory, infrastructure, etc.) in the client 1102 and the server computer 1106 may be shared by the client 1102, the server computer 1106, and users over one or more networks. Resources may be shared by multiple users and dynamically reallocated based on demand. In this regard, cloud computing is sometimes referred to as a model that enables access to a shared pool of configurable computing resources.

[0098] The client 1102 may execute a client application or web browser and communicate with a server computer 1106 executing a web server 1110. Such a web browser is typically a program such as MICROSOFT INTERNET EXPLORER / EDGE, MOZILLA FIREFOX, OPERA, APPLE SAFARI, or GOOGLE CHROME. Additionally, software executed on the client 1102 may be downloaded to the client computer 1102 from the server computer 1106 and installed as a web browser plug-in or an ACTIVEX control. Thus, the client 1102 may utilize ACTIVEX Component / Component Object Model (COM) or Distributed COM (DCOM) components to provide a user interface on the client's 1102 display. The web server 1110 is typically a program such as MICROSOFT'S INTERNET INFORMATION SERVER.

[0099] Web server 1110 may host Active Server Page (ASP) or Internet Server Application Programming Interface (ISAPI) applications 1112, which may execute scripts. The scripts invoke objects (called business objects) that perform business logic. The business objects then manipulate data in a database 1116 through a database management system (DBMS) 1114. Alternatively, database 1116 may be part of or directly connected to client 1102, instead of communicating / retrieving information from database 1116 over network 1104. When developers encapsulate business functionality in objects, the system is sometimes referred to as a Component Object Model (COM) system. Thus, scripts running on web server 1110 (and / or application 1112) invoke COM objects that implement the business logic. Additionally, the server 1106 may utilize MICROSOFT'S TRANSACTION SERVER (MTS) to access the necessary data stored in the database 1116 through interfaces such as ADO (Active Data Objects), OLE DB (Object Linking and Embedding Database), or ODBC (Open Database Connectivity).

[0100] Generally, all of these components 1100-1116 include logic and / or data embodied in or retrievable from a device, medium, signal, or carrier, such as a data storage device, a data communication device, a remote computer, or a device coupled to a computer via a network or other data communication device, and this logic and / or data can be read, executed, and / or interpreted to perform the steps necessary to implement and / or use the present invention.

[0101] Although reference is made herein to the terms "user computer," "client computer," and / or "server computer," it is understood that such computers 1102 and 1106 are interchangeable and may further include thin client devices having limited or full processing capabilities, portable devices such as mobile phones, laptop computers, pocket computers, multi-touch devices, and / or any other device with suitable processing, communication, and input / output capabilities.

[0102] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used in conjunction with computers 1102 and 1106. Embodiments of the present invention are implemented as a software / CAD application on client 1102 or server computer 1106. Additionally, as noted above, client 1102 or server computer 1106 may include a thin client device or a handheld device with a multi-touch based display.

[0103] conclusion This concludes the description of the preferred embodiment of the present invention. Below, we describe several alternative embodiments for achieving the present invention. For example, the present invention can be used with any type of computer, such as a mainframe, minicomputer, or personal computer, or with computer configurations such as a time-shared mainframe, a local area network, or a standalone personal computer.

[0104] In summary, embodiments of the present invention provide a neuromorphic diameter function approach that significantly speeds up high-quality mesh segmentation while maintaining adaptability and controllability. Embodiments of the present invention can generate multiple alternative segmentation solutions for the same mesh by using the generated SDF values ​​to adjust the splitting parameters of a k-fold graph cut algorithm.

[0105] Given the fact that production meshes often use adaptive tessellation to optimize memory on GPUs based on the local complexity of the topology, graph cut algorithms (unlike images) have proven less effective than deep neural networks for segmenting such irregular structures, where the number of neighbors per vertex varies widely. Embodiments of the present invention partially mitigate this by using layered networks to query the full-resolution mesh, but the current network output can be used as input to a single non-binary classifier.

[0106] The foregoing description of preferred embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

[0107] References [Attene et.al.2006]Marco Attene,Bianca Falcidieno, and Michela Spagnuolo.2006.Hierarchical mesh segmentation based on fitting primitives.The Visual Computer 22,3(2006),181-193.

[0108] [Chazelle et.al.1995]Bernard Chazelle,David P Dobkin,Nadia Shouraboura, and Ayellet Tal.1995.Strategies for polyhedral surface decomposition:An experimental study.In Proceedings of the eleventh annual symposium on Computational geometry.297-305.

[0109] [Cohen-Steiner et.al.2004]David Cohen-Steiner,Pierre Alliez, and Mathieu Desbrun.2004.Variational shape approximation.In ACM SIGGRAPH 2004 Papers.905-914.

[0110] [Eck et.al.1995]Matthias Eck,Tony DeRose,Tom Duchamp,Hugues Hoppe,Michael Lounsbery, and Werner Stuetzle.1995.Multiresolution analysis of arbitrary meshes.In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques.173-182.

[0111] [Ferreira et.al.2010]Alfredo Ferreira,Simone Marini,Marco Attene,Manuel J Fonseca,Michela Spagnuolo,Joaquim A Jorge, and Bianca Falcidieno.2010.Thesaurus-based 3D object retrieval with part-in-whole matching.International Journal of Computer Vision 89,2(2010),327-347.

[0112] [Garland et.al.2001]Michael Garland,Andrew Willmott, and Paul S Heckbert.2001.Hierarchical face clustering on polygonal surfaces.In Proceedings of the 2001 symposium on Interactive 3D graphics.49-58.

[0113] [Gelfand et.al.2004]Natasha Gelfand, and Leonidas J Guibas.2004.Shape segmentation using local slippage analysis.In Proceedings of the 2004 Eurographics / ACM SIGGRAPH symposium on Geometry processing.214-223.

[0114] [Graham et.al.2018]Benjamin Graham,Martin Engelcke, and Laurens Van Der Maaten.2018.3d semantic segmentation with submanifold sparse convolutional networks.In Proceedings of the IEEE conference on computer vision and pattern recognition.9224-9232.

[0115] [Hanocka et.al.2019]Rana Hanocka,Amir Hertz,Noa Fish,Raja Giryes,Shachar Fleishman, and Daniel Cohen-Or.2019.Meshcnn:a network with an edge.ACM Transactions on Graphics(TOG)38,4(2019),1-12.

[0116] [Hart et.al.2000]Peter E Hart,David G Stork, and Richard O Duda.2000.Pattern classification.Wiley Hoboken.

[0117] [Zhongping et.al.2006]Zhongping Ji,Ligang Liu,Zhonggui Chen, and Guojin Wang.2006.Easy mesh cutting.In Computer Graphics Forum,Vol.25.Wiley Online Library,283-291.

[0118] [Kalogerakis et.al.2010]Evangelos Kalogerakis,Aaron Hertzmann, and Karan Singh.2010.Learning 3D mesh segmentation and labeling.In ACM SIGGRAPH 2010 papers.1-12.

[0119] [Kalvin et.al.1996]Alan D Kalvin and Russell H Taylor.1996.Superfaces:Polygonal mesh simplification with bounded error.IEEE Computer Graphics and Applications 16,3(1996),64-77.

[0120] [Katz et.al.2003]Sagi Katz and Ayellet Tal.2003.Hierarchical mesh decomposition using fuzzy clustering and cuts.ACM transactions on graphics(TOG)22,3(2003),954-961.

[0121] [Kovacic et.al.2010]Maurizio Kovacic,Fabio Guggeri,Stefano Marras, and Riccardo Scateni.2010.Fast approximation of the shape diameter function.In Proceedings Workshop on Computer Graphics,Computer Vision and Mathematics(GraVisMa),Vol.5.

[0122] [Lai 2015]Matthew Lai.2015.Deep learning for medical image segmentation.arXiv preprint arXiv:1505.02000(2015).

[0123] [Lavoue et.al.2005]Guillaume Lavoue,Florent Dupont, and Atilla Baskurt.2005.A new CAD mesh segmentation method,based on curvature tensor analysis.Computer-Aided Design 37,10(2005),975-987.

[0124] [Lee et.al.2005]Yunjin Lee,Seungyong Lee,Ariel Shamir,Daniel Cohen-Or, and Hans-Peter Seidel.2005.Mesh scissoring with minima rule and part salience.Computer Aided Geometric Design 22,5(2005),444-465.

[0125] [Levy et.al.2002]Bruno Levy,Sylvain Petitjean,Nicolas Ray, and Jerome Maillot.2002.Least squares conformal maps for automatic texture atlas generation.ACM transactions on graphics(TOG)21,3(2002),362-371.

[0126] [Li et.al.2001]Xuetao Li,Tong Wing Woon,Tiow Seng Tan, and Zhiyong Huang.2001.Decomposing polygon meshes for interactive applications.In Proceedings of the 2001 symposium on Interactive 3D graphics.35-42.

[0127] [Lien et.al.2006]Jyh-Ming Lien,John Keyser, and Nancy M Amato.2006.Simultaneous shape decomposition and skeletonization.In Proceedings of the 2006 ACM symposium on Solid and physical modeling. 219-228.

[0128] [Lloyd 1982]Stuart Lloyd.1982.Least squares quantization in PCM.IEEE transactions on information theory 28,2(1982),129-137.

[0129] [Maglo et.al.2011]Adrien Maglo,Ian Grimstead, and Celine Hudelot.2011.Cluster-based random accessible and progressive lossless compression of colored triangular meshes for interactive visualization.In Proceedings of computer graphics international.

[0130] [Mitani et.al.2004]Jun Mitani and Hiromasa Suzuki.2004.Making papercraft toys from meshes using strip-based approximate unfolding.ACM transactions on graphics(TOG)23,3(2004),259-263.

[0131] [Podolak et.al.2006]Joshua Podolak,Philip Shilane,Aleksey Golovinskiy,Szymon Rusinkiewicz, and Thomas Funkhouser.2006.A planar-reflective symmetry transform for 3D shapes.In ACM SIGGRAPH 2006 Papers.549-559.

[0132] [Qi et.al.2017]Charles R Qi,Hao Su,Kaichun Mo, and Leonidas J Guibas.2017a.Pointnet:Deep learning on point sets for 3d classification and segmentation.In Proceedings of the IEEE conference on computer vision and pattern recognition.652-660.

[0133] [Qi et.al.2017]Charles Ruizhongtai Qi,Li Yi,Hao Su, and Leonidas J Guibas.2017b.Pointnet++:Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems 30(2017).

[0134] [Raab et.al.2004]R Raab,C Gotsman, and A Sheffer.2004.Virtual woodwork:Generating bead figures from 3d models.International Journal on Shape Modeling 10,1(2004),1-30.

[0135] [Rodrigues et.al.2018]Rui SV Rodrigues,Jose FM Morgado, and Abel JP Gomes.2018.Part-based mesh segmentation:a survey.In Computer Graphics Forum,Vol.37.Wiley Online Library,235-274.

[0136] [Sander et.al.2001]Pedro V Sander,John Snyder,Steven J Gortler, and Hugues Hoppe.2001.Texture mapping progressive meshes.In Proceedings of the 28th annual conference on Computer graphics and interactive techniques.409-416.

[0137] [Sander et.al.2003]Pedro V Sander,Zoe J Wood,Steven Gortler,John Snyder, and Hugues Hoppe.2003.Multi-chart geometry images.(2003).

[0138] [Shapira et.al.2008]Lior Shapira,Ariel Shamir, and Daniel Cohen-Or.2008.Consistent mesh partitioning and skeletonisation using the shape diameter function.The Visual Computer 24,4(2008),249-259.

[0139] [Kraevoy et.al.2007]Vladislav Kraevoy Dan Julius Alla Sheffer.2007.Shuffler:Modeling with interchangeable parts.Visual Computer journal(2007).

[0140] [Shlafman et.al.2002]Shymon Shlafman,Ayellet Tal, and Sagi Katz.2002.Metamorphosis of polyhedral surfaces using decomposition.In Computer graphics forum,Vol.21.Wiley Online Library,219-228.

[0141] [Sorkine et.al.2002]Olga Sorkine,Daniel Cohen-Or,Rony Goldenthal, and Dani Lischinski.2002.Boundeddistortion piecewise mesh parameterization.In IEEE Visualization,2002.VIS 2002.IEEE,355-362.

[0142] [Teimury et.al.2020]Fatemeh Teimury,Bruno Roy,Juan Sebastian Casallas,David MacDonald, and Mark Coates.2020.GraphSeam:Supervised Graph Learning Framework for Semantic UV Mapping.arXiv preprint arXiv:2011.13748(2020).

[0143] [Wang et.al.2017]Peng-Shuai Wang,Yang Liu,Yu-Xiao Guo,Chun-Yu Sun, and Xin Tong.2017.O-cnn:Octree-based convolutional neural networks for 3d shape analysis.ACMTransactions On Graphics(TOG)36,4(2017),1-11.

[0144] [Wu et.al.2005]Jianhua Wu and Leif Kobbelt.2005.Structure Recovery via Hybrid Variational Surface Approximation..In Comput.Graph.Forum,Vol.24.277-284.

[0145] [Wu 2 et.al.015]ZhirongWu,Shuran Song,Aditya Khosla,Fisher Yu,Linguang Zhang,Xiaoou Tang, and Jianxiong Xiao.2015.3d shapenets:A deep representation for volumetric shapes.In Proceedings of the IEEE conference on computer vision and pattern recognition.1912-1920.

[0146] [Yuan et.al.2016]Qing Yuan,Guiqing Li,Kai Xu,Xudong Chen, and Hui Huang.2016.Space-time cosegmentation of articulated point cloud sequences.In Computer Graphics Forum,Vol.35.Wiley Online Library,419-429.

Claims

1. 1. A computer-implemented method for volumetric-based three-dimensional (3D) object segmentation, comprising: (a) acquiring a training dataset, the training dataset including a plurality of volumetric-based three-dimensional (3D) training objects; (b) obtaining a neural network, (i) the neural network is defined by a plurality of layers; (ii) the plurality of layers includes at least one linear layer and at least one graph layer; (iii) the linear layer operates on 2D objects; (iv) the output of the linear layer comprises the input to the graph layer; (v) the graph layer performs a convolution operation on the vertices of the 3D training object; (c) generating a model based on the training data set, (i) the model approximates a shape diameter function; (ii) the shape diameter function determines a neighborhood diameter; (iii) the neighborhood diameter comprises a distance from a first vertex of the one or more 3D training objects to an opposing vertex of the one or more 3D training objects; (iv) generating the model by iterating the one or more 3D training objects with the neural network such that weights of each feature of the shape diameter function converge, and during each iteration, an error based on the weights is minimized; (d) obtaining an input mesh for a first new volumetric-based 3D object; and (e) using the converged weights to approximate a shape diameter function value; (f) using the approximated shape diameter function values ​​as input to a graph cut algorithm, the graph cut algorithm generating vertex clusters that define segmented portions of the input mesh; (g) visually displaying or providing the segmented portions.

2. The error based on the weights is The computer-implemented method of claim 1 , wherein: (a) predicted shape diameter function (SDF) values ​​for a single model compared to ground truth SDF values ​​are minimized.

3. The error based on the weights is The computer-implemented method of claim 1 , wherein: (a) a predicted signed distance function compared to a ground truth signed distance function is minimized.

4. The error based on the weights is The computer-implemented method of claim 1 , wherein: (a) the gradient of the predicted signed distance function compared to the gradient of the ground truth signed distance function is minimized.

5. (a) updating the model via machine learning (ML), wherein the model is updated at each iteration; 2. The computer-implemented method of claim 1, further comprising: (b) utilizing the updated model to recognize new portions and then feeding back to the ML for further updating of the model.

6. The computer-implemented method of claim 1 , wherein the approximated shape diameter function utilized as an input allows for user control.

7. 10. The computer-implemented method of claim 1, wherein the model supports a variety of input sizes by downsampling one or more of the 3D training objects or other 3D input objects to a coarser structure while maintaining near full resolution.

8. The downsampling step comprises: (a) Making graph networks resolution independent by using a fixed input size; (b) querying the full-resolution neighborhood upon convergence of the weights; (c) upon said convergence, updating said weights based on the contributions of said full-resolution neighbors.

9. 1. A computer-implemented system for volumetric-based three-dimensional (3D) object segmentation, comprising: (a) a computer having a memory; (b) a processor running on said computer; (c) the memory storing a set of instructions that, when executed by the processor, cause the processor to perform an operation, the operation being: (i) acquiring a training dataset, the training dataset including a plurality of volumetric-based three-dimensional (3D) training objects; (ii) obtaining a neural network, (1) the neural network is defined by a plurality of layers; (2) the plurality of layers includes at least one linear layer and at least one graph layer; (3) the linear layer operates on 2D objects; (4) the output of the linear layer includes the input to the graph layer; (5) the graph layer performs a convolution operation on the vertices of the 3D training object; and (iii) generating a model based on the training data set, (1) the model approximates a shape diameter function; (2) the shape diameter function determines a neighborhood diameter; (3) the neighborhood diameter comprises a distance from a first vertex of one or more of the 3D training objects to an opposing vertex of the one or more 3D training objects; (4) generating the model by iterating the one or more 3D training objects with the neural network such that weights of each feature of the shape diameter function converge, and during each iteration, an error based on the weights is minimized; (iv) obtaining an input mesh for a first new volumetric-based 3D object; and (v) using the converged weights to approximate a shape diameter function value; (vi) using the approximated shape diameter function values ​​as input to a graph cut algorithm, the graph cut algorithm generating vertex clusters that define segmented portions of the input mesh; (vii) visually displaying or providing the segmented portions.

10. The error based on the weights is The computer-implemented system of claim 10 , wherein: (a) predicted shape diameter function (SDF) values ​​for a single model compared to ground truth SDF values ​​are minimized.

11. The error based on the weights is The computer-implemented system of claim 10 , wherein: (a) a predicted signed distance function compared to a ground truth signed distance function is minimized.

12. The error based on the weights is The computer-implemented system of claim 10 , wherein: (a) the gradient of the predicted signed distance function compared to the gradient of the ground truth signed distance function is minimized.

13. The operation is (a) updating the model via machine learning (ML), wherein the model is updated at each iteration; 11. The computer-implemented system of claim 10, further comprising: (b) utilizing the updated model to recognize new portions and then feeding back to the ML for further updating of the model.

14. The computer-implemented system of claim 10 , wherein the approximated shape diameter function utilized as an input allows for user control.

15. 11. The computer-implemented system of claim 10, wherein the model supports a variety of input sizes by downsampling one or more of the 3D training objects or other 3D input objects to a coarser structure while maintaining near full resolution.

16. The downsampling step comprises: (a) Making graph networks resolution independent by using a fixed input size; (b) querying the full-resolution neighborhood upon convergence of the weights; (c) updating the weights based on the contributions of the full-resolution neighborhood during the convergence.

Images