Automated processing of dental scans using geometric deep learning

Abstract

Machine learning or geometric deep learning is applied to various dental processes and solutions. In particular, generative adversarial networks apply machine learning to smile design (finish smile), appliance rendering, scan cleanup, restorative appliance design, crown and bridge design, and virtual debonding. Vertex and edge classification applies machine learning to gum and tooth detection, tooth type segmentation, and brackets and other orthodontic hardware. Regression applies machine learning to coordinate systems, diagnoses, case complexity, and treatment duration prediction. Autoencoders and clustering apply machine learning to group doctors or technicians and preferences.

Description

【Background Art】 【0001】 Machine learning is used to automate and improve processes and various tasks in various industries and fields. In the dental field, including orthodontics, many processes and tasks are performed manually and may rely on user feedback or interaction to complete them. Machine learning can be used in the dental field to automate, partially automate, or improve such processes and tasks. 【Summary of the Invention】 【0002】 In embodiments, machine learning is used for various dental processes and solutions. In particular, in embodiments of adversarial generative networks, machine learning is applied to smile design (final smile), appliance rendering, scan cleanup, restoration appliance design, crown and bridge design, and virtual debonding. In embodiments of vertex and edge classification, machine learning is applied to gum and tooth detection, tooth type segmentation, and brackets and other orthodontic hardware. In embodiments of regression, machine learning is applied to coordinate systems, diagnosis, case complexity, and treatment period prediction. In embodiments of autoencoders and clustering, machine learning is applied to grouping dentists (or technicians) and preferences. 【Brief Description of the Drawings】 【0003】 [Figure 1] It is a diagram of a system that receives and processes a digital model based on a 3D scan. [Figure 2] It shows the raw version (left) of the same model and the version (right) that has been cleaned up and the gums have been clipped / bridged. [Figure 3] A flowchart for model development / training and model deployment is provided. [Figure 4] It shows pathological features including a root abfraction (left) and a chipped tooth (right). [Figure 5]This is an overview of the training pipeline. [Figure 6] This is a method workflow for model inference (deployment), where the generator produces given clean and unclean data points. [Figure 7] This is an overview of the training pipeline. [Figure 8] This is a method workflow for model inference (deployment), where the generator produces given clean and unclean data points. [Figure 9] Here are six examples of "good" division surfaces. [Figure 10] Here are six examples of "bad" or faulty split surfaces generated by temporarily corrupting specific lines of automated code. [Figure 11] This is the use of NN to distinguish between mold parting surfaces that pass and those that fail. [Figure 12] This is an operational case for the purpose of regression testing one of the automation code modules (for example, the code for generating the subdivision surface). [Figure 13] This is an operational case for determining whether a segmented surface is suitable for use in dental restorative appliances, within the context of a manufacturing system. [Figure 14] This is the use of NN to distinguish between mold parting surfaces that pass and those that fail. [Figure 15] This involves using a neural network (NN) to distinguish between acceptable and unacceptable mold parting surfaces in the context of regression testing. [Figure 16] This is an operational case for determining whether a segmented surface is suitable for use in dental restorative appliances, within the context of a production system. [Figure 17] The image shows 2D images of correct (left) and incorrect (right) center clip placement. [Figure 18] This is an operation in which 3D mesh components are created. [Figure 19]Figure 19 (left) shows a tooth that is precisely divided into two by a division plane, while the right side shows a tooth that is not precisely divided into two by a division plane. [Figure 20] The negative sample (left image) shows the same type of data sample as in Figure 19, except that the segmentation plane has penetrated excessively on the lingual side. [Figure 21] This is a more detailed use of Figure 18 in the context of the mold parting surface. [Figure 22] This is the verification process. [Figure 23] This figure illustrates the process shown in Figure 22 based on images. [Figure 24] This shows 30 diagrams of the left upper lateral incisor (tooth 10) divided into two by a dividing plane. [Figure 25] This shows 30 diagrams of the upper right canine tooth (tooth 6) precisely divided into two by a dividing plane. [Figure 26] This is the segmentation process. [Figure 27] Examples of segmentation results for several upper and lower dental arches are shown. [Figure 28] This is a training pipeline for segmentation. [Figure 29] This is a test pipeline for segmentation. [Figure 30] This shows the prediction of the tooth's coordinate system. [Figure 31] This is a training pipeline for predicting coordinate systems. [Figure 32] This is a test pipeline for predicting coordinate systems. [Figure 33] This shows the prediction for the second molar (UNS=1) in the upper right dental arch. [Figure 34] This is the process of grouping providers and preferences using machine learning. [Modes for carrying out the invention] 【0004】 Use geometric deep learning (GDL) or machine learning methods to process dental scans for several dental and orthodontic processes and tasks. By using GDL, for example, these processes and tasks can be automated, partially automated, or improved. The following are exemplary uses of GDL in dental and orthodontic applications, in addition to the embodiments described in the following sections. 【0005】 Use of transfer learning: In this method, transfer learning can be used in situations where there is insufficient good training data. In this method, a pre-trained model for a type of tooth for which sufficient training data exists is used as a base model, and its weights are fully or partially fine-tuned to create a new model suitable for operating on the first (data-deficient) type of tooth. 【0006】 Use of other modalities: GDL can be used with multi-view 2D projections, for example, a multi-view convolutional neural network (MVCNN). 【0007】 Use multiple modalities together: In this method, a pipeline can be created that uses all or some of the modality machines to create a hybrid pipeline. This pipeline can take in data from multiple modalities. 【0008】 Figure 1 shows a diagram of a system 10 for receiving and processing digital 3D models based on intraoral three-dimensional (3D) scans or scans of physical models using GDL. System 10 includes a processor 20 for receiving digital 3D models of teeth (12) from intraoral 3D scans or scans of dental impressions. System 10 may also include an electronic display device 16, such as a liquid crystal display (LCD) device, and an input device 18 for receiving user commands or other information. Systems for generating digital 3D images or models based on sets of images from multiple views are disclosed in U.S. Patents No. 7,956,862 and No. 7,605,817, both of which are incorporated herein by reference as if fully described. These systems may use an intraoral scanner to obtain digital images from multiple views of a tooth or other intraoral structure, and these digital images are processed to generate a digital 3D model representing the scanned tooth. System 10 may be implemented using, for example, a desktop, notebook, or tablet computer. System 10 may receive 3D scans locally or remotely over a network. 【0009】 I. Generative-Adversarial Networks (GANs) These embodiments include, for example, the following: 【0010】 Repairing the final smile design: Use a generative adversarial network to create a 3D mesh of the final smile based on the initial 3D mesh. 【0011】 Reconstructive orthotic design: Use a Generative Adversarial Network (GAN) to create a reconstructive orthotic based on a 3D mesh of the final smile design. 【0012】 Crown and bridge design: Using GANs, we provide the ability to show how orthotics (braces, brackets, etc.) will look during the treatment process. 【0013】 Virtual debonding: Using GANs, a scanned dental arch mesh without orthotics (brackets, retainers, or other hardware) is generated based on an initial 3D scan of the dental arch including orthotics. Alternatively, a machine learning segmentation module is used to identify brackets, retainers, or other hardware present within the scanned dental arch. The orthotics can then be removed from the scanned mesh using GANs or 3D meshing. 【0014】 A. Mesh Cleanup These embodiments include methods for automated 3D mesh cleanup for dental scans. There are three main approaches: a 3D mesh processing method, a deep learning method, and a combined method employing several 3D mesh processing elements and several deep learning elements. 【0015】 Section 1: 3D Mesh Processing for Dental Scan Cleanup This method receives raw (pre-cleanup) digital dental models generated by various intraoral and laboratory scanners, each possessing diverse 3D mesh characteristics. The method utilizes standard domain-independent 3D mesh restoration techniques to ensure specific mesh quality, avoiding meshing issues in subsequent operations. It also employs custom orthodontic / dental domain-specific algorithms, such as model base removal and partial gingival clipping / bridging, as shown in Figure 2, which illustrates both the raw (left image) and cleaned-up versions of the same model with gingival clipping / bridging (right image). These automated cleanup results can be further refined through manual interaction using commercially available or custom mesh manipulation software. Mesh cleanup, therefore, may include modifying the mesh or model by, for example, removing features, adding features, or performing other cleanup techniques. 【0016】 Section 2: Deep learning for dental scan cleanup. As more data is acquired, machine learning methods, and deep learning methods in particular, begin to perform comparably to or even surpass the performance of explicitly programmed methods. Deep learning methods have a significant advantage in that, throughout the training process, they can infer some useful features directly from the data using a combination of several nonlinear functions of higher-dimensional latent or hidden features, thus eliminating the need to manually create features. When attempting to solve mesh cleanup problems, it may be desirable to work directly on the 3D mesh using methods such as PointNet, PointCNN, MeshCNN, and FeaStNetre. 【0017】 Deep learning algorithms have two main development steps: 1) training the model, and 2) deploying the model. 【0018】 Model training utilizes multiple raw (pre-cleanup) and cleaned-up digital 3D models from historical case data. These raw or partially cleaned-up 3D models are input into a deep learning framework designed to generate predicted improved and cleaned-up 3D models. Optionally, data augmentation can be applied to the input model to increase the amount of data input to the deep learning model. Some data augmentation techniques include mesh translation and rotation, uniform and non-uniform scaling, edge flipping, and adding random noise to mesh vertices. The model is then trained through a process of iteratively adjusting the set of weights to minimize the difference between the predicted cleaned-up digital 3D model and the actual cleaned-up digital 3D model. Finally, the trained model is evaluated by generating cleaned-up meshes from a spare set of cases not used during training and comparing these generated 3D meshes to the cleaned-up meshes from the actual case data. 【0019】 In the model deployment phase, the trained model developed during model training is used. The trained model takes a raw digital 3D model of a new case it has never seen before as input and generates a cleaned-up 3D model of that case. 【0020】 Figure 3 provides flowcharts of the following model development / training and model deployment methods, which are further described herein. 【0021】 Model development / training: 1. Input: 3D models (22) of past case data. 2. Optional data expansion (24). 3. Train the deep learning model (26). 4. Evaluate the generated cleaned 3D mesh against the ground truth data (28). 【0022】 Model development: 1. Input: Digital 3D model of the new case (30). 2. Run the trained deep learning model (32). 3. Generate the proposed cleaned-up 3D mesh (34). 【0023】 Section 3: Combinatorial methods (3D mesh processing + deep learning). As described in Section 2, it is possible to use deep learning to generate a cleaned mesh from an input scan without explicitly programmed 3D meshing steps. However, some mesh cleanup operations (e.g., filling holes, removing triangular intersections, and removing isolated areas) are well-defined mesh operations and are likely to be performed more effectively using 3D meshing methods rather than deep learning. Alternatively, this method can implement a combined approach that uses deep learning in place of some, but not all, of the meshing steps described in Section 1. For example, deep learning can be used to identify the gingival margin in the mesh and remove excess material below the gingival margin. Deep learning can also be used to identify pathological features in a dental scan (see Figure 4), including depressions, missing teeth, root defects, and recesses. Once detected, these features can be repaired using a 3D meshing algorithm, or a deep learning model can be trained to directly repair the pathological features. Figure 4 shows pathological features including a root defect (left image) and a missing tooth (right image). 【0024】 B. Mesh cleanup using inference These methods use GANs and GDLs to automate the manual mesh cleanup process based on trends learned from the data. Mesh defects include topological holes and uneven surfaces. These methods use machine learning techniques to build a mapping between the uncleaned mesh and the cleaned mesh. This mapping is learned through adversarial training and, given the corresponding uncleaned source mesh, is embodied in a conditional distribution of the cleaned mesh. The model is trained using a dataset of point clouds (called data points) acquired using uncleaned intraoral scans and the corresponding mesh after a cleanup process performed either by a semi-automated software program or entirely manually by a trained human. 【0025】 This machine learning model can later be used as a preprocessing step for other geometric operations. For example, in the case of digital orthodontics, this model can be used to standardize the input point cloud into a coordinate system suitable for processing without requiring human intervention in the loop. This effectively and significantly reduces the processing time for each case and reduces the need to train human operators to perform this task. In addition, because the machine learning model is trained on data generated by many trained humans, it has the potential to achieve higher accuracy compared to when performed by a single person alone. 【0026】 Figure 5 shows the high-level workflow of the training pipeline for this method. The discriminator is used only to help train the generator and is not used during inference (deployment). The discriminator (36) learns to classify tuples of {unclean, clean} data points, and the generator (38) learns to generate false clean data points to deceive the discriminator. 【0027】 Figure 6 shows the method workflow for model inference (deployment), where the generator (40) generates clean data points (42) given unclean data points (44). At this stage, the discriminator is no longer needed. 【0028】 The following are the steps in the workflow. 【0029】 1. Pre-treatment. a. (Optional) Reduction / Expansion: This method can reduce the size of the point cloud and facilitate faster inference by using point cloud reduction techniques such as random downsampling, coverage-aware sampling, or other mesh simplification techniques (if a mesh is available). This method can also expand the size of the point cloud to achieve higher granularity by using mesh interpolation techniques. 【0030】 2. Model inference: The preprocessed mesh / point cloud is passed through a machine learning model to obtain the generated mesh / point cloud. The steps related to using the machine learning model are shown below. a. Model Training: The model is embodied as a set of tensors (called model weights). Meaningful values ​​for these model weights are learned through the training process. These weights are initialized completely randomly. 【0031】 The training process uses training data, which consists of pairs of uncleaned meshes / point clouds and clean meshes / point clouds. This data is assumed to be available before the model is created. 【0032】 The model has two main parts: one is a generator, and the other is a discriminator. The generator takes in a mesh / point cloud and generates another mesh / point cloud. This generated mesh / point cloud has several desired geometric features. The discriminator takes the generated mesh / point cloud and assigns a score to it. The discriminator is also given a clean mesh / point cloud of the corresponding ground truth and assigns another score to it. The adversarial loss encodes the difference between these two scores. The total loss function can also include other components. Several components can be introduced to enforce rule-based problem-specific constraints. 【0033】 This method passes a randomly selected batch from the training dataset to the model and calculates a loss function. It then estimates the gradient from the calculated loss function and updates the model's weights. During model training, the generator is updated to minimize the total function, and the discriminator is updated to maximize the total function. This process is repeated for a predetermined number of iterations, or until certain objective criteria are met. 【0034】 b. Model validation: Typically, the model is validated in parallel with training to monitor potential problems associated with training, such as overfitting. This method assumes that a validation set is available at the start of training. This dataset is similar to the training dataset in that it consists of paired sets of unclean and clean meshes / point clouds. 【0035】 After a set number of training iterations, this method runs the model through a validation set and calculates a loss function value. This value serves as a measure of how well the model generalizes to unseen data. The validation loss value can be used as a criterion for stopping the training process. 【0036】 c. Model testing: Model testing is typically performed on invisible data points that do not have a clean mesh / point cloud of associated ground truth. This is done in the deployment. 【0037】 C. Prediction of dental restoration These methods, given a mesh representing the initial state of the mesh, use GANs and GDLs to predict the mesh after dental restoration based on trends learned from a training set. 【0038】 These methods use machine learning techniques to construct a map between the mesh in an unclean state and the mesh in a clean state. This map is learned through adversarial training and embodied in the conditional distribution of the mesh of restored teeth, given the mesh corresponding to the initial state. The model is trained using a dataset of point clouds (called data points) obtained using intraoral scans of the initial, unrestored state, and the corresponding mesh after the restoration process. 【0039】 The inference machine learning model can be used retrospectively for smile prediction, which could allow the orthodontist to show the patient the final state of the restored dental arch after the restoration process is complete in the software. 【0040】 Figure 7 shows a high-level workflow for training the method. The discriminator is used only to help train the generator and is not used during inference (deployment). The discriminator (46) learns to classify tuples of {unclean, clean} data points, and the generator (48) learns to generate false clean data points to deceive the discriminator. 【0041】 Figure 8 shows the method workflow for model inference (deployment), where the generator (50) generates clean data points (52) given unclean data points (54). At this stage, the discriminator is no longer needed. 【0042】 The following are the steps in the workflow. 1. Pre-treatment: a. (Optional) Reduction / Expansion: This method can reduce the size of the point cloud and facilitate faster inference by using point cloud reduction techniques such as random downsampling, coverage-aware sampling, or other mesh simplification techniques (if a mesh is available). This method can also expand the size of the point cloud and achieve higher granularity by using mesh interpolation techniques. 【0043】 2. Model inference: The preprocessed mesh / point cloud is passed through a machine learning model to obtain the generated mesh / point cloud. The steps related to using the machine learning model are shown below. 【0044】 a. Model Training: The model is embodied as a set of tensors (called model weights). Meaningful values ​​for these model weights are learned through the training process. These weights are initialized completely randomly. 【0045】 The training process uses training data, which consists of pairs of uncleaned meshes / point clouds and cleaned meshes / point clouds. This data is assumed to be available before the model is created. 【0046】 The model has two main parts: one is a generator, and the other is a discriminator. The generator takes in a mesh / point cloud and generates another mesh / point cloud, this generated mesh / point cloud having some desired geometric features. The discriminator takes the generated mesh / point cloud and assigns a score to it. The discriminator is also given a clean mesh / point cloud of the corresponding ground truth and assigns another score to it. Adversarial loss encodes the difference between these two scores. 【0047】 The total loss function can include other components. Several components can be introduced to enforce rule-based problem-specific constraints. 【0048】 This method passes a randomly selected batch from the training dataset to the model and calculates a loss function. It then estimates the gradient from the calculated loss function and updates the model's weights. During model training, the generator is updated to minimize the total function, and the discriminator is updated to maximize the total function. This process is repeated for a predetermined number of iterations, or until certain objective criteria are met. 【0049】 b. Model validation: It is common practice to validate the model in parallel with training to monitor potential problems associated with training, such as overfitting. 【0050】 This method assumes that a validation set is available at the start of training. This dataset is similar to the training dataset in that it consists of paired sets of unclean and clean meshes / point clouds. 【0051】 After a set number of training iterations, this method runs the model through a validation set and calculates a loss function value. This value serves as a measure of how well the model generalizes to unseen data. The validation loss value can be used as a criterion for stopping the training process. 【0052】 c. Model testing: Model testing is typically performed on invisible data points that do not have a clean mesh / point cloud of associated ground truth. This is done in the deployment. 【0053】 D. Verification of dental restorations These methods determine the validation status of components for use in creating dental restoratives. These methods can facilitate the automation of the restorative production pipeline. There are at least two embodiments, namely, 1) an embodiment in which a GraphCNN is used to apply class labels (i.e., pass or fail) to 3D mesh components, and 2) an embodiment in which a CNN is used to apply class labels (i.e., pass or fail) to a set of one or more 2D raster images representing one or more views of the 3D mesh components. 【0054】 Each embodiment aims to use a neural network (NN) to distinguish between two or more states of representation of components used in dental restorative appliances, and optionally to determine whether the components are acceptable for use in constructing the appliance. 【0055】 In these embodiments, quality assurance (QA) can be performed on the finished dental restoratives. In some production pipelines, qualified personnel must inspect the finished restoratives and determine whether they pass or fail. These embodiments automate the process of verifying the restoratives, eliminating the labor of one of the largest remaining "hidden factories," and can reduce a pipeline process that often takes one to two days to as little as half an hour. 【0056】 These embodiments can verify the components of dental restoratives and / or the finished dental restorative. An advantage of using these embodiments for such a QA process is that the NN can evaluate the quality of generated and placed components more quickly and efficiently than is possible by manual inspection, enabling the QA process to be carried out on a scale far exceeding that of a few specialists. A further advantage is that, for example, if the NN recognizes subtle anomalies that a human might miss, it can produce a more accurate determination of the quality of the shape or placement of the components than is possible by manual inspection. Yet another advantage is that by using the NN and examining its results, it is possible to help train human operators to recognize the proper design of the orthotic components. In this way, knowledge can be transferred to new human specialists. 【0057】 In further applications, these embodiments support the creation of a comprehensive automated regression testing framework for code that generates and / or places components. The advantage of this further application is that comprehensive regression testing becomes possible. These embodiments allow the regression testing framework to automatically verify the output of dozens of processed cases, and verification can be performed each time the developer chooses to run the tests. 【0058】 Embodiment 1 - Use of 3D Data These embodiments can be implemented, for example, in part using MeshCNN, an open-source toolkit for implementing graph CNNs (GCNNs). MeshCNN has a sample program that takes a mesh as input and assigns class labels to that mesh. The sample program has a long list of possible classes. The sample program accompanying MeshCNN can classify these 3D meshes in order to assign appropriate labels. MeshCNN is adapted to distinguish between two or more states (e.g., pass / fail) of components (i.e., mold parting surfaces) used in the creation of dental restoratives. 【0059】 Embodiment 2 - Use of 2D Raster Images This implementation is similar to the implementation of Embodiment 1, except that GCNN is replaced with a CNN. The CNN is trained to classify 2D raster images. For a given component, the CNN is trained to recognize each of the different sets of views of the component (e.g., a split plane)'s 3D geometry, either alone, in relation to an input dental structure, or both, either alone or in combination with other features represented in the final orthotic design. These 2D raster images are generated using, for example, a commercially available CAD tool such as Geomagic Wrap, or an open-source software tool such as Blender. 【0060】 Application 1 - Regression Testing As a proof of concept, we trained a MeshCNN to distinguish between examples of "pass" and "fail" mold parting surfaces. "Pass" and "fail" are subjective labels that can be judged by experts and may vary among different experts. This type of label is in contrast to, for example, the label "dog" for an ImageNet image of a dog. The label "dog" is objective and does not include expert opinion. 【0061】 The NNs of these embodiments can be incorporated into a regression testing system for testing the quality of code that automates the production of components used in the manufacture of dental restorative devices. Generally, regression testing is used to determine whether recent changes to code or inputs have adversely affected the output of the system. In this case, it is necessary to be able to change a few lines of automation code and quickly determine whether those changes adversely affected the output of a set of test cases. Dozens of test cases may exist. While it is possible to manually check the output of dozens of test cases, this would be very costly in terms of the time required for a technician or someone else to manually check the output of all test cases. The advantage of this embodiment is that it streamlines this process. Even if one of the 36 test cases fails to produce an acceptable result after a code change, the NN from this embodiment is designed to detect that error. 【0062】 Application 2 - Repair material production pipeline Further applications include using neural networks (NNs) outside of regression testing and applying them as a QA step in production. Currently, 3D data related to the creation of dental prosthetics must be manually inspected by qualified personnel. There are several stages in the manufacturing process where this data must be validated. 【0063】 In one embodiment, a NN is used to verify the accuracy of the "mold splitting surface," a crucial component of the restorative appliance. It is important that the splitting surface is accurately formed. This new NN inspects the splitting surface for each tooth, observing how the splitting surface divides each tooth in two. 【0064】 Generation and arrangement of components These embodiments operate on the output of automation code, which may embody some or all of the content of PCT Patent Application No. PCT / IB2020 / 054778, titled "Automated Creation of Tooth Restoration Dental Appliances," and U.S. Provisional Patent Application No. 63 / 030144, titled "Neural Network-Based Generation and Placement of Tooth Restoration Dental Appliances." Some of these outputs are generated components. A list of generated components (not exhaustive) includes mold parting surfaces, gingival trim surfaces, facial ribbons, incisal ridges, tongue shelves, stiffening ribs, "doors and windows," and diastema matrices. Other of these outputs are placed components (e.g., pre-built library components that must be translated and / or rotated in a specific manner to align with the geometry of the patient's teeth). A list of components to be placed (not exhaustive) includes incisor alignment features, vents, rear snap clamps, door hinges, and door snaps. The technician must inspect the automated output to ensure that the generated components are properly formed and that the placed library components are properly positioned. The NN from this embodiment can be used to determine whether components are properly formed or placed. The advantage is that it saves the technician time and may allow for the production of higher quality dental restoratives by detecting errors in the shape or placement of components that the technician might overlook. Certain components are particularly important, such as mold parting surfaces. Mold parting surfaces form much of the basis for the subsequent formation of the appliance. If there are errors in the mold parting surfaces, there is great value in detecting these errors and in detecting them early in the appliance manufacturing process. 【0065】 Machine learning for both embodiments A machine learning system has two stages of operation: 1) training, and 2) verification / operation. The neural network in the embodiment must be trained on examples of good geometry and examples of bad geometry. In our initial proof of concept, we used mold parting surfaces for the 3D geometry. Figure 9 shows an example of a "pass" parting surface. Figure 10 shows an example of a "fail" parting surface. The "fail" parting surface was generated by intentionally and temporarily modifying the automated code to introduce errors. 【0066】 Training and holdout verification of Embodiment 1 The MeshCNN code was run (without modification) on this specific dataset of component parts of dental restoratives and trained to distinguish between "pass" and "fail" parts. The training dataset contained 14 "pass" and 14 "fail" parting faces. Each "fail" case was a corrupted instance of one of the "pass" cases (i.e., the code was modified so that the generated parting faces were corrupted). The test dataset contained 6 "fail" cases and 7 "pass" cases. The NN was trained over 20 epochs and achieved 100% accuracy on the holdout validation set. Each epoch included repeating each case once. For the purpose of implementing this proof of concept, parting faces were generated using fewer triangles than the production parting faces to conserve the RAM required by the NN and allow the NN to run on a typical laptop. 【0067】 Next, as is common when training machine learning models, we tested the neural network (NN) on a holdout validation dataset—that is, data samples that were not involved in the training process. We prepared 18 "pass" samples (i.e., good split surfaces) and 18 "fail" samples (i.e., poor split surfaces). The NN correctly classified these holdout validation data samples 100% of the time. 【0068】 Figure of Embodiment 1 Figure 11 shows the elements of Embodiment 1, in which a graph CNN (GCNN) (56) is used to directly apply pass / fail labels (58) to a 3D mesh (60) in order to distinguish between pass and fail mold parting surfaces. 【0069】 Figure 12 is a flowchart illustrating the operation of a trained GCNN in the context of regression testing and code development using Embodiment 1. In the context of code testing, both full-size and downscaled meshes are acceptable. These meshes can be generated using fewer triangles than required by the production system to create the orthotics (for example, components can be generated using one-tenth the number of triangles). The flowchart in Figure 12 provides the following methods, as will be further described herein. 【0070】 1. Input: 3D mesh (62) and automation parameters (64). a. Run the affected code (68). b. 3D mesh (70). 2. Input: NN parameters (66). a. Graph CNN (72). b. Class label of the partition surface (74). 3. Output: If label == "Fail", the output is "Failure". Otherwise, output "Pass" (76). 【0071】 Figure 13 is a flowchart illustrating the operation of a trained GCNN in the context of a production manufacturing system using Embodiment 1, where the fit of the components must first be evaluated before fabricating a dental restorative appliance using the components. In this latter application, the components must be full-size (i.e., the mesh must contain all triangles). The flowchart in Figure 13 provides the following method, as further described herein. 【0072】 1. Input: 3D mesh (78) and automation parameters (80). a. Implement landmark-based automation (84). b. 3D mesh (86). 2. Input: NN parameters (82). a. Graph CNN (88). b. Class label of the partition surface (90). 3. Output: If label == "Pass", the test passes. Otherwise, the test fails (92). 【0073】 Figure of Embodiment 2 Figure 14 shows the elements of Embodiment 2, in which a CNN (94) is used to apply pass / fail labels (96) to the mesh by analyzing a set of 2D raster images (98) of the 3D mesh obtained from various views (100) to distinguish between pass and fail mold parting surfaces. 【0074】 Figure 15 is a flowchart illustrating the operation of a trained CNN in the context of a regression test system using Embodiment 2. The flowchart in Figure 15 provides the following method, as will be further described herein. 【0075】 1. Input: 3D mesh (102) and automation parameters (104). a. Run the affected code (108). b. 3D mesh (110). c. A script (112) that generates 2D raster images of a 3D mesh (one image from each of several views of the mesh). d.2D raster image (114). 2. Input: NN parameters (106). a. CNN (116). b. Class label (118). c. Accumulate the "pass" or "fail" result for each image (120). 3. Output: If the label == "Fail" for any image, output "Failure". Otherwise, output "Pass" (122). 【0076】 Figure 16 is a flowchart illustrating the operation of a trained CNN in the context of a production manufacturing system using Embodiment 2, in which the fit of the components must be evaluated before fabricating a dental restorative appliance using the components. The flowchart in Figure 16 provides the following method, as will be further described herein. 【0077】 1. Input: 3D mesh (124) and automation parameters (126). a. Implement landmark-based automation (130). b. 3D mesh (132). c. A script (134) that generates 2D raster images of a 3D mesh (one image from each of several views of the mesh). d.2D raster image (136). 2. Input: NN parameters (128). a. CNN (138). b. Class label (140). 3. Output: The test passes if the label == "Pass" for all 2D raster images. Otherwise, the test fails (142). 【0078】 Figure 17 shows a successful (left) and unsuccessful (right) 2D image of a dental arch with a central clip in place. In the left image, the central clip is correctly positioned, while in the right image, it is not correctly positioned. 【0079】 verification This embodiment is an extension of other embodiments described herein. In this embodiment, in addition to the four items described above, another item is added. In this embodiment, NN is used to distinguish between two or more states of representation of a component used in a dental restorative appliance for the purpose of determining whether the component is acceptable for use in the construction of the appliance, and if the component is found to be unacceptable, NN may, in some embodiments, output instructions on how the component should be modified to correct the geometric shape of the component. 【0080】 The term “3D mesh component” is used to refer to a component generated from the above, a component placed from the above, or another 3D mesh intended for use with rapid prototyping, 3D printing, or stereolithography systems. A component can be either a positive or negative feature that is integrated into the final part by a Boolean operation. This embodiment helps to provide contextual feedback for automated feature generation, and there may be one algorithm or rule set for creating the component and one NN classification for checking the quality of that component. The relationship between the two components includes a recursive “guess and check” mechanism to ensure an acceptable outcome (create / generate > classify > regenerate > classify >...> final design). 【0081】 This embodiment includes 3D mesh components in the context of digital dentistry and automated production of dental appliances. Examples include restorative appliances, clear tray aligners, bracket bonding trays, lingual brackets, restorative components (e.g., crowns, dentures), and patient-specific custom devices. Dentists or providers can apply this embodiment to digital designs created by the provider in their dental practice. Other embodiments are also possible, for example, any application where design automation can benefit from this embodiment, including automated design of support structures for 3D printing and automated design of fasteners for component mounting. In addition, in a 3D printing lab, this embodiment can be applied to prototype parts, which are embodied as 3D meshes. In a manufacturing environment, this embodiment can be applied to custom components to be 3D printed, in which case the NN input is derived from photographs of the component or screen captures of meshes generated by scanning the physical part. This allows manufacturers to verify the quality of output parts without using general 3D analysis software, reducing or eliminating the effort required for human experts to verify the quality of output parts. This embodiment can be applied through user-software interaction, or it can be part of the background operation of a smart system that provides input to a process without direct user intervention. 【0082】 This embodiment is generally useful for detecting problems related to 3D meshes and for automatically correcting those problems. 【0083】 Figure 18 provides elements of this embodiment. A 3D mesh component is created (144). A validation neural network examines 2D raster images of the 3D mesh component (generated from various viewing directions) and determines whether the 3D mesh component passes (146). If the validation neural network indicates a pass, the 3D mesh component is cleared for use in its intended application (for example, a mold parting surface is cleared for use in a reconstructive appliance) (148). If the validation neural network determines that the 3D mesh component does not pass, the validation neural network may, in some embodiments, output instructions on how to modify the 3D mesh component (150). 【0084】 3D mesh components are created through the following means: automatic generation as described herein, automatic placement as described herein, manual generation by an expert, manual placement by an expert, or several other means, such as the use of CAD tools in a rapid prototyping lab or the use of other settings. 【0085】 The 3D mesh component is input to a validation neural network (of the type described herein, for example). The validation neural network indicates a result regarding the quality of the 3D mesh component, i.e., pass or fail. If the result is pass, the 3D mesh component is sent for use for its intended purpose (e.g., to be incorporated into a dental appliance). If the result is fail, in some embodiments, the validation neural network may output instructions on how to modify the 3D mesh component to bring it closer to what is expected. 【0086】 In the embodiments described below, the mold division plane is inspected near each tooth in the dental arch. If the mold division plane does not properly intersect the tooth, this embodiment outputs an instruction that the mold division plane should be moved either lingually or familially to more cleanly bisect the lateral cusp or incisal edge of the tooth. The mold division plane is intended to divide the familial and lingual portions of each tooth, meaning that the mold division plane should extend along the lateral cusp tip of the tooth. If the mold division plane cuts too far lingually or too far familially, the mold division plane does not properly divide the familial and lingual portions of each tooth. As a result, the mold division plane requires adjustment near that tooth. Software that automatically generates the mold division plane has parameters that can be operated to shift the familial / lingual positioning of the mold division plane near the tooth. In this embodiment, these parameter values ​​are stepped up in the appropriate direction so that the mold division plane more cleanly bisects each tooth. 【0087】 In this embodiment, it is possible to request changes to the mold parting surface near some teeth (i.e., if the mold parting surface does not exactly bisect the teeth), but not to request changes to the mold parting surface near other teeth (i.e., if the mold parting surface accurately or more cleanly bisects the teeth). 【0088】 In this embodiment, there are two validation neural networks, one called the lingual bias NN and the other called the facial bias NN. Both of these neural networks are trained on 2D raster images of a view of the 3D geometry of a tooth, the 3D geometry of the tooth is visualized in relation to a mold parting plane (see the description above in this section). A mold parting plane is an example of a 3D mesh component, as defined earlier. 【0089】 Options for creating 2D raster images of teeth related to the mold parting surface include: 1. The mold division surface can be rendered in the scene as a 3D mesh, along with the teeth, which are also meshes. 2. The mold parting surface may intersect with the teeth, resulting in lines that trace the contour of the intersection along the geometric shape of the teeth. 3. The mold parting surface may intersect the teeth in the form of a Boolean operation, thereby subtracting a portion of the teeth (e.g., either the lingual or facial side) from the scene. The remaining two faces of the geometry may be given color or different shading, for clarity, such as blue and red, or light and dark shading. 4. The mold parting surface may intersect with the teeth, resulting in a color-coded tooth mesh. The portion of the tooth mesh located on the facial side of the mold parting surface may be given, for example, red or a first shading. The portion of the tooth mesh located on the lingual side of the mold parting surface may be given, for example, blue or a second shading different from the first shading. This option is shown in Figures 19 and 20. 5. Combine any or all of the above. 【0090】 Figure 19 (left) shows a tooth that is precisely bisected by a bifurcation plane. Figure 19 (right) shows a tooth that is imprecisely bisected by a bifurcation plane (for example, the bifurcation plane extends too far towards the face). Figure 20 shows the same type of data sample as in Figure 19, except that the negative sample (left) corresponds to a bifurcation plane that has penetrated too far towards the lingual side. 【0091】 For each of the above, any view is considered. In some embodiments, a multi-view pipeline can be used to enable the use of any number of views with arbitrary camera positions and angles of the rendered image. 【0092】 training: The lingual bias NN is trained on two classes of images: 1) a class in which the mold parting plane is accurately formed and accurately bisects the teeth, and 2) a class in which the mold parting plane is inaccurately formed and does not accurately bisect one or more teeth. In this example, images were created that reflect several arbitrary views of each tooth in the dental arch. These views should show the teeth in relation to the parting plane, since the parting plane intersects the teeth (as listed above). In this case, option 4 above can be used, where the parting plane intersects the teeth and produces, for example, red and blue coloring or different shading on the teeth. 【0093】 In this embodiment, a lingual bias NN is trained to distinguish between two classes of images (i.e., images with acceptable parting surfaces and images with unacceptable parting surfaces). If the lingual bias NN indicates an unacceptable result for an input parting surface, the method recognizes that the parting surface has bifurcated the tooth in a manner that goes too far lingually. Therefore, when the mold parting surface is reworked by automated generation software (e.g., automated generation software as described herein), the method outputs an instruction that the parting surface has gone too far lingually with respect to that tooth and should be moved slightly in the opposite direction. The code for automatically generating the parting surface has a parameter for each tooth that can shift the parting surface either lingually or facially. This parameter can be adjusted so that subsequent iterations of the parting surface move it slightly facially with respect to that tooth. 【0094】 In another embodiment, a regression network working on tooth images can be used to substantially estimate the amount of surface displacement in the lateral facial direction. The regression network can be used to estimate "deviation" in the tongue or facial region when tooth images are given. It may be feasible to convert the amount of deviation into parameters. This modification in the feedback loop reduces the number of iterations / corrections in the method. 【0095】 The facial bias NN is trained using the same positive-class images as the lingual bias NN, but the negative-class images are generated using a parting plane that extends too far facially along the teeth. When the facial bias NN indicates a failure, the method recognizes that the mold parting plane extends too far facially along the teeth and must instruct the automated generation software to gradually move the mold parting plane lingually; otherwise, all the remaining training details are substantially the same. 【0096】 In another embodiment, a neural network can be trained to extract anomalies in the amount of deviation in either the lingual or facial direction. Such a NN has the ability to highlight what was the most prominent part of the dental arch mesh / image in its inference. 【0097】 In some embodiments, a regression network can be used to estimate the amount of deviation on the face side and adjust the corresponding parameters accordingly. 【0098】 Operation of trained neural networks: Each tooth is analyzed separately. Several images of each tooth are passed through a pipeline, and a pass / fail judgment is indicated for each image. When several images of a tooth / partition plane combination are passed through this pipeline, different options exist for determining the result. In some embodiments, if at least one of several views indicates a fail judgment, the method outputs an instruction that the partition plane in the vicinity of that tooth needs to be adjusted. In other embodiments, if some or most of the analyzed images indicate a fail judgment, the method outputs an instruction that the partition plane in the vicinity of that tooth needs to be adjusted. 【0099】 In one embodiment, the validation neural network includes a convolutional neural network (CNN). The CNN can embody a variety of different network configurations, in particular including networks having different numbers of layers, different numbers of nodes per layer, different use of dropout layers, different use of convolutional layers, different use of high-density layers, and so on. 【0100】 In another embodiment, the validation neural network may utilize elements of a multi-view CNN (MVCNN) architecture. In summary, the network input uses any number of images of a 3D scene. All images pass through a shared copy of a feature extraction CNN. These features are then pooled using a view pooling mechanism and fed into a classification network, which is typically a fully connected network. The fundamental difference from a standard CNN is that this type of architecture can use multiple views of the same scene. Training works in a similar way, but with one change: instead of passing one image and label / value at a time, this method passes multiple views of the mesh as images and labels / values ​​at once. 【0101】 In yet another embodiment, the validation CNN (which operates using 2D raster images) can be replaced with a neural network that operates directly with 3D data, such as a MeshGAN. In yet another embodiment, the validation CNN (which operates using 2D raster images) can be replaced with a GraphCNN (which operates directly on 3D data). In yet another embodiment, the validation CNN (which operates using 2D raster images) can be replaced with a GraphGAN (which operates directly on 3D data). 【0102】 In one example, images of the teeth related to the mold parting surface are provided for both 1) lingual bias NN and 2) facial bias NN. 1. If both neural networks indicate a pass / fail result, the mold parting surface is cleared for use in the production of restorative dental appliances. 2. If the lingual bias NN indicates a failure and the facial bias NN indicates a pass, this method outputs an instruction that the mold parting surface has gone too far in the lingual direction. The automatic mold parting surface generation software must adjust the mold parting surface little by little in the facial direction near the teeth when the next iteration of the mold parting surface is created. 3. If the lingual bias NN indicates a pass and the facial bias NN indicates a fail, this method outputs an instruction that the mold parting surface has gone too far in the facial direction. The automatic mold parting surface generation software must adjust the mold parting surface little by little in the lingual direction near the teeth when the next iteration of the mold parting surface is created. 4. If both the lingual bias NN and the facial bias NN indicate a failure, the result is provided to a human decision-maker who determines whether the mold parting surface requires adjustment near the teeth. 【0103】 This method involves looping through each tooth and determining whether the mold parting surface is accurately positioned relative to that tooth, or whether the mold parting surface requires adjustment in either the lingual or facial direction near the tooth. 【0104】 In another embodiment, a pair of neural networks (NNs), including a lingual-biased NN and a facial-biased NN, each capable of performing two-class classification, may be replaced with a single NN capable of performing three-class classification. This three-class classification NN is trained on 2D raster images from the following three classes: • Class 0 - A color-coded view of teeth where the teeth are divided by a mold splitting surface that has been intentionally modified to extend too far lingually. • Class 1 - A color-coded view of teeth where the teeth are divided into two by precisely formed mold parting surfaces. • Class 2 - A color-coded view of teeth where the teeth are divided by a mold splitting surface that has been intentionally modified to extend too far towards the face. 【0105】 The 3-class classification neural network shows its predictions from this set of three class labels. 【0106】 In another embodiment, an N-class classification neural network (NN) can be employed to assign one of N possible class labels to each data sample, corresponding to N distinct states of the orthotic components (i.e., mold parting surfaces). 【0107】 In another embodiment, instead of having two separate neural networks (NNs), both facial and lingual views can be incorporated into a single NN. In this case, the graphical convolutional network takes the entire tooth mesh as input and outputs a single regression value representing the "radial" adjustment amount for that particular tooth. The input to such an NN (the original 3D scene) contains, strictly speaking, more information than a few arbitrarily rendered images rendered from the scene. 【0108】 Figure 21 provides further details of the embodiment of Figure 18 in the context of the mold parting surface. The flowchart of Figure 21 provides the following method, as will be further described herein. 【0109】 1. Input: Automation parameters (152) and 3D mesh of the patient's teeth (154). a. The dividing surface is generated by an automated program (156). b. The division plane intersects the entire dental arch, and the resulting facial portion of the dental arch is colored red and the lingual portion is colored blue (or other colors or shading are applied to these portions) (158). 2. Generate N color-refined views of the teeth from various arbitrary viewing angles (168). a. Input: NN parameters (160). i. Perform a lingual bias neural network for each view (162). b. Input: NN parameters (166). i. For each view, run a face-biased neural network (164). 3. The results for each view are aggregated to show the determination for the part of the partition surface near Tooth_i (170). a. The partition plane does not require any changes in the neighborhood of Tooth_i (172). b. If the facial bias NN outputs a failure judgment and the lingual bias NN outputs a pass judgment, it is recorded that the division plane should move to the lingual side in the vicinity of Tooth_i (172). c. If the lingual bias NN outputs a failure judgment and the facial bias NN outputs a pass judgment, it is recorded that the division plane should move towards the face in the vicinity of Tooth_i (172). d. If both neural networks output a failure, do nothing at the Tooth_i position, or flag it and have a human operator inspect the split surface (172). 4. Consolidate adjustment commands for each tooth (174). a. Feedback: The aggregated adjustment commands are sent to the software to automatically generate the dividing surface (176), and the process returns to step 1.a (156). b. If no adjustments are needed, complete (178). 【0110】 The following is one embodiment of a neural network used in the implementation of a 2D raster image embodiment of the verification component. [Table 1] 【0111】 In one embodiment, a neural network is used that is trained on 1) examples of correct orthotic components and 2) examples of orthotic components that have been systematically modified to be incorrect. This neural network distinguishes between correct and incorrect orthotic components using multiple 2D renderings of the components obtained from different views. 【0112】 NN can be trained to distinguish between 1) a correct division plane and 2) a division plane that is too far lingual. NN was tested on a total of 54 teeth in the cases of three patients. In this test, 50 teeth were predicted to be correct, and 4 teeth were predicted to be incorrect. 【0113】 Figure 22 is a flowchart of the following verification process, and Figure 23 is a diagrammatic representation of this process. 【0114】 1. Input: Tooth data (180). 2. Automatically generate orthotic components (182). 3. Generate a 2D view of the patient's teeth related to the orthotic components (184). 4. A neural network (NN) validates the 2D view of the orthotic components (186). a. If NN returns a pass, the component will be cleared for use in the orthotic device. If b.NN returns a failure, in some embodiments, feedback can be sent to the repair automation code to refine the next iteration of the component design. 5. Output: Components in a state suitable for use in a reconstructive orthotic device (188). 【0115】 In this method, the neural network is trained to verify the accuracy of the mold parting surface. This embodiment reflects a two-class classification, where the neural network is trained on data from two classes, namely, the parting surface is one of the following: Class 0: Positioned too far lingually. or Class 1: Correct (not too far lingual or familial). 【0116】 Figure 24 shows 30 views of the upper left lateral incisor (tooth 10) bisected by a bifurcation plane. In this test case, the tooth was bisected by a bifurcation plane that was modified to extend 0.5 mm lingually. Other test cases were designed around a bifurcation plane that was modified to extend 1.0 mm lingually. Yet another test case was designed around a bifurcation plane that precisely bisected the tooth (i.e., neither lingually nor facially). 【0117】 Each of the 30 bifurcated tooth views (Figure 24) was passed through a neural network. The neural network made a prediction for each view and concluded that the dividing plane shown in that view was either Class 0 "overly lingual" or Class 1 "correctly positioned". In this test case, the ground truth label for all views was the same "Class 0". However, due to the ambiguity of the geometry of this particular dividing plane, the neural network was unable to correctly classify some of the views (i.e., it assigned the label "Class 1" to those views). These mispredicted views are shown in light gray in the figure. This effect in the image was achieved using an alpha channel. There are 11 such views where the neural network showed a prediction that did not match the ground truth label. The remaining 19 full-color views are views where the neural network obtained a prediction that matched the ground truth label. The validation system concluded that the dividing plane was overly lingual in a majority of 19 out of 30 views. This was a test case where ground truth data was available. 【0118】 This method of visualizing the results of a neural network is advantageous because it organizes multiple views of the teeth for a single test case, allowing a human to visually examine and grasp the results of the test case. 【0119】 Figure 25 shows 30 views of the upper right canine tooth (tooth 6) precisely divided by the dividing plane. In this case, the neural network made an incorrect prediction for only one of the views (see the light gray view near the upper left of the figure). Even in this test case, the dividing plane is correct in a majority of cases, with 29 out of 30 being correct. 【0120】 The neural network is trained on the following three classes of ground truth data: Class 0: The splitting plane is intentionally modified to be positioned too far lingually. Class 1: Correct (not too far lingual or familial). Class 2: The dividing plane is intentionally modified to be positioned too far towards the face. 【0121】 The various embodiments described herein can be used in various different neural networks. Embodiment 2 uses a CNN. Embodiment 1 uses a graph convolutional neural network (GraphCNN). Other embodiments may include elements derived in whole or in part from other types of neural networks, namely, perceptrons (P), feedforward (FF), radial basis networks (RBF), deep feedforward (DFF), recurrent neural networks (RNN), long-shortened memory (LSTM), gated recurrent units (GRU), autoencoders (AE), variational autoencoders (VAE), denoising autoencoders (DAE), sparse autoencoders (SAE), encapsulated autoencoders (CAE), stacked encapsulated autoencoders (SCAE), deep belief networks (DBN), deep convolutional networks (DCN), deconvolutional networks (DN), generative adversarial networks (GAN), liquid state machines (LSM), and neural Turing machines (NTM). 【0122】 GraphCNN can operate on dental data provided in 3D formats such as 3D meshes. A mesh contains both vertices and instructions on how to place those vertices on faces. The definition of a face implicitly includes information about the edges connecting the vertices. 【0123】 CNNs can operate on dental data provided in the form of 2D raster images. 2D raster images can use color or shading to highlight areas of interest within dental anatomical structures (for example, using red and blue coloring, or light and dark shading, to show the facial and lingual portions of a tooth resulting from applying a mold parting plane to the tooth). 【0124】 These neural networks can be trained with augmented data. For 3D mesh data, augmentation can include probabilistic or deterministic transformations applied to vertices or faces that alter the shape of the 3D mesh but do not change the essential identity of the mesh. This variation in mesh shape can help the classifier avoid overfitting when used as training data. For 2D raster images, the image can be resized, stretched, rotated, sheared, or noise introduced. Similarly, for 3D data, augmenting 2D data to use as training data can help the neural network avoid overfitting during training. 【0125】 The neural networks described herein can incorporate various activation functions, such as RELU. Other activation functions include binary step, identity, logistic, and TanH. Neural networks can incorporate downsampling techniques such as pooling and maximum pooling. Neural networks can reduce overfitting and generalization errors using regularization techniques such as dropout. 【0126】 Other verifications The following are other examples of dental appliances that may benefit from the verification techniques described herein. 【0127】 1. Custom orthodontic appliances (e.g., lingual brackets) In some embodiments, the validation techniques described herein can be applied to the design of custom lingual brackets. A validation neural network (NN) can be trained to pass or fail the design of a lingual bracket using a digital 3D view of the lingual bracket placed on a tooth. This feedback may be used by a trained technician or sent to automated software that generated the lingual bracket to improve the design of subsequent iterations of the lingual bracket. For lingual brackets, the bonding pad is created for a specific tooth by drawing the contour around the tooth, creating the thickness that forms the shell, and then subtracting the tooth via a Boolean operation. The bracket body is selected from a library, placed on the pad, and integrated into the pad via a Boolean addition. Various bracket components (e.g., hooks and wings) are adjusted to best fit the specific geometric shape of the tooth and gingiva, integrated into the bracket body, and the digital design of the bracket is completed, which is exported as a 3D geometry file. In some embodiments, the STL format can be used for the 3D geometry file. 【0128】 2. Custom indirect bonding of non-custom brackets Brackets are selected from a library and custom-positioned onto the teeth. Fine-tuning is performed based on the local anatomical structure of the tooth in the bonding area, and some customization of torque and rotation is possible through corrections within the adhesive bond line between the tooth and the bracket. The NN is trained to recognize inconsistencies in bracket placement, which can be either automated or technician-generated. 【0129】 3. Aligner or clear tray aligner (CTA) In another embodiment, the verification techniques described herein can be applied to 3D data used to design CTA, for example, an aligner tray. An example of such data is a 3D representation (e.g., a 3D mesh) of a patient's teeth, called a “fixture model,” which is then sent to a 3D printer. Parameters such as the location of trim lines, attachments, bite ramps, or slits can be verified. Trim lines are the areas where the aligner is trimmed during thermoforming. Aligners that are directly 3D printed can have more complex features (local thickness, geometry of reinforcing ribs, flap placement, etc.), and the verification techniques described herein can be applied to them. 【0130】 A validation neural network (NN) can be trained to perform pass / fail judgments on CTAs using digital 3D models of the patient's teeth and gums that show the trim lines. This feedback can be used by trained technicians or sent to automated software that generated the CTAs to improve the design of subsequent iterations of the CTAs. A CTA is a series of removable, nearly invisible plastic trays molded to progressively move the patient's teeth along a series of predetermined positions. 【0131】 Other dental devices that can be verified using the verification techniques described herein include data or structures relating to the design of implant placement or other types of dental restorations (such as veneers, crowns, or bridges). 【0132】 Furthermore, the verification techniques described herein can be used to verify bracket placement, including either or both manual placement by human experts and automated placement by algorithms. 【0133】 II. Classification of Vertices and Edges These embodiments include, for example, providing the ability to segment hardware from a scanned dental arch using a machine learning segmentation module. The hardware may take the form of brackets, braces, or other complex external prosthetics. 【0134】 A. Segmentation A deep learning model is used to automatically segment teeth from a 3D mesh. This process can be divided into two steps: model development / training and model deployment. During training (flowchart 1 in Figure 26), both unsegmented and segmented digital 3D models from multiple patients are input into the deep learning model, which is optimized to learn patterns that minimize the difference between predicted tooth segmentation and actual tooth segmentation. During model deployment (flowchart 2 in Figure 26), the trained deep learning model is used to generate segmentation predictions for new case data that has not been seen before. 【0135】 The flowchart in Figure 26 provides the following method, as will be further described herein. 【0136】 Model development / training: 1. Input: Unsegmented and segmented digital 3D models (190) from past case data. 2. (Optional) Data expansion, as well as mesh cleanup and resampling (192). 3. Train the deep learning model (194). 4. Evaluate segmentation predictions for ground truth segmentation data (196). 【0137】 Model development: 1. Input: Digital 3D model of a new case of malocclusion (198). 2. (Optional) Mesh cleanup and resampling (200). 3. Run the trained deep learning model (202). 4. Generate the proposed segmentation (204). 【0138】 As more data is acquired, machine learning methods, and deep learning methods in particular, begin to outperform explicitly programmed methods. Deep learning methods have a significant advantage in that, throughout the training process, they can infer some useful features directly from the data using a combination of several nonlinear functions of higher-dimensional latent or hidden features, thus eliminating the need to manually create features. While attempting to solve segmentation problems, it is sometimes desirable to work directly on a 3D mesh of malocclusion. 【0139】 Deep learning for tooth segmentation from the gingiva: The deep learning model uses MeshCNN to segment teeth from 3D mesh data. MeshCNN is a general-purpose deep neural network for 3D triangular meshes and can be used for tasks such as 3D shape classification or segmentation. This framework is advantageous over other methods because it includes convolutional, pooling, and unpooling layers applied directly to the mesh edges and is invariant to changes in mesh rotation, scaling, and translation. Deep learning algorithms, including MeshCNN, have two main development steps: 1) training the model, and 2) deploying the model. 【0140】 1. Model training Model training utilizes multiple unsegmented and segmented digital 3D models from historical case data. Before use, these 3D models can undergo some mesh cleanup and resampling. In our case data, many standard mesh cleanup operations were performed, including filling holes, removing fine edges, and removing isolated areas. For computational efficiency during model training, mesh decimation was also performed, reducing the number of faces to a smaller number (approximately 3000). Data augmentation techniques, including heterogeneous scaling, vertex shifting, and edge flipping, were used to increase the number of 3D mesh samples used to train the deep neural network. The labels of the unsegmented meshes and each mesh edge were input into the MeshCNN framework. As is standard in deep learning models, the model was trained through a process of iteratively adjusting the set of weights to minimize the difference between predicted segmented labels and actual segmented labels. The trained model was then evaluated by predicting segmented labels for a spare set of cases not used during training and measuring the accuracy. This model achieved an accuracy of 97% in correctly identifying edges as belonging to either teeth or gums. 【0141】 2. Model Development In the model deployment phase, the trained model developed during the model training in Step 1 is used. The trained model receives an unsegmented 3D scan of a new case as input. Any mesh cleanup or resampling performed on the 3D mesh during the model training phase should be applied to the new 3D scan data. For each edge, the trained model outputs a set of labels indicating whether the edge belongs to the "gingival" class or the "dental" class. 【0142】 Figure 27 shows examples of segmentation results for several upper and lower dental arches. 【0143】 Expansion of tooth type classification: The segmentation results created above were generated by assuming that the edges in the mesh belong to one of two classes: (1) teeth or (2) gingiva. Alternatively, edges can be labeled as belonging to one of several classes, for example: 1. Types of teeth: (1) molars, (2) premolars, (3) canines, (4) incisors, (5) gums. 2. Tooth type and dental arch: (1) Molars of the upper dental arch, (2) Premolars of the upper dental arch, (3) Canines of the upper dental arch, (4) Incisors of the upper dental arch, (5) Molars of the lower dental arch, (6) Premolars of the lower dental arch, (7) Canines of the lower dental arch, (8) Incisors of the lower dental arch, (9) Gingiva. 3. By tooth number: (1) gum, (2) tooth 1, (3) tooth 2, ..., (33) tooth 32. 【0144】 Deep learning models such as MeshCNN can be trained to label edges as belonging to one of several classes. 【0145】 B. Segmentation using inference This method uses GDL to infer parts or segments of an object scan using different scanning hardware. It employs machine learning techniques to infer segmentation of the input point cloud. These segments correspond to individual teeth and gums. The model is trained using a dataset of point clouds (hereinafter referred to as data points) acquired using intraoral scans, where the dataset is embodied as a set of (x,y,z) coordinates for each point in the point cloud and associated segmentation from the point to the teeth and gums. 【0146】 This map can later be used in other geometric calculations. For example, in the case of digital orthodontics, this model can be used to standardize the input point cloud in a coordinate system suitable for processing without the need to manually input loops. This effectively and significantly reduces the processing time for each case and also reduces the need to train human operators to perform this task. 【0147】 Figures 28 and 29 illustrate the workflow of this method. 【0148】 The flowchart in Figure 28 provides the following methods for a training pipeline, as will be further described herein. 1. Point cloud / mesh (206). a. Relevant segmentation (212) for training and validation only. 2. (Optional) Reduce / Expand (208). 3. (Extended) point cloud / mesh (210). a. Relevant segmentation (214) for training and validation only. 4. GDL machine learning models (216). 5. Predicted segmentation (218). 【0149】 The flowchart in Figure 29 provides the following method for a test pipeline, as will be further described herein. 【0150】 1. Point cloud / mesh (220). 2. (Optional) Reduce / Expand (222). 3. (Extended) point cloud / mesh (224). 4. GDL machine learning models (226). 5. Predicted segmentation (228). 【0151】 During training, both the point cloud and its associated segmentation are provided, but during testing, only the point cloud is provided. 【0152】 The workflow involves the following steps: 【0153】 1. Pre-treatment: a. (Optional) Point cloud reduction / expansion: This method can reduce the size of the point cloud and facilitate faster inference by using point cloud reduction techniques such as random downsampling, coverage-aware sampling, or other mesh simplification techniques (if a mesh is available). This method can also expand the size of the point cloud to achieve higher granularity by using mesh interpolation techniques. 【0154】 b. (Optional) Segmentation Reduction / Expansion: When a point cloud is decimated, the resulting segmentation of the point cloud is proportionally decimated by dropping out the decimated points. When a point cloud is expanded, segmentation labels for the newly created points are determined using nearest neighbor queries against the points in the original point cloud. 【0155】 2. Model Inference: The (extended) point cloud is passed through a machine learning model to obtain a corresponding approximate coordinate system. The steps involved in using the machine learning model are shown below. 【0156】 a. Model Training: The model is embodied as a set of tensors (called model weights), and meaningful values ​​for these model weights are learned through the training process. These weights are initialized completely randomly. 【0157】 The training process uses training data, which is a set of paired data points and their associated coordinate systems. This data is assumed to be available before the model is created. 【0158】 This method passes a randomly selected batch from the training dataset to the model and calculates a loss function. This loss function measures the dissimilarity between the ground truth coordinate system and the predicted coordinate system. 【0159】 This method estimates the gradient from the calculated loss function and updates the model weights. This process is repeated for a predetermined number of iterations, or until certain objective criteria are met. 【0160】 b. Model validation: It is common practice to validate the model in parallel with training to monitor potential problems associated with training, such as overfitting. 【0161】 This method can use a validation set available at the start of training. This dataset is similar to the training dataset in that it consists of a set of paired data points and their associated coordinate systems. 【0162】 After a set number of training iterations, this method passes a validation set to the model and calculates a loss function value. This value serves as a measure of how well the model generalizes to unseen data. The validation loss value can be used as a criterion for stopping the training process. 【0163】 c. Model testing: Model testing is typically performed on invisible data points that do not have associated annotated segmentation. 【0164】 III. Regression These embodiments include, for example, the following: 【0165】 Case Complexity: Use the regression module to classify the level of treatment complexity for a given case based on scanned dental arches. 【0166】 Case Characteristics: Using a regression model, scanned dental arch meshes are classified based on case characteristics such as occlusal relationships (class 1, 2, or 3), occlusion (overbite / deep bite), and midline shift. Using a regression model, scanned dental arch meshes are classified based on existing labels of case characteristics such as occlusal relationships (class 1, 2, or 3), occlusion (overbite, overjet, anterior / posterior crossbite), midline offset, anterior leveling, space / crowding, dental arch morphology, and applied protocols (extrusion, expansion, distal movement). 【0167】 Predicting Treatment Duration: Using a regression module, the complexity level of treatment for a given case with scanned dental arches is classified, which is then used to predict the amount of care required and the treatment time. 【0168】 A. Coordinate system This embodiment includes a machine learning method for determining the relative orientation or coordinate system of a 3D object with respect to a global reference frame. Such a method has implications for issues such as orthodontic treatment planning. 【0169】 The problem of determining the orientation of a 3D object is typically solved using computational geometry techniques. In particular, 3D orientation estimation from 2D images of humans and human faces is a well-studied problem. However, there are scenarios where the relative orientation of a 3D object given a reference frame is important, and information about the object's shape in 3D is available. Traditionally, orientation is determined using explicit descriptions of shape features and matching or alignment to a template. For example, the iterative nearest neighbor (IC) algorithm can be used to align the observed 3D shape of a target to a standard template. The inferred transformation matrix can then be used to transform the orientation of the reference template to the target shape. 【0170】 Deep learning methods directly applied to the representation of 3D shapes have been used to solve two problems: 1) object classification, and 2) semantic segmentation or vertex / element-wise classification. Similar techniques can be used to predict pose or coordinate system. The requirement is that the model predicts a set of real numbers or a transformation matrix representing the pose (the position and orientation of the 3D object relative to a global reference frame). This can be represented by seven output parameters (three for translation and four for the quaternion representation of rotation). This provides fewer parameters than the 12 parameters required to represent the complete transformation matrix. However, this representation is not limited, and other representations such as axis angles or Euler angles can also be used. 【0171】 Method: Given a large amount of training data of mesh geometry (e.g., a mesh representation of teeth) and corresponding output transformation parameters as labels, a mesh-based or point-based deep learning model can be trained using, for example, PointNet or PointCNN. Furthermore, during training, data augmentation such as undersampling, rotation, and point reordering can be performed on the input mesh. This can help generate thousands of augmented input data from a single source, greatly increasing the potential for the algorithm to perform better. Figure 30 shows the prediction of the tooth coordinate system. 【0172】 The following are exemplary embodiments for coordinate system prediction: a method for receiving 3D point cloud or mesh data, given a global reference frame, using a machine learning algorithm to predict relative orientation and position; and a method for receiving 3D point cloud or mesh data, using an alignment algorithm to align the point cloud to a known set of one or more templates, and then using the results to determine relative orientation and position with respect to a global reference frame. 【0173】 These embodiments can be used, for example, when a 3D point cloud represents teeth and the alignment algorithm may be ICP, ICP with a distance metric from point to plane, etc. 【0174】 B. Coordinate systems used for inference These methods use GDL to infer the object's orientation / coordinate system using only the point cloud acquired from its surface using different scanning hardware. 【0175】 These methods use machine learning techniques to infer a map between point clouds and their associated coordinate systems. An example of such an algorithm can be a modification of PointNet. This method trains a model using a dataset of point clouds (called data points) acquired using intraoral scans, which are embodied as a set of (x,y,z) coordinates and associated coordinate systems for each point in the point cloud, embodied in a 6-dimensional representation. The model acts as a regression map between the point cloud domain and the coordinate system domain, and given a point cloud, the model infers the associated coordinate system. 【0176】 This map can later be used in other geometric calculations. For example, in the case of digital orthodontics, this model can be used to standardize the input point cloud in a coordinate system that can be processed without requiring a human in the loop. This effectively and significantly reduces the processing time for each case and also reduces the need to train human workers to perform this task. 【0177】 Figures 31 and 32 illustrate the high-level workflow of this method. 【0178】 The flowchart in Figure 31 provides a method for a training pipeline, as will be further described herein. 1. Point cloud / mesh (230). a. Coordinate system (238) for training and validation only. b. Coordinate system transformation (240). 2. (Optional) Reduce / Expand (232). 3. Standardization (234). 4. Standardized point cloud / mesh (236). a. Coordinate system (242) for training and validation only. 5. GDL machine learning models (244). 6. Predicted coordinate system (246). 【0179】 The flowchart in Figure 32 provides a method for a test pipeline, as will be further described herein. 1. Point / cloud mesh (248). 2. (Optional) Reduce / Expand (250). 3. Standardization (252). 4. Standardized point cloud / mesh (254). 5. GDL machine learning models (256). 6. Predicted coordinate system (258). 【0180】 Method Workflow: The input point cloud is obtained by segmenting teeth from a scanned dental arch. This point cloud is originally in the "global coordinate system". The following are the steps in the workflow. 【0181】 1. Pre-treatment: a. (Optional) Point Cloud Reduction / Expansion: This method can reduce the size of the point cloud and facilitate faster inference by using point cloud reduction techniques such as random downsampling, coverage-aware sampling, or other mesh simplification techniques (if a mesh is available). This method can also expand the size of the point cloud to achieve higher granularity by using mesh interpolation techniques. 【0182】 b. Point Cloud Standardization: This method uses a whitening process to position the mean of the point cloud at the origin and align the principal axes of the point cloud with the X, Y, and Z axes. This method is based on Principal Component Analysis (PCA). This method subtracts the mesh mean from each point in the point cloud and rotates the point cloud using the inverse of an orthogonal matrix consisting of eigenvectors of the autocorrelation matrix of the point cloud extracted using PCA. While standardizing the point cloud, this method also modifies the associated coordinate system to reflect this affine transformation. 【0183】 c. Determining the Coordinate System: The coordinate system is encoded in a 6-dimensional vector. The first three, called translational components, encode the position of the origin of the local coordinate system in the global coordinate system. The remaining three, called rotational components, encode the orientation of the coordinate axes. This transformation uses the Cayley transformation. The original orientation may be encoded as an orthogonal matrix or as a set of Euler angles, and this method transforms them into the corresponding Cayley angles. 【0184】 2. Model inference: The standardized point cloud is passed through a machine learning model to obtain a corresponding approximate coordinate system. The steps involved in using the machine learning model are shown below. 【0185】 a. Model Training: The model is embodied as a set of tensors (called model weights). Meaningful values ​​for these model weights are learned through the training process. These weights are initialized completely randomly. 【0186】 The training process uses training data, which is a set of paired data points and their associated coordinate systems. This data is assumed to be available before the model is created. 【0187】 This method passes a randomly selected batch from the training dataset to the model and calculates a loss function. This loss function measures the dissimilarity between the ground truth coordinate system and the predicted coordinate system. 【0188】 This method estimates the gradient from the calculated loss function and updates the model weights. This process is repeated for a predetermined number of iterations, or until certain objective criteria are met. 【0189】 b. Model validation: It is common practice to validate the model in parallel with training to monitor potential problems associated with training, such as overfitting. 【0190】 This method assumes that a validation set is available at the start of training. This dataset is similar to the training dataset in that it consists of pairs of data points and their associated coordinate systems. 【0191】 After a set number of training iterations, this method passes a validation set to the model and calculates a loss function value. This value serves as a measure of how well the model generalizes to unseen data. The validation loss value can be used as a criterion for stopping the training process. 【0192】 c. Model testing: Model testing is typically performed on invisible data points that do not have an associated ground truth coordinate system. This is done in the deployment. 【0193】 3. (Optional) Post-processing: The estimated coordinate system is embodied in a 6-dimensional vector and can be converted to any desired format, such as Euler angles. This method can also use this estimated coordinate system to convert the input mesh to its local coordinates, which can then be used for other operations in the pipeline. 【0194】 The following describes the inventors' experimental setup for this task and the results for some teeth. 【0195】 Experimental setup: A set of 65 cases (potentially partially complete) was split into training and validation sets using a 4:1 split. Each case consisted of a point cloud and a set of associated coordinate systems annotated by humans. The point clouds corresponding to these cases had a variable input point density and were non-uniform in their size. Only the (x,y,z) coordinates of the points were used as feature vectors. 【0196】 result: Figure 33 shows some of the results from a validation set regarding the performance of our model, showing the prediction for the second molar (UNS=1) of the upper right dental arch. There are two figures for each case. The first figure (top) corresponds to the coordinate system overlay on the mesh. The second figure (bottom) corresponds to the difference in the point cloud transformed using the coordinate system prediction. Red (or the first shading) is used to show the prediction of our machine model, and blue (or the second shading, different from the first shading) is used to show the ground truth annotation corresponding to the validation point. 【0197】 IV. Automated Encoders and Clustering - Grouping of Providers and Preferences These embodiments include, for example, the following: 【0198】 Grouping of physicians and preferences: Using unsupervised methods such as clustering, providers (e.g., physicians, technicians, or others) are grouped based on their respective treatment preferences. Treatment preferences can be indicated in treatment prescription forms, or they can be based on characteristics in the treatment plan, such as setup characteristics (e.g., amount of occlusal or midline correction in the planned final setup), staging characteristics (e.g., treatment duration, tooth movement protocol, or overcorrection method), or outcomes (e.g., number of corrections / improvements). 【0199】 Using supervised methods and provider identification information from existing data, the recommendation system is trained for each provider based on their past preferences. 【0200】 Using supervised methods, long paragraphs of provider notes (e.g., from a physician) are translated or converted into the correct sequence of steps that setup technicians follow when designing treatments. 【0201】 The flowchart in Figure 34 provides a method for determining provider preferences. 【0202】 1. Input: Information on the provider's past treatments, e.g., each provider's treatment prescription form (260). 2. Use machine learning to summarize each provider's preferences, for example, using any of the machine learning techniques described herein (262). 3. Output: Customized treatment plans for each provider, based on provider preferences and past treatments (264). 【0203】 After provider preferences are summarized by machine learning, the treatment planning algorithm takes those preferences into account and generates a customized future treatment plan according to each provider's (e.g., physician's) past treatments. Customizing the treatment can reduce the number of plan revisions between physicians and technicians. The table below provides an exemplary data structure for storing these provider preferences and customized treatment plans. Customized treatment plans can be stored in other ways and templates. 【0204】 [Table 2] 【0205】 The methods and processes described herein can be implemented in a software module or firmware module so that they can be executed by one or more processors, such as processor 20. Information generated by these methods and processes can be displayed on a display device, such as display device 16. If user interaction is necessary or desired in these methods and processes, such interaction can be provided via an input device, such as input device 18. 【0206】 The GDL and machine learning embodiments described herein can be combined for the purpose of using GDL processing in any combination of embodiments such as the following: 【0207】 The mesh or model cleanup process described in Section I can be performed before or in conjunction with the dental restoration prediction process described in Section I to provide the cleaned mesh or model to the dental restoration prediction process. 【0208】 The mesh or model cleanup process described in Section I, along with the segmentation process described in Section II and the coordinate system process described in Section III, can be performed before or with the dental restoration prediction process described in Section I to provide the dental restoration prediction process with a segmented and coordinate-system-based cleanup mesh or model. 【0209】 The mesh or model cleanup process described in Section I can be performed before or in conjunction with the dental restoration verification process described in Section I, to provide the cleaned mesh or model to the dental restoration verification process. 【0210】 The mesh or model cleanup process described in Section I, along with the segmentation process described in Section II and the coordinate system process described in Section III, can be performed before or during the dental restoration verification process described in Section I to provide the dental restoration verification process with a segmented and coordinate system-based cleaned mesh or model. 【0211】 The dental restoration prediction process described in Section I, along with the dental restoration verification process described in Section I, can be performed either before the other, or at least partially simultaneously. 【0212】 The mesh or model cleanup process described in Section I can be performed before or in conjunction with the dental restoration prediction process described in Section I, which is performed together with the dental restoration verification process described in Section I, to provide the cleaned mesh or model to the dental restoration prediction process and the dental restoration verification process. 【0213】 The mesh or model cleanup process described in Section I, along with the segmentation process described in Section II and the coordinate system process described in Section III, can be performed before or at the same time as the dental restoration prediction process described in Section I is performed along with the dental restoration verification process described in Section I, to provide the cleanup mesh or model with a segmented coordinate system to the dental restoration prediction and verification processes. 【0214】 The mesh or model cleanup process described in Section I can be performed before or together with the segmentation process described in Section II to provide the cleaned mesh or model to the segmentation process. 【0215】 The mesh or model cleanup process described in Section I can be performed before or together with the coordinate system process described in Section III to provide the cleaned mesh or model to the coordinate system process. 【0216】 The segmentation process described in Section II, along with the coordinate system process described in Section III, can be performed either before the other or at least partially simultaneously to provide a segmented mesh or model having a coordinate system. 【0217】 The mesh or model cleanup process described in Section I can be performed before, or concurrently with, the segmentation process described in Section II and the coordinate system process described in Section III, and the cleaned-up mesh or model can be provided to the segmentation process and the coordinate system process. 【0218】 The mesh or model cleanup process, the dental restoration prediction process and the dental restoration verification process described in Section I, the segmentation process described in Section II, and the coordinate system process described in Section III can be selectively used together with the provider grouping process described in Section IV when generating a customized treatment plan.

Claims

[Claim 1] Non-temporary computer-readable memory and One or more computer processors that communicate with the aforementioned memory, Equipped with, The one or more processors described above Receive a digital 3D representation of one or more intraoral structures. A first trained machine learning model is applied to the digital 3D representation, where the trained machine learning model is Steps to access a partially trained machine learning model, The steps include receiving the relevant ground truth segmentation, The steps include generating predicted segmentation using the partially trained machine learning model, The steps include calculating a loss value that quantifies the difference between the aforementioned related ground truth segmentation and the aforementioned predicted segmentation, A step of generating a trained machine learning model by modifying one or more features of the partially trained machine learning model, wherein the modification iteratively adjusts the set of weights to minimize the difference between the predicted segmentation labels and the actual segmentation labels. It was trained using The trained machine learning model outputs one or more labels for one or more features of the 3D representation. Computer system. [Claim 2] The first trained machine learning model is a neural network. The system according to claim 1. [Claim 3] The one or more features of the neural network are weights. The modification of one or more of the above features includes modifying one or more of the above weights based at least partially on the loss value. The system according to claim 2. [Claim 4] The neural network comprises at least one of one or more convolutional layers, one or more pooling layers, and one or more unpooling layers. The system according to claim 2. [Claim 5] The predicted segmentation includes at least one tooth related to the patient's dental anatomical structure. The system according to claim 1.

Images