Defect detection, mesh cleanup and mesh cleanup verification in digital dentistry

Abstract

Systems and techniques are disclosed for training one or more neural networks to automatically identify one or more aspects of a digital representation used in digital oral care, including identifying one or more aspects of a first digital representation that performs additional processing based on a list of 3D elements, generating a predicted representation by tagging that aspect of the additional processing is performed, generating an accuracy score specifying a difference between the one or more predicted representations and one or more corresponding reference representations identifying one or more aspects of the first digital representation to perform the additional processing, and modifying at least one aspect of the neural network based on the accuracy score.

Description

Technical Field

[0001] The present disclosure relates to various improved machine learning techniques for use in digital oral care including the disciplines of digital dentistry and digital orthodontics. Background Art

[0002] Dentists often use dental appliances to reshape or restore a patient's dental anatomy, or to move teeth using orthodontic appliances. These appliances are typically constructed from a model of the patient's dental anatomy that is modified to the desired end state. The model can be a physical model or a digital model. Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. Problems with past approaches include loss of mapping accuracy and inefficiencies in the data processing that produces the 2D to 3D conversion.

[0003] For example, according to existing implementations, projection operations performed by existing systems may result in 3D mesh elements receiving conflicting labels due to two or more projection operations. This may require the execution of additional machine learning models to disambiguate those conflicting labels, thereby increasing the complexity and error of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An example processing unit operating in accordance with the techniques of this disclosure is shown.

[0005] Figure 2 Example general techniques for training a generator or other neural network according to various aspects of the present disclosure are shown.

[0006] Figure 3 Example general techniques for using a trained generator or other neural network in accordance with various aspects of the present disclosure are shown.

[0007] Figure 4 Another example general technique for training a generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0008] Figure 5 Another example general technique for using a trained generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0009] Figure 6 An example machine learning architecture according to various aspects of the present disclosure is shown.

[0010] Figure 7Example techniques for performing 2D verification on dental data are shown.

[0011] Figure 8 Example general techniques for validating output generated by a machine learning model in accordance with various aspects of the present disclosure are shown.

[0012] Figures 9 to 11 Example techniques for cleaning 3D mesh geometry according to various aspects of the present disclosure are shown.

[0013] Fig.12 Example techniques for training machine learning models are shown.

[0014] Figures 13 to 16 Two types of 2D raster image views of a tooth are shown, each of which can be used to train a neural network to validate a mesh cleaning operation.

[0015] Figures 17 to 22 Shown are example results when a mesh cleaning operation is performed to remove excess material from gingival tissue.

[0016] Figure 23 to Figure 24 Examples related to the hardware removal aspect of mesh cleaning are shown.

[0017] Fig.25 Dental arch mesh data that has been labeled to serve as ground truth data for training a mesh cleaning machine learning model is shown. DETAILED DESCRIPTION

[0018] The present disclosure describes various automation techniques that can be implemented throughout the process of manufacturing dental and orthodontic appliances. Therefore, the present disclosure contemplates improvements in the field of digital oral care, including digital dentistry and digital orthodontics. The automated geometry generation techniques of the present invention are intended to simplify the manufacturing process that would otherwise be extremely time-consuming. Another advantage of these automated geometry generation techniques is that the accuracy of dental appliances is improved. In some cases, the algorithm can produce geometries with higher quality and higher accuracy than those produced by human technicians. Although in some cases, human technicians can modify or "fine-tune" the design output by the automated tool, the automated tool improves the quality of the resulting appliance by providing a common construction baseline for multiple technicians. In addition, by studying the output of the automated tools in the present disclosure (e.g., both the tools for geometry generation and the tools for geometry verification), untrained or new human technicians can understand the appropriate techniques for creating dental and orthodontic appliances (generally used as oral care appliances in this article). The transfer of knowledge to other technicians and the standardization of techniques are important benefits of the disclosed technology. Due to all of the above reasons, another advantage is that more accurate geometry and knowledge transfer can improve the restoration results associated with the use of manufactured dental or orthodontic appliances.

[0019] Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. The technology disclosed herein takes a more direct approach in that mesh elements are labeled directly without the need for intermediate 2D images and the projection of information from those 2D images onto a 3D mesh. Thus, for example, 3D mesh elements can be directly labeled for segmentation and mesh cleaning, which is not possible when using existing systems that rely on 2D mapping techniques. This direct element labeling approach improves the accuracy of the underlying machine learning (ML) model and provides greater efficiency with respect to the use of computational resources because the computational overhead of generating images and mapping them back onto 3D geometry can be avoided.

[0020] As used herein, a 3-dimensional ("3D") mesh (or 3D geometry) includes data corresponding to the edges, vertices, and faces of the 3D mesh. These edges, vertices, and faces are also referred to as digital representations such as one or more aspects of a 3D mesh. In some examples, the aspects of a 3D mesh may refer to the shape or geometric features of the mesh. In some cases, the aspects of one mesh may be compared to the aspects of another mesh, such as during a validation operation. Although these three types of data are interrelated, they are different. Vertices are points in 3D space that define the boundaries of a mesh. Therefore, if there is no additional information about how the points are connected to each other, these points can be regarded as point clouds. However, in the context of 3D meshes, edges provide structure to point clouds. An edge includes two points and may also be referred to as a line segment. A face includes both edges and vertices. For example, in the case of a triangular mesh, a face includes three vertices, where these vertices are interconnected to form three continuous edges. Although 3D meshes are typically formed using triangles, other specific implementations may use quadrilaterals, pentagons, or some other n-sided polygons to define 3D meshes. Some meshes may contain degenerate elements, such as non-manifold geometries. Non-manifold geometry is a digital geometry that cannot exist in the real world. For example, one definition of non-manifold is a 3D shape that cannot be unfolded into a 2D surface, so that the unfolded shape has all surface normal vectors pointing in the same direction. An example of a possible non-manifold geometry is that a face or edge is squeezed but not moved, which causes two identical edges to form on top of each other. Typically, such non-manifold geometry is deleted before processing can be performed. Other mesh preprocessing operations are also possible. The 3D data of each example in the present disclosure can be presented to a machine learning model as a 3D mesh and / or output from a machine learning model as a 3D mesh. Other 3D data representations include voxels, finite elements, finite differences, discrete elements, and other 3D geometric representations of dental data and / or appliances. Other specific implementations can use non-discrete methods to describe 3D geometries, thereby using mathematical formulas to regenerate the geometry during processing. Such formulas may contain expressions including polynomials, cosines, and / or other trigonometric or algebraic terms. An advantage of non-discrete formats can be to compress data and save storage space. Digital 3D data may require different coordinate systems such as XYZ (Euclidean), cylindrical, radial, and custom coordinate systems.

[0021] That is, a 3D mesh is a data structure that can describe the structure, geometry, and / or shape of an object related to oral care, including but not limited to teeth, non-organic objects (e.g., hardware elements), or gum tissue of a patient. The geometry of a 3D mesh can define aspects of the physical size, scale, and / or symmetry of the mesh. The structure of a 3D mesh can define the count, distribution, and / or connectivity of mesh elements. A 3D mesh can include one or more mesh elements, such as one or more vertices, edges, faces, and combinations thereof. In some implementations, mesh elements can include voxels, such as in the context of sparse mesh processing operations. Various spatial and structural features can be calculated for these mesh elements and provided to the predictive models of the present disclosure, with the advantage of improving the accuracy of those predictive models. For example, in some implementations, mesh element features can quantify certain aspects of a 3D mesh that is close to or associated with one or more mesh elements, as described elsewhere in the present disclosure.

[0022] Depending on the specific implementation, it may be beneficial to pre-process the information to generate one or more mesh feature elements. That is, each 3D mesh may undergo pre-processing before being input into a prediction architecture (e.g., including at least one of an encoder, a decoder, an autoencoder, a multi-layer perceptron (MLP), a transformer, a pyramid encoder-decoder, a U-Net, or a graph CNN). Such pre-processing may include converting the mesh into a list of mesh elements such as vertices, edges, faces, or voxels in the case of sparse processing. For one or more selected mesh element types (e.g., vertices), a feature vector may be generated. In some examples, a feature vector is generated for each vertex of the mesh. Each feature vector may contain a combination of spatial and / or structural features, as specified by the following table:

[0023] Consistent with Table 1, a voxel may also have features that are computed as an aggregation of other mesh elements (e.g., vertices, edges, and faces) that either intersect the voxel or, in some implementations, are primarily or completely contained within the voxel. Rotating a mesh may not change structural features, but may change spatial features. Also, as described elsewhere, the term "mesh" should be considered in a non-limiting sense to include 3D meshes, 3D point clouds, and 3D voxelized geometries. In some cases, a 3D point cloud may be derived from the vertices of a 3D triangle mesh.

[0024] Techniques that can operate on the feature vectors of the above features include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental restoration appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification. Such feature vectors can be presented to the input of a prediction model. In some embodiments, such feature vectors can be presented to one or more internal layers of a neural network that is part of one or more of those prediction models.

[0025] But a 3D mesh is only one type of 3D representation that can be used. Therefore, it should be understood that, without loss of generality, there are various types of 3D representations contemplated herein. For example, a 3D representation may include, be one or more of the following, or be a part thereof: a 3D polygonal mesh, a 3D point cloud, a 3D voxelized representation (e.g., a collection of voxels), or a 3D representation described by a mathematical equation. Although the term "mesh" is frequently used throughout this disclosure, in some specific implementations, the term should be understood to be interchangeable with other types of 3D representations. A 3D representation may describe an element of the 3D geometry and / or 3D structure of an object. And a patient's dentition may include one or more 3D representations of the patient's teeth, gums, and / or other oral anatomical structures. Depending on the specific implementation, a 3D scanner (such as an intraoral scanner, a computed tomography (CT) scanner, an ultrasound scanner, a magnetic resonance imaging (MRI) machine, or a mobile device capable of performing stereophotogrammetry) may be used to generate an initial 3D representation.

[0026] In light of the above, the techniques described herein relate to operations performed on 3D representations to perform tasks associated with geometry generation and / or verification. For example, the present disclosure relates to improved automated techniques for segmentation generation and verification, coordinate system prediction and verification, clear tray appliance setup verification, dental prosthetic appliance verification, bracket and attachment (or other hardware) placement and verification, 3D printed component verification, prosthetic design generation and verification, fixture model verification, and clear tray appliance trim line verification, to name a few examples. The present disclosure also relates to improved automated techniques for verifying many of those examples.

[0027] In general, the use of side information ensures that the machine learning model is insensitive to different input orders of 3D elements. A notable exception is the implementation of coordinate system prediction, which operates on 3D point clouds instead of 3D meshes. These and other differences are described in more detail below.

[0028] Certain examples in this disclosure mention the use of MeshCNN or encoders to process 3D mesh geometry (e.g., encoder structures for 3D verification and bracket / accessory placement, and MeshCNN for marking mesh elements in segmentation and mesh cleaning). Without limitation, each of these examples may employ other kinds of neural networks to process 3D mesh geometry in addition to or in place of the specified neural network. The following neural networks may be interchangeable in various implementations of the 3D mesh geometry examples of this disclosure: ResNet, U-Net, DenseNet, MeshCNN, Graph-CNN, PointNet, Multilayer Perceptron (MLP), PointNet++, PointCNN, and PointGCN. In other cases, an encoder structure may be used.

[0029] In some cases, the system of the present disclosure may be deployed in a clinical environment (such as a dental or orthodontic clinic) for use by clinicians (e.g., doctors, dentists, orthodontists, nurses, hygienists, oral care technicians). Such systems deployed in a clinical environment may enable clinicians to process oral care data (such as dental scans) in a clinical environment or in some cases in a "chairside" environment (e.g., in near "real time" when the patient is located in the clinical environment). A non-limiting list of examples of techniques may include: segmentation, mesh cleaning, coordinate system prediction, CTA trim line generation, restoration design generation, appliance component generation or placement or assembly, generation of other oral care grids, verification of oral care grids, setting prediction, hardware removal from tooth grids, hardware placement on teeth, estimation of missing values, clustering of oral care data, oral care grid classification, setting comparison, metric calculation, or metric visualization. In some cases, the execution of these techniques may enable patient data to be processed, analyzed, and used by clinicians in appliance creation before the patient leaves the clinical environment (this may facilitate treatment planning because feedback may be received from the patient during the treatment planning process).

[0030] The system of the present disclosure can use representation learning to train ML models. The advantages of representation learning include the fact that, in contrast to receiving inputs of variable size or structure, the generative network (e.g., a neural network that predicts a transformation) is guaranteed to receive inputs of known size and / or standard format. Representation learning can produce performance superior to other methods because the noise in the input data can be reduced (e.g., because the representation generation model extracts important aspects of the input mesh or point cloud through loss calculation or the network architecture selected for this purpose). Such loss calculation methods include KL divergence loss, reconstruction loss, or other losses disclosed herein. Representation learning can reduce the size of the data set required to train the model because the representation model learns the representation, while the generative network can focus on learning the generation task. Since meaningful features are available to the generative network, the result can be an improved model generalization. In some cases, transfer learning can first train the representation generation model. The representation generation model can then be used (in whole or in part) to pre-train subsequent models, such as the generative model (e.g., generating transformation predictions).

[0031] In some cases, federated learning may be used to train the techniques of the present disclosure. Federated learning may enable multiple remote clinicians to iteratively improve machine learning models (e.g., validation of 3D oral care representations, mesh segmentation, mesh cleaning, other techniques involving labeling mesh elements, coordinate system prediction, non-organic object placement on teeth, appliance component generation, dental restoration design generation, techniques for placing 3D oral care representations, setting predictions, generation or modification of 3D oral care representations using autoencoders, generation or modification of 3D oral care representations using transformers, generation or modification of 3D oral care representations using diffusion models, 3D oral care representation classification, imputation of missing values) while protecting data privacy (e.g., clinical data may not need to be sent "over the network" to a third party). Data privacy is particularly important for clinical data that is protected by applicable law. Clinicians may receive a copy of the machine learning model, use a local machine learning program to further train the ML model using locally available data from a local clinic, and then send the updated ML model back to a central hub or a third party. A central hub or third party can integrate updated ML models from multiple clinicians into a single updated ML model that benefits from learning from patient data recently collected at various clinical sites. In this way, new ML models can be trained that benefit from additional and updated patient data (potentially from multiple clinical sites) without that patient data actually being sent to a third party. In some cases, training on devices within a local clinic can be performed when the equipment is idle or otherwise during off-hours (e.g., when a patient is not being treated in the clinic). Equipment used in a clinical environment to collect data and / or train ML models for the techniques described herein may include intraoral scanners, CT scanners, X-ray machines, laptops, servers, desktop computers, or handheld devices (such as smartphones with image collection capabilities).

[0032] In addition to federated learning techniques, in some implementations, contrastive learning can be used to at least partially train the ML models described herein. In some cases, contrastive learning can enhance samples in a training dataset to emphasize differences between samples of different categories and / or increase similarities between samples of the same category.

[0033] Figure 1An example processing unit 102 operating according to the technology of the present disclosure is shown. The processing unit 102 provides a hardware environment for training one or more neural networks described throughout the specification. In general, and as will be described in more detail elsewhere, training the one or more neural networks is accomplished by providing one or more training data sets. Therefore, the quality and composition of the training data set for the neural network can have a significant impact on any neural network trained thereby. Dataset filtering and outlier removal can be advantageously applied to the training of neural networks of various technologies of the present disclosure (e.g., mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixed body model verification and CTA trim line verification, verification using autoencoders, and setting prediction).

[0034] In the depicted example, the processing unit includes a processing circuit system that may include one or more processors 104 and a memory 106, which in some examples provide a computer platform for executing an operating system 116, which may be, for example, a real-time multitasking operating system or other types of operating systems. In turn, the operating system 116 provides a multitasking operating environment for executing one or more software components (such as applications or other training routines). The processor 104 is coupled to one or more I / O interfaces 114, which provide an I / O interface for communicating with devices such as keyboards, controllers, display devices, image capture devices, other computing systems, and the like. In addition, the one or more I / O interfaces 114 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network. Additionally, the processor 104 may be coupled to an electronic display 108.

[0035] In some examples, processor 104 and memory 106 may be separate, discrete components. In other examples, memory 106 may be on-chip memory co-located with processor 104 within a single integrated circuit. Multiple instances of processing circuit systems (e.g., multiple processors 104 and / or memory 106) may exist within processing unit 102 to facilitate parallel execution of applications and / or processes (including applications and / or processes related to machine learning). Multiple instances may be of the same type, such as a multi-processor system or a multi-core processor. Multiple instances may be of different types, such as a multi-core processor with associated multiple graphics processor units (GPUs). In some examples, processor 104 may be implemented as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or equivalent discrete or integrated logic circuit systems, or a combination of any of the foregoing devices or circuit systems.

[0036] Figure 1 The architecture of the processing unit 102 shown in the figure is shown for example purposes only. The processing unit 102 should not be limited to the example architecture shown. In other examples, the processing unit 102 can be configured in various ways. The processing unit 102 can be implemented as any suitable computing system (e.g., at least one server computer, workstation, mainframe, appliance, cloud computing system, and / or other computing system) capable of performing the operations and / or functions described according to at least one aspect of the present disclosure. As an example, the processing unit 102 may represent a cloud computing system, a server computer, a desktop computer, a server farm, and / or a server cluster (or a portion thereof). In other examples, the processing unit 102 may represent or be implemented by at least one virtualized computing instance (e.g., a virtual machine or container) of a data center, a cloud computing system, a server farm, and / or a server cluster. In some examples, the processing unit 102 includes at least one computing device, each computing device having a memory 106 and at least one processor 104.

[0037] Storage unit 134 may be configured to store information within processing unit 102 during operation (e.g., 3D geometry, transformations to be performed on the 3D geometry, etc.). Storage unit 134 may include a computer-readable storage medium or a computer-readable storage device. In some examples, storage unit 134 includes at least short-term memory or long-term memory. Storage unit 134 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetic disk, an optical disk, a flash memory, a magnetic disk, an optical disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable memory (EEPROM).

[0038] In some examples, storage unit 134 is used to store program instructions for execution by processor 104. Storage unit 134 can be used by software or applications running on processing unit 102 to store information during program execution and to store results of program execution. For example, storage unit 134 can store any number of neural networks 110a-110n, including those described herein. According to some implementations, neural networks 110a-110n can be neural networks trained according to the techniques disclosed herein. In other implementations, one or more of neural networks 110a-110n can be untrained or partially trained.

[0039] As will be described in more detail elsewhere, machine learning models (e.g., one or more neural networks) can be trained in a supervised and unsupervised manner. Supervised models that can be trained to make recommendations described herein include: regression models (such as linear regression), decision trees, random forests, boosting, Gaussian processes, k-nearest neighbors (KNN), logistic regression, naive Bayes, gradient boosting algorithms (e.g., GBM, XGBoost, LightGBM, and CatBoost), support vector machines (SVM), or fully connected neural network models that have been trained for classification. In some cases, a multilayer perceptron (MLP) can be used to predict missing program parameters based on known program parameters.

[0040] Unsupervised models that can be trained to make the recommendations described herein include clustering techniques such as K-means clustering, density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture models, balanced iterative reduction and clustering based on hierarchy (BIRCH), affinity propagation clustering, mean shift clustering, ordering points to identify cluster structures (OPTICS), agglomerative hierarchical clustering, and spectral clustering.

[0041] Whether the training is supervised or unsupervised, there are a variety of optimization methods that can be used to train the neural network of the present disclosure (e.g., update the neural network weights), including gradient descent (which uses first-order derivatives to determine the training gradient and is commonly used in the training of neural networks), Newton's method (which can use second-order derivatives in the loss calculation to find a better training direction than gradient descent, but may require the calculation of the Hessian matrix), and conjugate gradient method (which may converge faster than gradient descent, but does not require the Hessian matrix calculation that may be required by Newton's method). In some specific implementations, in addition to or instead of the aforementioned methods, other methods may be used to update the weights. These other methods include: Levenberg-Marquardt method and simulated annealing. The back propagation algorithm is used to transmit the results of the loss calculation back to the network so that the network weights can be adjusted to perform learning.

[0042] Neural networks facilitate the functionality of many applications of the present disclosure, including, but not limited to: mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of custom oral care appliance components, placement of custom oral care appliance components, verification of custom oral care appliances (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders. The neural networks of the present disclosure may embody portions or all of a variety of different neural network models. Examples include U-Net architecture, multi-layer perceptron (MLP), transformer, pyramid architecture, recurrent neural network (RNN), autoencoder, variational autoencoder, regularized autoencoder, conditional autoencoder, capsule network, capsule autoencoder, stacked capsule autoencoder, denoising autoencoder, sparse autoencoder, conditional autoencoder, long / short term memory (LSTM), gated recurrent unit (GRU), deep belief network (DBN), deep convolutional network (DCN), deep convolutional inverse graphics network (DCIGN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), neural Turing machine (NTM) and generative adversarial network (GAN). In some implementations, an encoder structure or a decoder structure can be used. Each of these models has its own specific advantages. A specific model may be particularly suitable for one or another model.

[0043] In some implementations, the neural networks of the present disclosure may be adapted to operate on 3D point cloud data (alternatively 3D meshes or 3D voxelized representations). Many neural network implementations may be applicable to the processing of 3D representations and may be applied to training predictive and / or generative models for oral care applications, including: PointNet, PointNet++, SO-Net, Spherical Convolution, Monte Carlo Convolution and Dynamic Graph Networks, PointCNN, ResNet, MeshNet, DGCNN, VoxNet, 3D-ShapeNets, Kd-Net, Point GCN, Grid-GCN, KCNet, PD-Flow, PU-Flow, MeshCNN, and DSG-Net. Oral care applications include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of custom oral care appliance components, placement of custom oral care appliance components, verification of custom oral care appliance components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification and CTA trim line verification, verification using autoencoders, setup prediction, and generation of dental prosthetic appliances.

[0044] In some specific implementations, some techniques of the present disclosure may use autoencoders. Possible autoencoders include, but are not limited to: AtlasNet, FoldingNet, and 3D-PointCapsNet. Some autoencoders may be implemented at least in part based on PointNet.

[0045] Some technologies of the present disclosure relate to hardware placement. Representation learning can be used to enhance machine learning models for hardware placement. For example, representation learning can involve training a first neural network to learn representations of teeth, and training the same neural network or a second neural network to learn representations of hardware, and then using a third neural network to generate transformations for the hardware to place the hardware on the teeth. In other implementations, one or more appliance components can be placed relative to one or more teeth. Some implementations can use U-Net to generate representations. Some implementations can use autoencoders (such as VAEs or capsule autoencoders) to learn representations of basic features of one or more meshes related to the field of oral care (including information about the structure of tooth meshes in some cases). Then, the representation (latent vector or latent capsule) can be used as an input to a module that generates one or more transformations for one or more hardware elements or appliance components. In some implementations, these transformations can place hardware elements or appliance components into the posture required for appliance generation (e.g., dental restoration appliances or indirect bonding trays). In some implementations, the transformation can be described by a 9x1 transformation vector (e.g., specifying a translation vector and a quaternion). In other implementations, the transformation can be described by a transformation matrix (e.g., a 4x4 affine transformation matrix). In some implementations, a principal component analysis can be performed on the oral care grid, and the resulting principal components can be used as at least part of a representation of the oral care grid in a later machine learning and / or other predictive or generative process.

[0046] Depending on the particular implementation, additional methods may also be used to improve the performance of the machine learning model. For example, end-to-end training may be applied to the disclosed technology involving two or more neural networks, where the two or more neural networks are trained together (e.g., weights are updated simultaneously during the processing of each batch of input oral care data). In some implementations, end-to-end training may be applied to hardware / component placement by simultaneously training a neural network that learns representations of one or more oral care objects and a neural network that can process those representations.

[0047] Another way to improve the machine learning models described herein is to use transfer learning. In some specific implementations, a network (e.g., U-Net) can be trained for a first task (e.g., coordinate system prediction) and then used to provide one or more starting neural network weights for the training of another neural network, which is trained to perform a second task (e.g., setting prediction). The first network can learn low-level neural network features of an oral care grid and is shown to perform well in the first task. By using the first network as a starting point for training, the second network can experience faster training and / or improved performance. Certain layers can be trained to encode the neural network features of the oral care grid in the training data set. These layers can then be fixed (or receive small fine-tuning during training) and combined with other neural network components (such as additional layers) that are trained for one or more oral care tasks. In this way, a portion of a neural network used for one or more technologies of the present disclosure can receive initial training for another task, which can produce important learning in the trained network layers. This encoded learning can then be built on with further task-specific training. In some implementations, a neural network used to make predictions based on an oral care grid can first be partially trained on one or more general / publicly available datasets and then further trained on oral care data.

[0048] In some embodiments, a neural network previously trained on a first data set (oral care data or other data) may subsequently receive further training on oral care data and be applied to oral care applications (such as mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances or components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders). Transfer learning may be used to further train any of the following networks from published literature: GCN (graph convolutional network), PointNet, ResNet, or any other neural network from published literature listed earlier in this section.

[0049] Yet another approach involves adding an attention gate to a machine learning model. In general, an attention gate may be integrated with one or more of the neural networks of the present disclosure, with the advantage of enabling the associated neural network architecture to focus on one or more input values. In some implementations, an attention gate may be integrated with a U-Net architecture, with the advantage of enabling the U-Net to focus on certain inputs. An attention gate may also be integrated with an encoder or autoencoder, such as a VAE or capsule autoencoder. Some implementations of the presently disclosed techniques may benefit from one or more attention layers in a transformer, where the transformer is trained to generate a 3D oral care representation.

[0050] Figure 2 is an example technique 200 that can be used to train the machine learning models described herein. In general, the receiving module 202 is configured to receive patient case data 204. Typically, the patient case data 204 represents a digital representation of a patient's mouth. This may mean, for example, that the receiving module 202 may receive one or more malocclusion arches (e.g., 3D meshes representing upper and lower arches of a patient's teeth, i.e., a dentition of a patient's mouth including multiple aspects of the patient's dental anatomy, which may include teeth and may include gums). Depending on the specific implementation, the malocclusion arches may be arranged in an occlusal position or other orientation. In other implementations, a single arch may be required. For purposes of illustration, additional implementations will be described in more detail below. In other words, the receiving module 202 may receive mesh data corresponding to a 3D mesh of one or more patient's dentitions. It should be understood that both the amount of 3D mesh data and the type of 3D mesh data received by the receiving module 202 as part of the patient case data may vary based on the specific implementation. For example, in an implementation regarding verification of bracket placement, the mesh data received as part of the patient case data 204 may include only 3D mesh data regarding a particular tooth and associated bracket, while in an implementation regarding verification of a 3D printed component, the 3D data received as part of the patient case data 204 may include 3D mesh data associated with a component examined in the form of a CT scan or other diagnostic image, to name a few additional examples. Depending on the particular implementation, the patient case data 204 may also include a 3D representation of the patient's gingival tissue.

[0051] As shown in the example, the receiving module 202 also receives "reference truth" data 206. Generally speaking, these "reference truth" data 206 specify the expected results of applying other technologies disclosed herein, which are mesh segmentation, coordinate system prediction, mesh cleaning, restoration design and bracket / attachment placement, as well as all verification applications of the present disclosure, just to name a few examples. "Reference truth" and "reference" used herein will be used interchangeably. For example, it should be understood that for the purposes of the present disclosure, a "reference" transformation vector is equivalent to a "reference truth" transformation vector. According to a specific implementation, and as will be described in more detail below, the "reference truth" data 206 may include a "reference truth" unique hot vector describing the expected transformation of a 3D geometry. For another example, the "reference truth" data 206 may include expected labels for various aspects of the 3D geometry. Other examples are also provided below. According to a specific implementation, the "reference truth" data 206 may be predefined or provided as a result of executing one or more other technologies disclosed herein.

[0052] Depending on the particular implementation, the receiving module 202 may also be configured to perform data augmentation on one or more aspects of the received data, including the patient data 204 and the "ground truth" data 206. Data augmentation is described in more detail below.

[0053] The system 100 can be configured to provide each mesh received by the receiving module 202 to the mesh preprocessor module 205, thereby allowing for preprocessing of any 3D mesh data received in the patient case data 206. This preprocessing step allows the system to convert the mesh into a form that allows the input mesh to be "consumed" by a neural network or other machine learning technique. In one specific implementation, the mesh preprocessor module 205 can be configured to generate a combination of edge, vertex, and face lists. One or more of these generated lists can be provided to both the generator 211 and the mesh feature module 208, as described in more detail below.

[0054] In addition to utilizing the mesh preprocessor module 205, the system 100 may also perform a number of additional operations before and after providing the patient case data 204 to the mesh preprocessor module 205. For example, according to a particular implementation, the system 100 may perform mesh cleaning on the patient case data 204 before providing the patient case data 204 to the mesh preprocessor module 205. Additionally, the system 100 may resample or update any information generated by the mesh preprocessor module 205. For example, in an implementation in which the mesh preprocessor module 205 generates a combination of edge, vertex, and face lists, the system may resample, update, or otherwise modify the labels identified in those lists. Additionally, according to a particular implementation, the system 100 may perform data augmentation on the resampled data.

[0055] The mesh feature module 208 may be configured to receive the list generated by the mesh preprocessor module 205 and generate feature information associated therewith, which may be used by the machine learning model to generate predictions. For example, in one implementation, the mesh feature module 208 may calculate one or more of the following: edge midpoints, edge curvatures, edge normal vectors, edge normalized vectors, edge translation vectors, and other information related to each tooth in the 3D mesh received by the receiving module 202. Depending on the particular implementation, the mesh feature module 208 may or may not be utilized. That is, it should be understood that the calculation of any of the edge midpoints, edge curvatures, edge normal vectors, and edge translation vectors of the 3D mesh data included in the patient data 206 is optional. One advantage of using the mesh feature module 208 is that a system utilizing the mesh feature module 208 may be trained more quickly and accurately, but the technique 200 still performs better than prior art techniques that do not utilize the mesh feature module 208.

[0056] The technique 200 also utilizes a generative adversarial network ("GAN") to achieve certain aspects of the improvements. Generally speaking, a GAN is a machine learning model in which two neural networks "compete" with each other to provide predictions, these predictions are evaluated, and then the evaluations of the two models are used to improve each other's training. In some implementations, the GAN can be a conditional GAN, in which the generated output is adjusted based on some input data. One example in which conditional GANs have been found to provide benefits is in the field of restoration design. In those implementations, these adjusted input data can be an unrestored mesh and associated textual specifications. In some implementations, and as will be described below, the textual specifications can be a process that uses natural language processing (NLP) to extract key values, such as an added height or added width that has been specified for each treated tooth (e.g., in the example of dental restoration design, which produces a target geometry for each treated tooth).

[0057] As shown in this example, the two neural networks of the GAN are the generator 211 and the discriminator 235. In other implementations, models other than neural networks can be used for the generator or the discriminator.

[0058] The generator 211 receives input (e.g., one or more 3D meshes included in the patient case data 206). Depending on the particular implementation, the generator 211 uses the received input to determine a predicted output 207 related to the 3D mesh. For example, for segmentation, the generator 211 may be configured to predict a segmentation label, and in an implementation of predicting a clear tray appliance setting, the prediction may include one or more vectors corresponding to one or more transformations applied to the 3D mesh included in the patient case data 206. Other predicted outputs 207 are also possible. In some implementations, the generator 211 may also receive random noise, which may include junk data or other information that may be used to purposefully attempt to confuse the generator 211. Depending on the particular implementation, and as described above, the generator 211 may implement any number of neural networks, including MeshCNN, ResNet, U-Net, and DenseNet. In other cases, the generator may implement an encoder.

[0059] Since the generator 211 can be implemented as one or more neural networks, the generator 211 can include an activation function. The activation function determines whether a neuron in the neural network will fire (e.g., send an output to the next layer). Some activation functions may include: a binary step function and a linear activation function. Other activation functions give the network nonlinear behavior, including: a sigmoid / logistic activation function, a Tanh (hyperbolic tangent) function, a rectified linear unit (ReLU), a leaky ReLU function, a parametric ReLU function, an exponential linear unit (ELU), a softmax function, a swish function, a Gaussian error linear unit (GELU), and a scaled exponential linear unit (SELU). The linear activation function is well suited for some regression applications (and other applications) in the output layer. In the output layer, the sigmoid / logistic activation function is well suited for some binary classification applications (and other applications). The sigmoid activation function is well suited for some multi-class classification applications (and other applications) in the output layer. In the output layer, the sigmoid activation function is well suited for some multi-label classification applications (and other applications). The ReLU activation function works well for some Convolutional Neural Network (CNN) applications in the hidden layers (among other applications). The Tanh and / or sigmoid activation functions work well for some Recurrent Neural Network (RNN) applications, for example, in the hidden layers (among other applications).

[0060] After the generator 211 determines one or more predicted outputs 207, the generator 211 may be trained. Generally speaking, training the generator 211 involves comparing the predicted outputs 207 to corresponding ground truth inputs 208. For example, the predicted output 207 associated with the lower left canine corresponding to number twenty-seven of the International Dental Numbering System is compared to the ground truth output 208 for the same canine. As previously described, the ground truth input is an input that has been verified as a correct label for a particular portion of the 3D mesh data included in the patient case data 206. Depending on the particular implementation, the ground truth input 208 may be derived or otherwise determined from the ground truth data 206, or may be the ground truth data 206.

[0061] The differences between the predicted output 207 and the reference true input 208 can be used to calculate one or more loss values ​​G1216. For example, these differences can be used as part of the loss function calculation, or for the calculation of reconstruction errors. Some specific implementations may involve comparing the volumes and / or areas of two meshes (i.e., representations 207 and 208). Some specific implementations may involve calculating the minimum distance between corresponding vertices / faces / edges / voxels of the two meshes. For a point in one mesh (e.g., a vertex, a midpoint on an edge, or a triangle center), the minimum distance between the point and the corresponding point in the other mesh is calculated. Different methods can be considered in the case where the other mesh has a different number of elements or there is no clear mapping between corresponding points of the two meshes.

[0062] Regardless of the manner in which the difference between the predicted output 207 and the ground truth input is determined, various loss values ​​may be determined as part of the technique 200 or any other technique described herein. These losses include L1 loss, L2 loss, MSE loss, cross entropy loss, etc. Losses may be calculated and used to train neural networks such as multi-layer perceptrons (MLPs), U-Net structures, generators and discriminators (e.g., for GANs), autoencoders, variational autoencoders, regularized autoencoders, masked autoencoders, transformer structures, etc. For example, in learning sequences, some implementations may use triplet loss or contrastive loss.

[0063] The loss can also be used to train the encoder structure and the decoder structure. The KL divergence loss can be used at least in part to train one or more neural networks of the present disclosure, such as a mesh reconstruction autoencoder, which has the advantage of imparting Gaussian behavior to the optimization space. This Gaussian behavior can enable the reconstruction autoencoder to produce better reconstructions (i.e., when the latent vector representation is modified and the modified latent vector is reconstructed using the decoder, the resulting reconstruction is more likely to be a valid instance of the input representation). There are other techniques for calculating the loss that can be described elsewhere in the present disclosure. This loss can be based on quantizing the difference between two or more 3D representations.

[0064] The mean squared error (MSE) loss may involve the computation of the average squared distance between two sets, vectors, or data sets. The MSE may generally be minimized. The MSE may be applied to regression problems where the predictions generated by a neural network or other machine learning model may be real numbers. In some implementations, the neural network may be equipped with one or more linear activation units at the output to generate the MSE predictions. Mean absolute error (MAE) loss and mean absolute percentage error (MAPE) loss are also possible.

[0065] In some implementations, cross entropy can be used to quantify the difference between two or more distributions. In some implementations, cross entropy loss can be used to train the neural network of the present disclosure. In some implementations, cross entropy loss may involve comparing the predicted probability with the benchmark true probability. Other names for cross entropy loss include "logarithmic loss", "logistic loss" and "log loss". A small cross entropy loss may indicate a better (ie, more accurate) model. The cross entropy loss may be logarithmic. In some implementations, cross entropy loss may be applied to binary classification problems. In some implementations, the neural network may be equipped with a sigmoid activation unit at the output to generate a probability prediction. In the case of multi-class classification, cross entropy may also be used. In this case, in some implementations, a neural network that has been trained to perform multi-class predictions may be equipped with one or more softmax activation functions at the output (e.g., where there is an output node for the class to be predicted).

[0066] Other loss calculation techniques that can be applied in the training of the neural network of the present disclosure include one or more of the following: Huber loss, hinge loss, classification hinge loss, cosine similarity, Poisson loss, Logcosh loss, or mean square logarithmic error loss (MSLE). Other loss calculation methods are described herein and can be applied to the training of any neural network described in the present disclosure.

[0067] In some implementations, one or more neural networks of the present disclosure may be trained, at least in part, by a loss based on at least one of: point-wise mesh Euclidean distance (PMD) and earth mover distance (EMD). Some implementations may incorporate a Hausdorff distance (HD) calculation into the loss calculation. Calculating the Hausdorff distance between two or more 3D representations (such as 3D meshes) may provide one or more technical improvements because the HD considers not only the distance between the two meshes, but also the way those meshes are oriented and the relationship between the shapes of the meshes in those orientations (or positions or poses). The Hausdorff distance may improve comparisons of two or more instances of two or more tooth meshes, such as tooth meshes in different poses (e.g., comparisons of predicted settings to baseline true settings, which may be performed in the process of calculating a loss value for training a setting prediction neural network).

[0068] See again Figure 2 , G1 216 may represent the regression loss between the predicted output 207 and the ground truth input 208. That is, according to one implementation, the loss G1 216 reflects the percentage by which the predicted output 207 deviates from the ground truth input 208. That is, the generator loss G1 216 may be an L2 loss, a smoothed L1 loss, or some other type of loss. According to a particular implementation, the L1 loss is defined as , where P represents the predicted output 207 and G represents the ground truth input 208. Depending on the specific implementation, the L2 loss can be defined as , again where P represents the predicted output 207 and G represents the reference true input 208. In addition, and as will be described in more detail below, a loss value G1 216 may be provided to the generator 211 to further train the generator 211, for example, by modifying one or more weights in the neural network of the generator 211 to train the underlying model and improve the model's ability to generate a predicted output 207 that mirrors or substantially mirrors the reference true input 208. Any of these losses may be used to provide a loss value for training the neural network through a suitable training algorithm (such as back propagation). In some cases, an accuracy score may be used for training the neural network. The accuracy score quantifies the difference between the predicted data structure and the reference true data structure. The accuracy score (e.g., in normalized form) may be fed back into the neural network during the training of the network, for example, through back propagation. In the case of segmentation, the accuracy score may count the matching labels between the predicted grid and the reference true grid (i.e., where each grid element has an associated label). The higher the percentage of matching labels, the better the prediction (i.e., when comparing the predicted labels to the reference true labels). A similar accuracy score can be calculated in the case of mesh cleaning, which also predicts labels for mesh elements. The number or percentage of matches between the predicted labels and the ground truth labels can be used as an accuracy score, which can be used to train the neural network driving mesh cleaning (i.e., the accuracy score can be normalized).

[0069] Additionally, depending on the particular implementation, the system 100 can use the predicted output 207 to generate a predicted representation 220. Furthermore, the system 100 can use the reference ground truth input 208 to generate a reference ground truth representation 211. For example, in an implementation related to clear tray appliance generation, the predicted transformation and the reference ground truth transformation can be applied to the patient case data 206 to generate a predicted transformation and a reference ground truth transformation of the patient case data 206.

[0070] Depending on the particular implementation, the predictive representation 220 and the ground truth representation 211 may be labeled or otherwise annotated to indicate whether the representation corresponds to the ground truth data 206. Additionally, depending on the particular implementation, the representation 220 may be assigned a "false" value to indicate that the representation does not correspond to the ground truth label 208, while the representation 221 may be assigned a "true" value.

[0071] According to a particular implementation, representations 220 and 221 are provided as inputs to discriminator 235. Additionally, according to a particular implementation, 3D mesh data in patient case data 206 is also provided to discriminator 235. That is, discriminator 235 may receive various representations of data corresponding to patient case data 206, predicted output 207, ground truth data 206, ground truth input 208, and representations 220 and 221. In general, discriminator 235 is configured to determine when an input is generated from predicted output 207 or when an input is generated from ground truth input 208. The output of discriminator 235 is described in more detail in conjunction with the implementations discussed herein.

[0072] The discriminator 235 may be initially trained in various ways. For example, the discriminator 235 may be configured as an encoder structure, which in some cases (such as those described herein) may be configured to perform verification when used as a generator. For example, the initial encoder included in the discriminator 235 may be configured with random edge weights. Using back propagation, the encoder and thus the discriminator 235 may be continuously improved by modifying the value of the weights to allow the discriminator 235 to more accurately determine which inputs should be identified as "true" reference true representations and which inputs should be identified as "false" reference true representations. In other words, while the discriminator 235 may be initially trained, the discriminator 235 continues to develop / be trained as the technique 200 is executed. And similar to the generator 211, the accuracy of the discriminator 235 improves with each execution of the technique 200. Although, as will be appreciated by one of ordinary skill in the art, the improvement to the discriminator 235 will reach a limit where the accuracy of the discriminator 235 is not statistically improved, the training of the discriminator 235 is considered to be complete at this point. In other words, the system 100 may consider the training of both the generator 211 and the discriminator 235 to be complete when the discriminator 235 has difficulty distinguishing the predicted representation 220 from the reference true representation 221. As used herein, when the training of the generator 211 and the discriminator 235 is complete, they are described as being fully trained.

[0073] After the discriminator 235 generates an output, the technique 200 then compares the output of the discriminator 235 with the input to determine whether the discriminator 235 accurately distinguishes the predicted representation 220 from the reference ground truth representation 221. For example, the output of the discriminator 235 can be compared to the annotation of the representation. If the output and the annotation match, then the discriminator 235 accurately predicts the type of input received by the discriminator 235. Conversely, if the output and the annotation do not match, then the discriminator 235 inaccurately predicts the type of input received by the discriminator 235. In some implementations, and similar to the generator 211, the discriminator 235 can also receive random noise to intentionally attempt to confuse the discriminator 235.

[0074] In addition, depending on the particular implementation, the discriminator 235 may generate additional values ​​that can be used to train various aspects of the system implementing the technique 200. In one example, the discriminator 235 may generate a discriminator loss value 236 that reflects how accurately the discriminator 235 determines whether an input corresponds to the predicted representation 220 and / or the ground truth representation 221. Depending on the particular implementation, the discriminator loss 236 is larger when the prediction of the discriminator 235 is less accurate, and smaller when the prediction of the discriminator 235 is more accurate. In another example, the discriminator 235 may generate a generator loss value G2 238. Depending on the particular implementation, although not directly inversely proportional to the discriminator loss 236, the generator loss value G2 238 generally exhibits an inversely proportional relationship with the discriminator loss 236. That is, when the discriminator loss 236 is larger, the generator loss G2 238 is smaller, and when the discriminator loss 236 is smaller, the generator loss G2 238 is larger. In some implementations, the discriminator loss 236 may be determined using a binary cross entropy loss function calculated for both the "true" model and the "fake" model. In some implementations, the generator loss may consist of two losses: 1) the first loss is the generator loss G2 238 as determined by the discriminator (so binary cross entropy may be used); and 2) the second loss may be implemented by the l1-norm or mean squared error measuring the difference between the desired output and the actual output of the generator 211, for example as specified by the generator loss G1 216.

[0075] In other words, Figure 2 As shown, the generator loss G2 238 can be added to the generator loss G1 216 using a summation operation 240. And the sum of the generator loss G1 216 and G2 238 can be provided to the generator 211 for the purpose of training the generator 211. That is, it should be understood that the calculation of the generator loss G1 216 is for Figure 2Training of the GAN shown is not required. In some implementations, it is possible to train the generator 211 or the discriminator 235 using only a combination of the generator loss G2 238 and the discriminator loss 236. However, like other optional aspects of the present disclosure, the use of the generation loss G1 216 can be used to train the discriminator 235 more quickly to produce more accurate predictions. Depending on the specific implementation, the system 100 may use other steps or operations as part of the described techniques. For example, as already described but not depicted, an implementation related to the clear tray-type aligner setting may use one or more transformation steps to transform the patient data 206 using a predicted output 207 and a reference real input 208 corresponding to one or more 3D mesh transformations (e.g., scaling, rotation, and / or translation operations).

[0076] Depending on the particular implementation, the loss G1 216 and the loss G2 238 may also include one or more reasoning metrics specifying one or more differences between the predicted output 207 and the ground truth input 208 and / or the predicted representation 202 and the ground truth representation 221. That is, as an optional step, the system 100 may generate these reasoning metrics to further refine the training of one or more neural networks or machine learning models. These reasoning metrics may include: intersection over union metric, average boundary distance metric, boundary percentage metric, and over segmentation rate, to name a few examples.

[0077] In general, the intersection-over-union metric specifies the percentage of correctly predicted edges, faces, and vertices within a mesh after a certain operation (such as segmentation) is completed. The average boundary distance specifies the distance between the predicted output 207 (or predicted representation 220) and the reference true input 208 (or reference true representation 221) for a 3D representation (such as a 3D mesh). The boundary percentage specifies the percentage of the mesh boundary length of a 3D mesh (such as a segmented 3D mesh) where the distance between the reference true input 208 (or reference true representation) and the predicted output 207 (or predicted representation 220) is below a threshold. For example, the threshold may determine whether one or more predicted outputs 207 (such as a small line segment between each pair of boundary points) are close enough to the reference true input 208. In the case where the technique 200 is used to implement a segmentation process, if the distance is below the threshold, the system 100 may mark a particular line segment as a perfect boundary segment. The percentage represents the ratio of segments that are within the predicted boundary to segments that are within the reference true boundary. And the over-segmentation rate specifies the percentage of the boundary length of the teeth that are over-segmented. Depending on the specific implementation, one or more inference metrics can be used to additionally train the generator 211 or the discriminator 235, or both.

[0078] The technology of the present disclosure may include operations such as 3D convolution, 3D pooling, 3D deconvolution, and 3D up-pooling. 3D convolution can help segmentation processing, such as when downsampling a 3D representation (such as a 3D mesh or point cloud). 3D deconvolution is, for example, the inverse operation of 3D convolution in U-Net. 3D pooling can help segmentation processing, such as in a generalized neural network feature map. 3D up-pooling is, for example, the inverse operation of 3D pooling in U-Net. These operations can be implemented by one or more layers in the prediction or generation neural network described herein. These operations can be directly applied to aspects of the 3D representation such as mesh elements, which may include mesh edges or mesh faces. These operations provide technical improvements over other methods because these operations are invariant to mesh rotation, scaling, and translation changes. Generally speaking, these operations depend on edge (or face) connectivity, so as long as the edge (or face) connectivity is maintained, these operations will not be affected by mesh changes in 3D space. That is, these operations can be applied to an oral care mesh and produce the same output regardless of the orientation, position, or scale of the oral care mesh, which can improve data accuracy. MeshCNN is a general deep neural network library for 3D triangulated meshes that can be used for tasks such as 3D shape classification or mesh element labeling (e.g., for segmentation or mesh cleaning). MeshCNN implements these operations on mesh edges. Other toolkits and specific implementations can operate on edges or faces.

[0079] The technique 200 can be used to train machine learning models for many digital dental and digital orthodontic applications. Table 2 shows how the technique 200 can receive different data 204 and 206 for certain digital dental applications, and the form that the prediction output 207 can take depending on the particular implementation.

[0080] Machine learning models (such as those described herein) can be trained to generate transformations to place prefabricated components (e.g., from a component library) for creating dental restoration appliances. Such dental restoration appliances can be used to shape dental composites in a patient's mouth while curing the composites (e.g., using a curing light), ultimately producing veneers on one or more teeth of the patient. 3M FILTEK Matrix is ​​an example of such a product. Dental restoration appliance components (e.g., library components) that can be placed using the technology of the present disclosure include: drainage holes (e.g., which can allow composites to flow out of the appliance), rear snap clips (e.g., which can enable the appliance to be grasped or manipulated), door hinges (e.g., which can enable the door to be rotated open or closed), door snaps (e.g., which can secure the door in a closed position), incisal alignment features (e.g., which can help align the appliance), center clips (e.g., which can enable the appliance to be aligned), custom labels, manufacturing box frames, diastema matrix handles, and the like. Further details on placing features and generating features can be found in PCT patent application WO2021 / 240290A1, the entirety of which is incorporated herein by reference.

[0081] For example, in a segmentation implementation, each patient case in the data set 204 consists of a pre-segmented dental arch. In some implementations, the technique 200 can be used to segment each tooth in the dental arch and label the tooth with its identity (i.e., performing traditional tooth segmentation). In some implementations, the technique 200 can be used to separate the face and tongue of the dental arch (i.e., performing face-tongue segmentation). In some implementations, the technique 200 can be used to separate the gingival portion of the dental arch from the teeth (i.e., performing gingival segmentation). In some implementations, the technique can be used to segment extraneous material directly from the gums (i.e., performing trim line segmentation). Some segmentation implementations may use MeshCNN to predict mesh element labels. Some implementations may train a U-Net structure to generate a representation of a 3D mesh and may also be simultaneously trained to predict mesh element labels. Other implementations may use other models to predict mesh element labels.

[0082] As described elsewhere in this specification, the receiving module 202 receives patient case data. In the depicted example, the receiving module 202 may receive patient case data 204 including dental arch data after one or more mesh cleaning operations have been performed on the patient's 3D dental arch geometry. For example, in one example, this may produce one or more cleaned dental arch geometries. The mesh cleaning operations may use one or more models of MeshCNN, U-Net, or other models to predict mesh element labels.

[0083] According to a particular implementation, the 3D dental arch geometry may include a 3D mesh geometry for the patient's gingival tissue, while in other implementations, the 3D dental arch geometry may omit the 3D dental arch geometry for the patient's gingival tissue. In addition, the receiving module 202 may be configured to also receive a reference true label as a reference true label 206, which describes a verified or otherwise known accurate label (e.g., labels "correct" and "incorrect") of a mesh element associated with a segmentation result performed on the 3D geometry. According to a particular implementation, the labels described with respect to the segmentation operation are used to specify a specific set of mesh elements (such as "edge" elements, "face" elements, "vertex" elements, etc.) for specific aspects of the 3D geometry. For example, a single triangular polygon of a 3D mesh includes 3 edge elements, 3 vertex elements, and 1 face element. Therefore, it should be understood that a segmented dental geometry consisting of many polygons may have a large number of labels associated with the segmented dental geometry.

[0084] In addition, the received geometry may have one or more labels that are applied to the corresponding geometry to generate representations 220 and 221. For example, in one embodiment, at each iteration of generator 211, generator 211 may output a label for each mesh element found in the input dental arch. Each of these labels marks the corresponding mesh element (e.g., an edge) as belonging to a gingival or tooth structure in the input mesh. In the case where the mesh element belongs to a tooth, the identity of the tooth is also specified. For example, an edge may be assigned a label to indicate that the mesh element belongs to the gingiva. Another mesh element may be assigned a label to indicate that the mesh element belongs to the upper right third molar. Yet another mesh element may be assigned a label to indicate that the mesh element belongs to the lower left central incisor. Other labels are also possible.

[0085] Once trained, the generator 211 may be used to generate accurate prediction outputs 207 for the patient case data 206 received by the receiving module 202 . Figure 3 An example technique 300 for generating predicted labels 207 is shown in FIG. Generally speaking, technique 300 uses the same computer modules and components to perform many of the same steps as technique 200. That is, as can be seen from this example, technique 300 does not train generator 211, but relies on the training in technique 200 to generate predicted output 307. In addition, technique 300 does not include a discriminator. As discussed above with respect to Figure 2 It should be understood from the discussion of that when generator 211 is trained, predicted output 207 will eventually be equal to or substantially equal to predicted output 307.

[0086] Some of the techniques described in Table 2 (and elsewhere in this disclosure) can benefit from training a representation learning model. In some implementations, such a representation model can be used to implement Figure 2, Figure 3 , Figure 4 and Figure 5 211 in the generator. In some embodiments, the representation learning model may include a first module and a second module, the first module may be trained to generate representations of received 3D oral care representations (e.g., teeth, gums, hardware and / or appliance components), and the second module may be trained to receive those 3D representations and generate one or more output oral care representations. In some cases, such output oral care representations may include transformations that may be applied to hardware or appliance components for placement relative to one or more teeth. In some cases, such output oral care representations may include one or more coordinate system axis definitions. In some cases, such output oral care representations may include meshes or labels on mesh elements corresponding to teeth, gums, or other aspects of the dentition (e.g., utilizing mesh cleaning, mesh segmentation, or dental restoration design).

[0087] In some implementations, a first module of a representation learning model may be trained to generate a 3D representation of one or more teeth (and / or gums or hardware) suitable for providing to a second module, wherein the second module is trained to output one or more predicted transforms (or other oral care representations). In some implementations, one or more layers including convolution kernels (e.g., with kernel size 5 or some other size) and pooling operations (e.g., average pooling, max pooling, or some other pooling method) may be trained to create representations for one or more received oral care 3D representations in the first module. In some implementations, one or more U-Nets may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more autoencoders may be trained to generate representations for one or more received oral care 3D representations in the first module (e.g., where a 3D encoder of the autoencoder is trained to convert one or more tooth 3D representations into one or more latent representations, such as latent vectors or latent capsules, wherein such latent representations may be reconstructed into a facsimile of one or more input tooth meshes via a 3D decoder of the autoencoder). In some implementations, one or more 3D encoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more pyramid encoder-decoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. Other methods of encoding representations are also possible.

[0088] The representation of one or more teeth may be input to a second module of a representation learning model, such as an encoder structure, a multi-layer perceptron (MLP), a transformer (e.g., including at least one of a 3D encoder and a 3D decoder, which may be configured with a self-attention mechanism that enables the network to focus training on key inputs), an autoencoder (e.g., a variational autoencoder or a capsule autoencoder), which has been trained to output one or more representations (e.g., a transformation to place an oral care grid, such as the grid in the example of hardware and appliance component placement technology). In some embodiments, the transformation may include one or more 4x4 matrices, Euler angles, or quaternions. The second module may be trained at least in part by calculating one or more loss values ​​(such as L1 loss, L2 loss, MSE loss, reconstruction loss), or one or more of other loss calculation methods found elsewhere in the present disclosure. Such loss functions may quantify the difference between one or more generated representations and one or more reference representations (e.g., a benchmark true transformation known to have good function). In some implementations, either or both of module one and module two may receive one or more mesh element features associated with one or more oral care meshes (e.g., a mesh element feature vector may be calculated for one or more mesh elements of an input tooth, gum, hardware article, or appliance assembly). The advantages of receiving mesh element features generally relate to improving the underlying system. For example, such implementations allow the first module to more accurately represent the received 3D representation and allow the second module to generate a more accurate output 3D representation (e.g., a transformation, a representation of a tooth anatomy, or a label on a mesh element).

[0089] Figure 4 Depicted is a technique 400 for training a machine learning model according to certain aspects of the present disclosure. In general, the technique 400 uses the above Figure 2 That is, now describe Figure 4Certain additional aspects of . For example, depending on a particular implementation, it may be inappropriate or incorrect to apply the prediction output directly to the patient data to generate the prediction representation. For example, in a segmentation-based implementation, it is appropriate to apply one or more prediction labels to generate the prediction representation 220 because, for example, the underlying representation of the patient data is not modified. In contrast, in other implementations, the prediction output 407 may be one or more vectors describing one or more transformations, and it may be necessary to apply incremental processing steps to apply those transformations to the patient data. For example, when the prediction output 207 is the prediction output vector 407, the grid transformation module 418 may be used to apply the one or more prediction vectors to the patient data to generate the prediction representation 420. Similarly, when the reference input 207 is the reference input vector 407, the grid transformation module 426 may be used to apply the prediction vectors to the patient data to generate the prediction representation 421. The transformers 418 and 426 may use conventional techniques to apply the corresponding vectors to the patient data 204, translate, scale, and rotate the patient data 204 to generate the prediction representation 420 and the reference representation 421, respectively.

[0090] One specific example involves coordinate system generation. Digital dental and digital orthodontic applications may require the definition of coordinate systems in order to operate on 3D mesh models of teeth and gums. Some coordinate systems may be defined relative to the entire dental arch of teeth and are referred to as global coordinate systems. Some coordinate systems may be defined relative to individual teeth and are referred to as local coordinate systems.

[0091] In general, a tooth coordinate system comprises a set of XYZ axes that are used to facilitate mathematical transformations and other operations on a tooth mesh. A tooth coordinate system operates relative to the tooth with the origin located at a carefully selected central position relative to the tooth mesh. The local coordinate system of a tooth is contrasted with a global coordinate system, which has its origin located relative to the center of the entire dental arch. The global coordinate system is used to facilitate mathematical transformations and other operations on the entire dental arch. The correct choice of a tooth coordinate system is critical to the correct function of operations in the design of dental and orthodontic appliances relative to the tooth.

[0092] In a specific implementation related to coordinate system prediction, each patient case in the dataset 204 includes: 1) a set of segmented teeth in a dental arch; and 2) a set of transformations for describing a coordinate system relative to each of the teeth. In the depicted example, the generator 211 can be configured to generate one or more prediction vectors 407. In addition, the reference real input 208 is in Figure 4407 and 408 are represented as reference truth vector 408. As already mentioned, both vectors 407 and 408 represent transformations to be performed on patient case data 204 in order to generate one or more predicted representations 420 and reference truth representations 421, respectively. Vectors 407 and 408 can be of any size, but it has been observed that vectors with 4x4 dimensions are well suited for technique 400.

[0093] According to the depicted example, the technique 400 transforms the patient case data 204 using mesh transformation modules 418 and 426, generating prediction representations 420 and 421, respectively. In addition, and consistent with other aspects of the present disclosure, for each prediction transformation (e.g., as defined by the prediction vector 407), the system 100 calculates the LossG1 216 between the generated prediction vector 407 and the corresponding reference true vector 408. The LossG1 216 is fed back to update the weights of the generator 211. In addition, as already described, both the generated vector 407 and the reference true vector 408 are provided to the discriminator 235 (along with the relevant patient data 204, such as a dental mesh). The discriminator 235 attempts to label the vectors 407 and 408, thereby distinguishing true (reference true) from false (generated).

[0094] Depending on the specific implementation, the generator 211 may be replaced by an encoder, which may be viewed as Figure 4 The first half of the U-Net structure depicted in . Specifically, the encoder may include any number of mesh convolution operators 402 and any number of mesh pooling operators 404, but typically does not include an on-mesh pooling operator 406 or a mesh deconvolution operator. That is, the mesh convolution operator 402 generates a high-dimensional feature for each mesh element by collecting neighbor information of the element based on topology (i.e., based on mesh surface connectivity information). The mesh pooling operator 404 at each layer of the encoder simplifies the input mesh to a coarser resolution by reducing the count of mesh elements and aggregating neighboring features of each element. The high-dimensional features aggregated at the last layer are further processed by multiple fully connected layers and are ultimately transformed into a final regression output (e.g., a transformation matrix corresponding to a tooth coordinate system for tooth movement in 3D).

[0095] In some implementations, the techniques disclosed herein can simultaneously predict two orthogonal coordinate axes. From these two orthogonal coordinate axes, a third coordinate axis can be calculated, for example, using the Gram–Schmidt process.

[0096] Depending on the particular implementation, the coordinate system prediction operates on a six-dimensional representation. In addition, while it is possible to use the technique 400 to perform coordinate system prediction on a point cloud (e.g., a 3D point cloud), it is advantageous to perform coordinate system prediction on a 3D geometry such as a 3D mesh. This is because, in general, a 3D mesh (as opposed to a 3D point cloud) is more accurate in its ability to capture the local surface structure of an object. For example, two surfaces may be very close in Euclidean space, but very far apart from each other in a mesh topology (or in geodesic space). Therefore, a 3D mesh is a better choice for representing a surface.

[0097] In addition, for edges and vertices, vertex elements in a 3D mesh may have infinitely (theoretically) connected adjacent vertices, while edge elements in a 3D mesh have a fixed number of adjacent edges (e.g., 4 adjacent edges). The border edge can be given two virtual edges, making the number four. The use of a grid makes grid convolutions in 3D more straightforward. The fixed number of adjacent edges also makes the grid convolution output relatively more stable during training. From a grid topology perspective, the number of edges in a 3D grid is typically greater than the number of vertices (e.g., typically 3 times). In a sense, the grid resolution can be increased by using edges for prediction because there are many more edges than vertices in a typical grid. In addition, it should be understood that neural networks generally benefit from training on a large number of elements. Therefore, by using a 3D grid, the resulting reasoning is improved, and the beneficial effects are passed on to later post-processing steps, resulting in an overall more accurate system.

[0098] Similar to Figure 2 and Figure 3 Once trained, the generator 211 can be used to generate an accurate prediction vector 407 for the patient data 204 received by the receiving module 202. An example technique for generating the prediction vector 407 is Figure 5 4. The technique 500 shown in FIG. 4 shares many of the same features as the techniques 300 and / or 400 described above.

[0099] Figure 6 is an illustration of an example machine learning architecture 600 that can be used by the system 100 to design and manufacture dental appliances for restoring a patient's dental anatomy in accordance with various aspects of the present disclosure. For example, many of the techniques described herein rely on some form of the architecture 600 as the basis for the machine learning models described herein.

[0100] In the depicted example, the machine learning model 600 is a U-Net architecture. The eponymous architecture is configured as one or more grid convolution operators 602a-602n, grid pooling operators 604a-604n, grid on-pooling operators 406a-406n, and grid deconvolution operators arranged in an inverted pyramid or "U" configuration. As used herein, it should be understood that the term "operator" is synonymous with and interchangeably used with the terms "node" and "layer", which are also used to describe similar operations in machine learning parlance.

[0101] In general, the U-Net architecture 600 involves mesh pooling and on-mesh pooling operations, which aid in the process of extracting mesh element neighbor information. Each successive pooling layer helps the model learn neighbor geometry information by reducing the resolution relative to the previous layer. Each successive on-mesh pooling layer helps the model expand this summarized neighbor information back to a higher resolution. A series of mesh pooling layers followed by a series of on-mesh pooling layers will enable efficient and accurate training of the U-Net, and enable the U-Net to output features for each element containing both local and global geometry information.

[0102] According to a specific implementation, one purpose of the U-Net architecture 600 is to compute a high-dimensional feature vector for the input mesh. For example, according to a specific implementation, the U-Net architecture 600 computes a feature vector for each mesh element (e.g., a 128-element feature vector for each edge, vertex, or face element). The vector exists in a high-dimensional space that is capable of representing the local geometry of the edge within the context of the local teeth, and also represents the global geometry of the two dental arches. The encoder uses the high-dimensional features of the elements within each tooth to make predictions. This combination of local and global information contributes to the accuracy of the predictions. The combination of local and global information enables the U-Net architecture 600 to take into account geometric constraints. For example, during the treatment of clear tray-type aligners, it is undesirable for teeth to collide in 3D space. The combination of local and global information enables the U-Net architecture 600 to generate transformations that reduce or eliminate the incidence of conflicts, and thus produce higher accuracy relative to the prior art. In the event of a collision, the technology of WO2020 / 136587A1 "Methods to automatically remove collisions between digital mesh objects and smoothly move mesh objects between spatial arrangements" can be used to detect and remove collisions between tooth meshes of a dental arch. Consistent with the present disclosure, geometric quantities, such as penetration depth, penetration direction, and counts of overlapping mesh elements (such as vertices), can be calculated, consistent with the detection and removal of tooth mesh collisions. Consistent with the present disclosure, the mesh shape and / or position can be perturbed or changed to reduce or eliminate the occurrence of collisions, which in some cases may still exist after operation of the neural network structure of the present disclosure. Typically, information provided to the machine learning model 600 is first processed by propagating "downward" through operators 602a, 604a, 602b, 604n, etc. until the information reaches the bottom operator (represented here by the mesh convolution operator 602c). This information then propagates "upwards" through operators 606a, 602d, 606n, etc. until it is output by a final grid convolution operator 402n, which may be used by various aspects of the present disclosure, as will be described in more detail below.

[0103] Figure 6 The example U-Net architecture shown is depicted as having a total of nine layers (or nine operators), but it should be understood and appreciated that the U-Net architecture can be configured to have any number of convolutional layers, any number of grid pooling layers, and any number of on-grid pooling layers to achieve desired results.

[0104] In general, each of the operators 602a-602n, 604a-604n, and 606a-606n can be configured using conventional techniques to modify received inputs related to 3D mesh data (including, for example, mesh size and pose, as embodied by edge lengths, edge curvatures, edge normals, edge midpoints, and other edge data) to produce specific outputs suitable for each of the operators 602a-602n, 604a-604n, and 606a-606n, as will be described in more detail below.

[0105] According to a particular implementation, the mesh convolution operators 602a-602n disclosed in the present disclosure may be configured to be agnostic to the size and pose (e.g., position and / or orientation) of the input 3D mesh according to a particular implementation. The advantage of this agnostic approach is that, as opposed to an input mesh of a fixed size and / or orientation, the mesh cleaning operator can be used to process an original input mesh of arbitrary orientation.

[0106] However, in other implementations, size and pose information is required, such as in the context of regression operations. In implementations where size and pose information is desired, convolution operations may alternatively be configured to be agnostic to size and pose information. For example, convolution filters used as part of the convolution operators 602a-602n machine learning model may be specifically configured to be sensitive to size and pose information, where such systems should not be agnostic to the information. In other implementations, there may be specific aspects of operations that benefit from size and pose information. A specific example is for 3D mesh segmentation, which benefits from size and pose agnostic modes in some applications (e.g., segmentation of gingiva, which is used to find general areas of intraoral scans containing teeth), but does not benefit in other applications (e.g., tooth segmentation, which benefits from information about the left and right sides of the mesh). Therefore, it should be understood that within a particular type of task (e.g., a segmentation task), various aspects of the machine learning model may be configured to be size and pose agnostic for those operations that benefit, and other aspects of the machine learning model may be configured to be size and pose sensitive for those operations.

[0107] The mesh pooling operators 604a-604n are configured to resample the input mesh to a lower resolution. As a result, the mesh is continuously refined and resampled to a lower resolution through each successive layer of the mesh pooling operators 604a-604n. This allows downsampling or shrinking of the mesh input. For example, downsampling of information in 3D space may take a 3×3×3 set of information and combine it into a single 1×1×1 representation. For example, in the context of 3D mesh information, the four adjacent edges of a given edge will be combined into a single edge at the next resolution level. The mesh resolution (mesh surface area) after downsampling will be reduced by a factor of 4.

[0108] One of the many advantages of this approach is that the grid pooling operators 604a-604n cause each feature to collect information about its neighbors and summarize that information into a form that is passed to the next layer. Thus, as the grid information moves through the U-Net architecture 600, the output of the lowest level convolution operation 602 (such as 602c in the depicted example) takes the form of a downsampled grid that reveals global information about the original input grid. In other words, the output of the lowest level convolution operation 602 is considered to constitute fully summarized information and it can be used in accordance with various techniques of the present disclosure. For example, the downsampled output of the lowest level grid convolution operation 602 can be used in classification operations (e.g., for 3D verification) and regression operations (e.g., for coordinate system prediction), to name just a few examples.

[0109] In addition, the fully summarized information may undergo further processing by additional operators (e.g., depicted as operators 602n, 604n, and 606n). For example, the fully summarized information output by operator 402c may be processed by on-mesh pooling operators 606a and 606n to increase the resolution of the mesh information. As depicted by the example machine learning model 600, there is a 1:1 relationship between mesh pooling operators 604a and 604b and on-mesh pooling operators 606a and 606n. That is, after a sufficient number of on-mesh pooling operations (e.g., performed by operators 606a-606n) equal to the number of mesh pooling operations (e.g., performed by operators 604a-604b) have been performed, sufficient information that allows other automatic techniques to identify element categories (e.g., edges, faces, vertices) in the 3D mesh is surfaced. For example, this allows an automated system to perform mesh segmentation on the output of the convolution operator 602n (eg, perform tooth segmentation, gingival segmentation, facial-lingual segmentation, etc.).

[0110] Now go to Figure 7 In the depicted example, in step 702, a system (such as system 100) receives one or more 3D oral care representations, such as a 3D mesh of a patient's dentition (which may include information related to the patient's teeth, gingival tissue, and other aspects of the patient's oral anatomy), as well as other information. The received 3D mesh may vary according to a particular purpose. For example, in a specific implementation regarding mesh segmentation, the received 3D information may be related to a dental arch of a patient's mouth, which may include a 3D representation of teeth and / or gingival tissue, a specific implementation for verification of hardware or appliance component placement. The received 3D mesh may include a 3D representation of specific teeth and associated hardware. In a specific implementation regarding verification of a 3D printed part, the received 3D mesh may include 3D mesh data associated with a part examined in the form of a CT scan or other diagnostic imaging, to name just a few additional examples.

[0111] In step 703, the system 100 may receive a fully trained neural network, such as the fully trained generator 211 described above.

[0112] In step 704, the system 100 may optionally process the received 3D oral care representation to prepare for subsequent steps. For example, in one embodiment, the system 100 may generate or otherwise place components for dental restoration appliances on corresponding teeth in the 3D mesh that must be verified. In another embodiment, the system 100 may place brackets or accessories (or other hardware, such as buttons or hooks attached to teeth, to which resistance bands may be attached and attached to teeth) relative to specific teeth in the 3D oral care representation. In a related embodiment, the system 100 may predict a coordinate system of one or more teeth (e.g., including one or more local coordinate axes of each tooth). In yet other embodiments, the 3D oral care representation may be processed to facilitate identification or marking of mesh elements in a 3D mesh (or 3D point cloud) of a patient's dentition. Examples of where this may be useful include segmentation applications (e.g., tooth segmentation), mesh cleaning, or automatic restoration design generation. In another embodiment, and relative to segmentation, specific teeth may be marked as being correctly segmented or incorrectly segmented. Other types of verification regarding other aspects of the present disclosure are also possible. In other words, depending on the details of a particular implementation, there may be many ways to train a neural network that can validate a 3D oral care representation.

[0113] In step 706, the system 100 may use a 3D modeling tool to generate multiple 2D raster views for each tooth. According to a specific implementation, a 3D modeling tool such as GEOMAGIC may be used, for example, through an automated script. In some examples, other 3D modeling and rendering engines may be used. As used herein, a view may be defined as a specific orientation of a camera within a modeling tool, providing a specific representation of a 3D mesh having a 3-dimensional space represented in the modeling tool. In other words, at step 706, the camera within the modeling tool may be positioned so that each tooth in the 3D mesh is observed from a slightly different angle or vantage point within the modeling tool. The number of views generated may vary according to a specific implementation or a specific use case. For example, according to one specific implementation, fifteen different views of the 3D mesh are generated, although any number of views may be generated for a particular tooth. Therefore, if fifteen views are generated at step 706, a total of 480 2D images for the patient's mouth may be generated at step 706 for a patient having thirty-two teeth, just to give one example.

[0114] Depending on the particular implementation, the 2D raster image generated in step 706 can be used as a comparator when performing other techniques described herein. For example, with respect to tooth segmentation, the segmented tooth mesh (e.g., generated in step 704) can be overlaid on top of the 3D mesh data received in step 702. Then, aspects of the 2D raster image that align with the scan data can be identified. For example, in one implementation, the result of the overlay is a red portion of the geometry corresponding to the segmented tooth mesh and a blue portion of the geometry corresponding to the scan data.

[0115] One advantage of applying a visualization such as described above is that such visualization allows a human user to identify potential misclassifications of the training data. Additionally, applying an essentially binary treatment to the teeth allows a binary classification machine learning model (as described elsewhere in this specification) to be trained to provide accurate predictions. It should be understood that, without loss of generality, each of the 2D and 3D validation examples of the present disclosure may operate under n-class classifications, such as where there are multiple "correct" validation results and multiple "incorrect" validation results.

[0116] In step 708, the system 100 may accumulate or otherwise aggregate 2D views over multiple patient cases. For example, according to one implementation, 60 patient cases may be used. In other words, if 480 2D images are generated for each patient, then in an implementation using 60 patient cases, the training data may include 28,800 different 2D images, as just one example.

[0117] In step 710, the system 100 may train the neural network received in step 703 to validate the accumulated views of one or more cases. For example, when it comes to validating a digitally generated setup for orthodontic alignment treatment, running the fully trained neural network may assign one or more standard scores that specify whether one or more aspects of the received views of the generated setup are correctly formed.

[0118] In step 712, the system 100 outputs both the test results and the resulting neural network. For example, depending on the particular implementation, the output may specify whether the received 3D mesh passes the validation check. If the received 3D mesh does not pass the validation check, the output may also include a correction to the received information describing one or more corrective measures. For example, if the 3D mesh represents a scan of a 3D printed part, the corrective measures may describe how to modify the already manufactured 3D printed part to fit the patient's dental anatomy. Various conditions can be measured or analyzed in this manner. For example, the technology can measure whether the generated settings are correctly formed with respect to measurement criteria for alignment, marginal ridges, buccal and lingual inclinations, occlusal relationships, occlusal contacts, overjet (or overbite), interproximal contacts, and root angulation, to name just a few examples. In other examples, the corrective measures may provide guidance on how to correct the function of the 3D printer (e.g., address a partially blocked nozzle that causes a deformed 3D printed part).

[0119] Although neural networks are used to describe technology 700, one or more steps of technology 700 may also be performed using machine learning models other than neural networks, such as support vector machines (SVNs), random forests, K-nearest neighbors (KNNs), and other machine learning models. In order to understand how such other machine learning models can be used, the data can be divided into two categories of data: "TECH" (class 01) and "RAW" (class 00) data. The TECH class is data obtained by manual intervention of professional technicians. The RAW class is data output from automated tools. TECH class data can generally represent a more correct data set than RAW class data because the TECH class data has been fixed / improved / fine-tuned by expert technicians. The following method is related to non-neural network methods for distinguishing between TECH class (class 01) and RAW class (class 00).

[0120] For an effective texture feature-based validation classifier, combining segmentation markers via color with tooth / gum geometry can produce different kinds of artifacts for each class. There are multiple existing texture feature descriptors that can be used as part of texture feature-based validation, including HOG, SURF, SIFT, GLOH, FREAK, and Kadir-Brady. These texture-based validation classifiers can be used by less complex machine learning models, like some image enhancements can improve the classifiers, such as increasing the contrast between the tooth and gum segmentations so that the feature vectors find more differences around the tooth / gum line when comparing computer- and technician-generated segmentations. Each of the validation applications of the present disclosure may describe a specific implementation involving operations based on texture features.

[0121] For example, verification based on texture features using SIFT classification may include an optional step of converting training images to grayscale, and the steps of finding SIFT keypoints on each image, generating descriptors for these keypoints, selecting only the top N descriptors (where N is the minimum number of descriptors found in all training sample input images), and training a support vector machine (SVM) model on the image descriptors. As an example, other specific implementations may, for example, replace the training of an SVM model on the image descriptors with fitting a k-nearest neighbor (KNN) classifier on the image descriptors.

[0122] That is, while more simplified non-neural network machine learning models can be used, there are various advantages to using a neural network approach. For example, a neural network can be designed to have a sufficiently large number of parameters (i.e., weights) to encode solutions to complex problems, such as understanding 2D raster image views and 3D geometric shapes (i.e., 3D meshes). In addition, texture features may not detect all relevant attributes of an image, such as attributes that indicate defects or errors that the verification process wants to detect.

[0123] Figure 8 An example general technique 800 for validating outputs generated by a machine learning model according to various aspects of the present disclosure is shown. The validation ML model may be trained to process the following non-limiting list of 3D representations: 1) mesh element labels for segmentation or mesh cleaning; 2) coordinate system axes of teeth (e.g., as encoded by transforms); 3) dental restoration design; orthodontic settings; 4) custom lingual brackets; 5) bonding pads for brackets (which can be generated for specific teeth by outlining the perimeter on the teeth, specifying the thickness to form a shell, and then subtracting the teeth through Boolean operations); 6) clear tray appliances (CTA); 7) location or shape of trim lines (e.g., CTA trim lines); 8) shape or structure or posture of attachments; 9) bite bevels or slits; 10) 3D printed appliances (local thickness, rib geometry, flap positioning, etc.); 11) 3D models of the patient's teeth and gums showing trim lines (e.g., fixture models), data or structures related to implant placement; 12) hardware placement; 13) other types of dental restoration designs (e.g., veneers, crowns, or bridges); 14) and other 3D printed parts that are part of the oral care routine or other areas.

[0124] Technique 1800 may use the steps of receiving a 3D mesh of one or more teeth, and additional optional data related to a dental procedure. This information may be provided to one or more anomaly detection networks for verification. In some implementations, this may include generating one or more 2D raster views of the 3D mesh. Next, system 100 may use a neural network to analyze each aspect of the 2D and / or 3D representation to present a pass / fail determination regarding those aspects. If a sufficient number of aspects receive a passing accuracy score, the representation is considered to have passed, at which point system 100 may provide geometry for use in other dental procedures. If a sufficient number of aspects do not receive a passing accuracy score, system 100 may generate information about why one or more aspects of the representation failed, and in some implementations, automatically train one or more neural networks based on the results, and then perform method 800 again with additional training of the neural network to see if a passing score can be achieved. In various implementations, this 2D verification method may be applied to each of the various verification applications described in the present disclosure.

[0125] The technique 800 can be performed in near real time, allowing dental professionals and other professionals to perform scans and other dental procedures while the patient is in the chair, thereby improving the effectiveness of dental treatment and providing a more pleasant experience for the patient. For example, the verification method can be applied to the patient's intraoral scan data immediately after the intraoral scan is performed. The advantage is that if there is a problem with the scan data, the dentist can be notified, and if a rescan is necessary, the patient can do so (and in fact, without even leaving the chair). Detected mesh errors include holes in the mesh, incompletely scanned teeth, missing teeth, foreign matter obscuring teeth, and / or upper / lower dental arches being incorrectly identified / interchanged. The results of the verification can be displayed to the dentist (or technician) using one or more heat maps, possibly superimposed on the dental model. Problematic areas of the mesh can be highlighted in a patchwork manner using different color coding. The disclosure regarding mesh cleaning describes mesh defects detected during the mesh cleaning verification process. The application of this near real-time method can also benefit from performing inspections to detect these conditions, so that the intraoral scan can be re-performed under different conditions (e.g., more careful techniques employed by the technician or doctor). In this case, the need for subsequent mesh cleanup operations can be reduced or eliminated.

[0126] Specific errors or flaws in the scan are highlighted using color, bounding boxes, arrows, or other graphical elements and displayed to the dentist / technician. For example, if the verification engine determines that a portion of a tooth is missing from the mesh, a bounding box may be drawn onto a visualization of the mesh over the area of ​​the missing or incomplete tooth. A text report on the quality of the scan may be prepared and sent via SMS, email, or other electronic means, or displayed to the dentist / technician in the dentist's office. In some cases, there may be an LCD display located near the scanner that displays the verification report to the dentist. As another example, the verification engine may apply a parting surface to the tooth, resulting in each edge / vertex / face element in the tooth mesh being labeled as A) facial or B) lingual: 1) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the facial direction (e.g., 1.0 mm or 0.5 mm); 2) a facial portion of a tooth where the parting surface is correct; 3) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the lingual direction (e.g., 1.0 mm or 0.5 mm). Depending on the particular implementation, there may be more than one label. For example, some implementations may use both element labels and result labels. The element label describes whether the edge / vertex / face element is on the facial side of the tooth mesh or on the lingual side of the tooth mesh. The result label indicates whether the parting surface near the tooth is 1) too facial, 2) correct, or 3) too lingual, to name just one example.

[0127] According to the technology of the present disclosure, an ML model can be trained on examples of 3D oral care representations, wherein benchmark real data is provided to the ML model and a loss function is used to quantify the difference between the predicted examples and the benchmark real examples. The loss value can then be used to update the validation ML model (e.g., update the weights of a neural network). Such validation techniques can determine whether a trial 3D oral care representation is acceptable or suitable for creating oral care appliances. In some cases, "acceptable" may mean that the trial 3D oral care representation conforms to the distribution of benchmark real examples used in training the ML validation model. In some cases, "acceptable" may mean that the trial 3D oral care representation is correctly shaped or correctly positioned relative to one or more aspects of the dental anatomy.

[0128] In the example of a generated appliance component (e.g., a dental restoration appliance component, such as a mold parting surface), the technology can determine whether the component intersects the correct landmarks or other parts of the tooth anatomy (e.g., incisal edges and cusps for mold parting surfaces). The technology can also determine one or more of the following: 1) whether the CTA trim line intersects the gingiva in a manner that reflects the distribution of the benchmark truth; 2) whether the library component is correctly placed relative to one or more target teeth (e.g., a snap clip placed relative to posterior teeth or a center clip placed relative to incisors), or correctly placed relative to one or more landmarks on the target teeth; 3) whether the hardware elements are placed on the face of the tooth with edges that reflect the distribution of the benchmark truth examples; 4) whether the mesh element labels used for segmentation (or mesh cleaning) operations conform to the label distribution in the benchmark truth examples; and 5) whether the shape and / or structure of the dental restoration tooth design conforms to the distribution of tooth designs in the benchmark truth training examples, to name a few examples. Other verification conditions and / or rules are possible for verification of various 3D oral care representations.

[0129] Figures 9 to 11 Example techniques for cleaning a 3D mesh (e.g., the structure and / or geometry of a mesh) according to various aspects of the present disclosure are shown. In general, in some implementations, the mesh cleaning techniques described herein can add one or more mesh elements to a test 3D mesh, remove one or more mesh elements from a test 3D mesh, transform (e.g., translate, rotate, smooth, etc.) one or more mesh elements in a test 3D mesh, and combinations thereof. Fig. 9 A first mesh cleaning technique is shown. A 3D mesh is a data structure that describes the surface shape of a 3D object and contains elements such as vertices, edges, and faces (sometimes voxels). A machine learning model is proposed to identify certain aspects of the mesh geometry so that these aspects of the geometry are used in conjunction with mesh processing operations to mark other mesh elements for further processing. Mesh processing techniques known to those skilled in the art can then be applied to those marked mesh elements to remove, modify, and / or expand (e.g., hole filling, bridging, boundary expansion, etc.) areas of the mesh. In some specific implementations, for example, during a removal operation, one or more mesh elements may be deleted from the mesh. In some specific implementations, for example, during an expansion operation, one or more mesh elements may be added to the mesh. In some cases, one or more mesh elements may undergo a transformation, for example, to smooth various aspects of the mesh. In this way, the mesh can be "cleaned".

[0130] The neural network can be trained to apply labels to mesh elements (e.g., for purposes of either or both of mesh segmentation and mesh cleaning of a 3D mesh) at least in part by computing one or more loss functions (e.g., a cross entropy loss or another loss disclosed herein), where the loss function quantifies the difference between at least one predicted mesh element label and a corresponding ground truth mesh element label.

[0131] Techniques for mesh segmentation and mesh cleaning of oral care meshes may involve marking of mesh elements such as edges. Some specific implementations may mark voxels, vertices, faces, or other aspects of a 3D representation. Marking mesh elements for mesh cleaning may be assisted by calculating a mesh element feature vector (as defined elsewhere in this disclosure) for each mesh element. For example, in the case of marking edges during mesh cleaning, mesh element features may include: edge position (e.g., defined by vertices defining edge endpoints), edge normal vector, edge curvature, and edge length may be calculated and provided to a mesh cleaning neural network (or mesh cleaning neural network) at training time (and / or at deployment time). An advantage of training a neural network to adjust mesh element features is that the neural network may obtain additional information about the shape and / or structure of the received oral care mesh (e.g., obtaining information about the cleaned dental arch before the received mesh, which may include teeth, gums, hardware, and / or other oral care objects).

[0132] In some embodiments, a neural network such as a U-Net, a residual network (ResNet), a 3D pyramid encoder-decoder, an autoencoder, a transformer, or a 3D transformer encoder-decoder structure can be trained to label mesh elements in an oral care mesh for mesh cleaning. The U-Net architecture can be trained to extract at least one of local and global information from an oral care mesh being cleaned (e.g., a dental arch), which has the advantage of improving the understanding of mesh geometry. A better understanding of mesh geometry contributes to the predictive accuracy of the mesh cleaning techniques described herein. The U-Net can have a U-shaped layer that can use a mesh convolution layer and a mesh pooling layer to deconstruct the oral care mesh into a series of increasingly coarse resolutions, and then reconstruct the mesh through a series of increasingly fine resolutions. The U-Net architecture can have jump connections between the neural network layers at each level of the U-shaped structure, which has the advantage of preserving information about the oral care mesh at different resolutions and enabling the information to influence and improve the accuracy of the output of the U-Net. ResNet may have one or more skip connections from the head of the ResNet to the tail of the ResNet, which in some cases may have the advantage of reducing information loss that can sometimes occur in a neural network (e.g., when information passes through multiple consecutive layers). Skip connections can also help the network alleviate numerical problems, such as vanishing gradients that may occur when training via backpropagation. These advantages of skip connections also apply to skip connections in other architectures described herein (such as U-Net). In some specific implementations, the transformer decoder can be trained to label 3D mesh elements (e.g., for segmentation or mesh cleaning), for example, by transfer learning, where a transformer (such as Mask3D) that has been trained for semantic instance segmentation can be further trained with oral care data (e.g., dental arches from an intraoral scanner). The transformer in this specific implementation has the advantage of being trained to iteratively focus on certain important inputs related to oral care data, and can focus on associated neural network features at multiple scales of mesh (or point cloud) resolution. Processing at multiple scales enables local and global features to be considered during processing, which increases the ability of the network to encode the data set and generate correct outputs. Transformers have further advantages in processing oral care data in large batches or sequentially, such as processing most or all teeth of a dental arch at once in some cases. The advantage of such concurrent processing is that the output can take into account global aspects of the dental arch. For example, in some cases, when a transformer is trained to label mesh elements in a 3D mesh (or point cloud) during mesh segmentation or mesh cleaning, the transformer can process multiple teeth simultaneously. Other mesh cleaning techniques for oral care meshes include autoencoder methods (such as using variational autoencoders or capsule autoencoders) and diffusion methods for labeling mesh elements (e.g., for segmentation or mesh cleaning).Some specific implementations of autoencoder-based mesh cleaning techniques can train a reconstruction autoencoder to reconstruct a specific type of 3D oral care mesh (or point cloud). The encoder portion of such an autoencoder can be used in a deployment to convert an instance 3D oral care mesh (e.g., a dental arch) into a latent form (e.g., a latent vector), which may include a reformatting or dimensionality reduction of the input 3D oral care mesh. The latent form can then be reconstructed using a decoder of a trained reconstruction autoencoder. When the reconstruction error is below a threshold, the reconstruction is displayed as successful. When the reconstruction is unsuccessful (e.g., when the reconstruction error is above a threshold), the reconstruction error calculation can mark one or more portions of the input mesh (e.g., a collection of mesh elements) as abnormal. Mesh processing techniques can be used to remove or modify (e.g., smooth) such abnormal mesh elements. In this way, an autoencoder can be trained for mesh cleaning.

[0133] 3D convolution can help with grid element labeling for grid cleaning, such as when downsampling a 3D grid. 3D deconvolution is the inverse operation of 3D convolution, such as in U-Net. 3D pooling can help with grid element labeling for grid cleaning, such as when summarizing neural network feature maps. 3D up-pooling is the inverse operation of 3D pooling, such as in U-Net. 3D convolution, 3D deconvolution, 3D pooling, or 3D up-pooling operators can be implemented in the context of operating on one or more of the grid elements disclosed herein, which grid elements include: edges, faces, points, vertices, or voxels. In grid edge-based convolution, there may not be a well-defined ordering of edges, and there may be a variable number of adjacent edges, both of which require additional algorithmic complexity to accommodate. The advantage of this method is that edge-based (or face-based) convolution is invariant to rotation, scaling, and translation changes. Edge-based convolution is based on geodesic space, so the connectivity between elements is taken into account. In voxel-based convolutions and pooling (such as may be used in sparse processing), there may be a well-defined ordering of voxels (i.e., a given voxel may have a well-defined set of neighboring voxels in a grid, with a defined order). There may be a fixed number of neighboring voxels. Improvements to voxel-based operations include lower memory (RAM) usage, and the ability to load larger dataset models for concurrent processing.

[0134] In this example, the machine learning model is a neural network, such as a generative adversarial network (GAN). The neural network is designed to perform segmentation. The segmentation operation produces a set of points that define the gingival boundary. In the case of a mesh of a dental arch, elements need to be removed to support subsequent appliance design operations. The problematic elements may come from intraoral scanning artifacts, or they may be legitimate scan data that is irrelevant or problematic for further processing, such as extraneous materials, depressions, undercuts, internal fractures (abfractions), or non-organic geometry, such as orthodontic brackets (or other hardware, such as attachments, hooks, buttons, separators, retainers—such as lingual retainers, mini screws, or pins). In a specific implementation C2 as described, a neural network is used to perform a gingival segmentation (i.e., tooth-gingival segmentation) operation, which in turn is used to define the contour of the gingiva. A trim line is defined as a variable offset away from a key point on the gingival boundary. Mesh processing is used to trim (or shear) the dental arch mesh, where the dental arch mesh intersects the trim line and bridges (or fills) any gaps between the trim line and the mesh boundary (see Figure 5 That is, the bridging technique can be used to smooth the rough edges of a 3D mesh by filling in the missing mesh elements.

[0135] Fig.10 A second mesh cleaning technique 1000 is shown and also performs removal of unwanted mesh elements. The depicted technique focuses on identifying mesh elements to be removed, but Fig.10 The techniques shown may also be extended to include modification operations. For example, after the AI ​​identifies mesh elements representing the tongue bar, these mesh elements may be offset and smoothed rather than removed and hole filled. In a specific implementation, hole filling is the process of filling missing mesh elements (rather than along edges). Filling mesh elements along edges is defined as bridging, as described above. As another example, certain points identified by the AI ​​on the teeth may be used as input to metric generation operations, marking operations, or other calculations that do not modify the mesh at all. That is, the systems and techniques described herein are configured to identify aspects of interest in a 3D representation, which may be used by other aspects of the system or subsequent techniques to achieve the various advantages disclosed herein.

[0136] In some cases, the transformation prediction techniques described herein may be trained based on benchmark true transformations from past patient datasets to generate transformations that place a 3D oral care representation (e.g., a dental arch produced by an intraoral scanner before or after mesh segmentation) into a pose relative to one or more global coordinate axes. Such a pose may represent a canonical pose suitable for subsequent processing or visualization (e.g., predicting a transformation that can orient the dental arch using one or more global coordinate axes). Such a transformation prediction operation may first convert the 3D oral care representation to a first representation (e.g., using a U-Net or autoencoder) and then generate a transformation based on the first representation (e.g., using an MLP, encoder, or transformer). Having a dental arch oriented with one or more global coordinate axes may be beneficial for subsequent processing steps, such as Fig.10 Some examples of mesh cleaning techniques are shown in Figures 18 to 22 For example, orienting the dental arch enables the location of landmarks during later processing operations such as the first mesh cleaning technique 900. Other mesh cleaning techniques such as Fig.11 The canonical pose transformation step is useful in data visualization, for example, to render a dental arch mesh in a canonical orientation in a clinical processing software application (e.g., so that the relevant portions of the dental anatomy can be clearly viewed and analyzed through the process of processing and appliance creation).

[0137] Fig.11 A generalized mesh cleaning technique 1100 is shown. As described above, a 3D mesh is a data structure that describes the surface shape of a 3D object and contains elements such as vertices, edges, and faces. A machine learning model is proposed to mark problematic parts of a 3D mesh so that those marked elements can be removed and the resulting holes or boundary gaps can be optionally filled to approximate the correct surface. In this way, the mesh can be "cleaned". In this example, the machine learning model is a neural network, such as a generative adversarial network (GAN). Labels are applied to mesh elements of interest. In the case of a mesh of a dental arch, elements need to be removed to support subsequent appliance design operations. The problematic elements may come from intraoral scanning artifacts, or they may be legitimate scan data that is irrelevant or problematic for further processing, such as irrelevant materials, depressions, undercuts, internal fragmentation, or non-organic geometries, such as orthodontic brackets. In some cases, holes may still exist after problematic elements have been marked and removed. Classical mesh processing techniques can be used for hole filling. In some cases, uneven or jagged mesh edges may be left after problematic mesh elements have been removed. Such edges can be smoothed or extended through a process known as bridging.

[0138] In one example, GANs are used to mark edges in a mesh that correspond to depressions (see the maroon material in the figure below). These marked edge elements can then be removed from the mesh using 3D mesh processing techniques known to those skilled in the art. When the marked edge elements are removed, the mesh is left with one or more holes, which can then be filled using 3D mesh processing techniques known to those skilled in the art (e.g., "Filling Holes in Meshes", P. Liepa, Symposium on GeometryProcessing, 2003). In another example, GANs are used to mark edges in a mesh that correspond to extraneous materials (see the light green material in the figure below). These marked edge elements can then be removed from the mesh using 3D mesh processing techniques known to those skilled in the art. When the marked edge elements are removed, the mesh may be left with an uneven outer boundary. This outer boundary can be smoothed or extended by bridging.

[0139] Some specific implementations of the mesh cleaning techniques described herein can be trained to remove (or modify) general triangle mesh defects, such as: degenerate triangles with zero surface area; redundant triangles, which cover the same surface area as another triangle; non-manifold edges, also known as "flaps", with more than two adjacent triangles; non-manifold vertices with more than one adjacent connected triangle sequence (triangle fan); intersecting triangles (where two triangles pass through each other); spikes - sharp features composed of multiple triangles, usually cone-shaped, caused by one or more vertices deviating from the actual surface; folds (sharp features composed of multiple triangles, usually Z-shaped with small undercut areas, caused by one or more vertices deviating from the actual surface); islands / small components, which represent disconnected objects that should only contain a single object in the scan (e.g., typically smaller objects are deleted); small holes in the mesh surface, either from the original scan or from deletion due to previous defects (e.g., holes can be removed by filling the holes, such as by adding one or more mesh elements); rough boundaries - smooth boundaries are conducive to extending the gingival surface and creating a model basis.

[0140] Some specific implementations of the mesh cleaning techniques described herein can be trained to remove (or modify) general triangular mesh defects, such as: extraneous material (portions of the intraoral scan outside the anatomical region of interest, for example, non-tooth surfaces that are not within a certain distance of the tooth surface, or scanning artifacts that do not represent the actual anatomical structure); concavities - depressions in the surface (for example, they can be scanning artifacts that should be repaired, or anatomical features that are usually left intact); undercuts (tooth sides that are smaller than the crown radius, so physical impressions or appliances may be difficult to remove or place). Undercuts can be natural features or caused by damage such as internal chipping. Other features that may require the cleaning operation of the present disclosure include internal chipping (tooth erosion near the gum line, causing or exacerbating undercuts); appliances - orthodontic hardware, such as attachments, brackets, wires, buttons, lingual bars, Carriere appliances, etc., which may be present in the intraoral scan. In some cases, digital removal and replacement with synthetic tooth / gingival surfaces may be required before any subsequent appliance creation steps are performed.

[0141] In some implementations, a 3D representation, such as a 3D mesh used to represent a geometric shape used in oral care treatment (e.g., an appliance, appliance assembly, or other shape described herein), can be validated by directly classifying the 3D representation. For example, the mesh classification function of the MeshCNN toolkit can be trained to classify oral care meshes. In some examples, such oral care meshes can include appliance assemblies and be classified as suitable or unsuitable for use in appliance creation (e.g., dental restorative appliance creation). Such mesh classification can function at least in part by classifying one or more mesh elements in one or more meshes to be classified. The classification of the entire mesh can be determined at least in part by the classification of at least one mesh element among the mesh elements contained within the mesh.

[0142] Fig.12An example technique 1200 for training a machine learning model (e.g., classifying a 3D mesh for the purpose of 3D mesh or point cloud verification) is shown. The verification systems and techniques of the present disclosure may assign one or more labels to one or more aspects of a representation to be verified (e.g., correctly formed or labeled, incorrectly formed or labeled, etc.). The verification systems and techniques of the present disclosure may benefit from the computation of mesh element features. 3D oral care mesh verification may be applied to segmentation, mesh cleaning, coordinate system prediction, dental restoration design, CTA setup verification, CTA trim line verification, fixture model verification, archform verification, orthodontic hardware placement verification, appliance component placement verification, 3D printed part verification, chairside scan verification, and other verification techniques described herein. In the event that a 3D verification check produces a failed output, one or more instructions or feedback data may be transmitted to an algorithm, process, or model that creates a 3D oral care representation, so that further iterations of 3D oral care representation generation may improve the design and hopefully mitigate the condition that caused the verification check to fail. In some implementations, a neural network trained to classify a 3D mesh (or point cloud) for verification may take mesh element features as input (e.g., a mesh element feature vector may be calculated for one or more mesh elements in the mesh or point cloud to be verified). In some cases, a mesh element feature vector may accompany each mesh element as input to the verification neural network. In some cases, the verification neural network may form a reformatted (or sometimes reduced dimension) representation of the input mesh or point cloud. The mesh element features may improve this reformatted (or reduced dimension) representation by providing additional information about the shape and / or structure of the input mesh or point cloud. By using the mesh element features, the precision and accuracy of the resulting verified data is improved.

[0143] Fig.13 Training data that can be used to train a mesh cleaning validation machine learning model (e.g., a neural network, such as a CNN) is shown. Such a neural network can be trained to distinguish between tooth meshes that exhibit defects that require mesh cleaning and tooth meshes that do not have significant defects. One of the mesh cleaning operations is called recess filling. This operation is an example of a "positive fill." This operation is performed in the manufacture of CTAs (clear tray aligners) and can be applied in the manufacture of other dental or orthodontic appliance components. Recesses can be the result of scanning errors. Recesses can also occur due to the natural anatomical geometry of the patient's teeth. The advantage of removing recesses in the manufacture of CTAs is that the appliance tray will not engage in the recess in an undesirable manner. In some cases, naturally occurring recesses can be skipped, but scanning artifacts cannot be skipped.

[0144] The images in the bottom row show examples of 2D raster views from a category where recesses are present. The images in the top row show teeth without recesses. In accordance with the present disclosure, these two images represent 2D raster views of two related categories that can be used in quantity to train a machine learning classifier. In one embodiment, a neural network can be used for the ML classifier. In keeping with the present disclosure, the resulting classifier is used to determine whether recesses are present (e.g., to verify the output of a recess removal operation). In other embodiments, the present invention can be used to determine whether a recess removal operation is necessary to clean up raw scan data. In some embodiments, feedback from the verification engine can be fed into the automatic recess removal process to improve the output of subsequent iterations of the process.

[0145] about Fig.14 , shows the results of a hole filling operation. Another mesh cleaning operation is called hole filling. Holes in the mesh are areas of missing data caused by incomplete scans. Data can be added to the mesh via interpolation to fill the holes and create a complete surface. Holes must be filled carefully to avoid creating surfaces that are more concave than the original object. Such surfaces can create interference and cause poor fit of the appliance, retainer, attachment template, bonding tray, 3D printed mold, or other oral care appliance.

[0146] The images in the top row are examples of 2D raster views from a category where holes are present. The images in the bottom row show teeth without holes. In accordance with the present disclosure, these two images represent 2D raster views of two related categories that can be used in quantity to train a machine learning classifier. In one implementation, a neural network can be used for the ML classifier. In keeping with the present disclosure, the resulting classifier is used to determine whether holes are present (e.g., to verify the output of a hole removal operation). In other implementations, the present invention can be used to determine whether a hole removal operation is necessary to clean up raw scan data. In some implementations, feedback from the verification engine can be fed into the automatic hole removal process to improve the output of subsequent iterations of the process.

[0147] about Fig.15 , showing the results of removing extraneous data. Extraneous data are surfaces included in the scan that are not part of the tooth object or gingival tissue immediately adjacent to the tooth object. Examples are misplaced soft tissue, operator gloves, or scan merging errors. The purpose of the operation is to remove data that is not part of the patient's anatomy and may cause processing difficulties and / or result in a poor fit of the appliance, retainer, attachment template, or bonding tray. The advantage of performing this operation is that it will prevent downstream processing difficulties. This operation may also be applied in the manufacture of other dental and orthodontic appliances.

[0148] The images in the top row of the figure below are examples of 2D raster views from a category where extraneous material is present. The images in the bottom row show teeth without extraneous material. In accordance with the present disclosure, these two sets of images represent 2D raster views of two related categories that can be used in quantity to train a machine learning classifier. In one embodiment, a neural network can be used for the ML classifier. In keeping with the present disclosure, the resulting classifier is used to determine whether extraneous material is present (e.g., to verify the output of an extraneous material removal operation). In other embodiments, the present invention can be used to determine whether an extraneous material removal operation is necessary to clean up the raw scan data. In some embodiments, feedback from the verification engine can be fed into the automatic extraneous material removal process to improve the output of subsequent iterations of the process.

[0149] Fig.16 Another example of irrelevant data removal is shown. For example, Fig.16 A 3D mesh of teeth is shown in two situations: a clear tray appliance (CTA) trim line has not been applied (i.e., excess gingival material is present), and a CTA trim line has been applied (i.e., excess gingival material has been removed). These images can also be used to train a validation engine in accordance with the present disclosure. The validation engine can be used to determine whether a trim line operation is required or whether the operation was performed correctly.

[0150] While these figures show several examples for cleanup operations, other examples are possible according to Table 4:

[0151] Figures 17 to 22 Results related to gingival trimming and bridge mesh cleanup are shown. This new example of mesh cleanup is implemented as a hybrid of geometric deep learning (GDL) and classical mesh processing techniques. The GDL operation involves a segmentation operation. The deep learning model predicts the segmented tooth regions (represented by binary segmentation labels) directly from the input scans without any explicitly programmed 3D mesh processing steps. The gum line geometry is designed to define the tooth regions and is generated by identifying landmark points at each tooth boundary along the dental arch morphology in 3D ( Fig.17 The trimming lines are created using classic mesh processing techniques by finding the vertices (lowest points) along the gum line around each tooth region ( Fig.18 ), offset from those points along (and sometimes beyond) the mesh surface in a direction away from the tooth ( Fig.19 ), and define a spline that passes through those offset points ( Fig. 20 ). Trim away the excess scan grid area beyond the trim line and keep only the area of ​​interest for oral care applications ( Fig.21). Use bridging to fill in any missing areas of the desired mesh area. Bridging involves creating new mesh surfaces within the desired trim line boundaries where the original mesh did not extend to the desired offset. ( Fig. 22 ).

[0152] exist Fig.17 In GDL Gingiva Segmentation, each mesh edge is labeled as belonging to a tooth or gingiva. Boundaries between these labeled regions are identified. A boundary surrounds one or more teeth. If there is enough gap between adjacent teeth, there can be multiple boundaries. Each boundary can consist of a sequence of mesh vertices (blue spheres) connected by mesh edges (cyan lines) or other representations such as splines.

[0153] exist Fig.18 In the , the apex (lowest point) along the gingival border of each tooth can be identified. The point behind the posteriormost tooth is also identified.

[0154] exist Fig.19 , trim line control points are identified by offsetting them a desired distance from a vertex along the mesh surface. If a mesh boundary is reached before the desired offset distance is reached, the offset continues beyond the mesh boundary into empty space using the previous offset direction. The offset process may terminate due to local mesh conditions before the desired distance is reached. The desired offset distance may vary depending on tooth location or other considerations, for example, offsets are typically smaller behind posterior teeth.

[0155] exist Fig. 20 , you can generate a trimline spline that passes through the trimline control points. The trimline spline is sampled to create a trimline polyline (hereafter referred to as a "trimline"), which consists of an ordered sequence of vertices that are different from the mesh vertices. When the trimline vertices are above or below the mesh surface, they are projected onto the mesh surface but are still different from the mesh representation. The points shown correspond to the portions of the trimline on the mesh surface that need to be clipped. The red points correspond to the portions that need to be bridged.

[0156] exist Fig.21 , the mesh is clipped along the trim line. Notice that the mesh no longer extends beyond the trim line.

[0157] exist Fig. 22 In the example above, the mesh is bridged where its borders do not extend to the trim lines. The green surface areas show where bridging occurs (i.e., where the outer rough borders of the mesh are filled to extend to the trim lines).

[0158] Fig.23 and Fig.24Related to the hardware removal example of mesh cleaning. Figures 43 and 44 show a dental arch mesh including bracket hardware on the teeth. According to the mesh cleaning technique of the present disclosure, elements of the mesh have been marked so that individual brackets can be identified and removed from the dental arch mesh.

[0159] Scans of a patient's teeth may include hardware that obscures the patient's anatomy, such as orthodontic brackets, lingual bars, etc. In some applications, such hardware must be removed and an approximation of the actual anatomy reconstructed. For example, for a patient who has completed orthodontic bracket / wire treatment and is ready to transition to a clear tray appliance (CTA), a CTA case scan may be taken before the brackets are removed. The CTA must be designed without the brackets, so this hardware must be virtually removed from the scan. This is a time-consuming operation, and it would be advantageous to automatically detect the hardware in the scan and identify the mesh elements that represent the hardware.

[0160] Hardware removal is an important application of removal operations. In some cases, an upcoming scan of the dental arch will show brackets or other hardware attached to the teeth. This geometry must be removed to support subsequent processing. The present invention can be used to mark mesh elements corresponding to hardware so that those elements can be removed and the resulting holes can be filled.

[0161] During the treatment planning process, there may be existing hardware in the patient’s mouth from previous or current dental or orthodontic treatment. The 3D capture process will include this hardware as part of the output scan, but for proper treatment planning we must consider only the patient’s natural dentition. Automatic detection of this hardware aids the process by highlighting portions of the 3D scan that can then be verified as hardware and removed manually, while automated hardware removal takes this a step further and removes irrelevant portions of the scan without human involvement. Automated hardware removal is extremely valuable from a time and money saving perspective (i.e., as an alternative to paying a human technician to complete the work). The results of automated methods can also be more accurate and precise.

[0162] Fig.25 A dental arch is shown with labeled elements (i.e., ground truth labels) that can be used to train a neural network to label mesh elements according to performing any of the techniques 1900-2100 described above. Techniques 1900-2100 can use one or more neural networks (such as MeshCNN or U-Net) to label mesh elements. Such neural networks are trained to label mesh elements corresponding to anomalies that require further processing (such as removal). Examples of these anomalies are shown in FIG. 45. FIG. 45 illustrates a mesh cleaning technique that can be used to clean mesh elements using two techniques (e.g., the first and second mesh cleaning techniques, as described above with respect to each other). Fig.19 and Fig. 20The mesh cleaning techniques described herein may, in some implementations, add one or more mesh elements to the trial 3D mesh, may, in some implementations, remove one or more mesh elements from the trial 3D mesh, and may, in some implementations, transform (e.g., translate, rotate, smooth, etc.) one or more mesh elements in the trial 3D mesh.

[0163] Various aspects of the present disclosure may be used for different purposes in one or more digital dental fields, including segmentation, coordinate systems, mesh cleaning, setting of clear tray-type aligners, dental prosthesis, brackets and accessories, 3D printed parts, prosthetic designs, and fixture models. These fields may involve the generation of one or more (2D or 3D) representations and the verification of one or more (2D or 3D) representations. One or more of these fields may be combined, for example, certain techniques may combine the following concepts: 1) segmentation, 2) calculation of the geometry of dental prosthesis, and 3) mesh verification. For example, the results of face-tongue segmentation may be used by an algorithm for generating a mold parting surface, with the goal of improving the resulting mold parting surface (i.e., relative to the mold parting surface that would be generated without the benefit of the previous face-tongue segmentation). The resulting mold parting surface may then be checked by a verification module (i.e., using 2D or 3D processing). If the verification module determines that the generated mold parting surface is poor, the algorithm that generated the mold parting surface can be rerun, potentially using actionable feedback from the verification engine (e.g., tips on how to adjust the mold parting surface on a tooth-by-tooth basis, whether the parting surface should move in the facial or lingual direction near each tooth). If the verification module determines that the generated mold parting surface is acceptable, the mold parting surface is output.

[0164] Although this specification sets forth many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular implementation. Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations, even when initially claimed, in some cases one or more features from a claimed combination may be deleted from that combination, and a claimed combination may refer to a subcombination or a variation of a subcombination.

[0165] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described components and systems may generally be integrated together in a single system or distributed in multiple systems.

Claims

1. A computer-implemented method for training one or more neural networks to automatically recognize one or more aspects of a 3-dimensional (3D) oral care representation used in digital dentistry for further processing, the method comprising: receiving, by one or more computer processors, a first digital 3D oral care representation of a patient's dentition, wherein the first digital 3D oral care representation is not generated by mapping one or more 2-dimensional (2D) images onto the 3D representation; generating, by the one or more computer processors, a list of mesh elements associated with the first digital 3D oral care representation; using, by the one or more computer processors, the initially trained neural network to identify one or more aspects of the one or more digital 3D oral care representations for which additional processing is to be performed; automatically training, by the one or more computer processors, the neural network based on the use, wherein the training of the neural network is modified by performing operations comprising: identifying, by the neural network, one or more aspects of the first digital 3D oral care representation for which additional processing is to be performed based on the list of mesh elements; generating a predicted 3D oral care representation by marking the one or more aspects for which additional treatment is to be performed; generating an accuracy score specifying a difference between one or more predicted 3D oral care representations and one or more corresponding reference 3D oral care representations that identify the one or more aspects of the first digital 3D oral care representation for which additional processing is to be performed; and At least one aspect of the neural network is modified based on the accuracy score. 2 . The computer-implemented method of claim 1 , wherein identifying the aspect of the first representation comprises assigning one or more tags that specify whether the aspect of the first representation should be additionally processed.

3. The computer-implemented method of claim 2, wherein the accuracy score is normalized and quantifies how well the neural network performed in labeling the first representation.

4. The computer-implemented method of claim 1, wherein the additional processing to be performed includes at least one of a removal operation, a metric generation operation, a marking operation, or a modification operation that does not remove the corresponding aspect.

5. The computer-implemented method of claim 4, wherein the additional processing to be performed is a removal operation on one or more of the identified aspects, and wherein the neural network identifies the one or more aspects to be removed by performing operations comprising: generating one or more prediction values ​​specifying whether a corresponding aspect of the one or more aspects should be removed; comparing the one or more predicted values ​​to a threshold value; as well as When the predicted value is greater than or equal to the threshold, the aspect corresponding to the predicted value is identified as an aspect to be removed. The computer-implemented method of claim 4 , wherein the removing operation causes a hole in the first digital representation. The computer-implemented method of claim 6 , wherein the removing operation is followed by a hole filling operation.

8. The computer-implemented method of claim 4, wherein the removal operation involves at least one of one or more representations of dental hardware or one or more composite-based structures.

9. The computer-implemented method of claim 4, wherein the removal operation is configured to remove an aspect of the first representation corresponding to at least one of: gingival tissue along a trimming line, one or more depressions, one or more undercuts, or one or more internal fractures.

10. The computer-implemented method of claim 7, wherein the holes are repaired using at least one of boundary identification, hole triangulation, a thinning operation, or a smoothing operation.

11. The computer-implemented method of claim 1 , wherein the identified aspects in the digital representation correspond to one or more non-organic geometric shapes.

12. The computer-implemented method of claim 1, wherein the first digital representation is a 3-dimensional (3D) representation of at least one of the dental arches of the designated patient.

13. The computer-implemented method of claim 1, wherein the neural network is initially trained using historical digital representations that identify one or more grid elements for which additional processing is to be performed.

14. The computer-implemented method of claim 1, wherein the method is performed in near real time while the patient is present in a clinical setting.

15. The computer-implemented method of claim 1, wherein the neural network is selected from the group consisting of: Transformer, Autoencoder, Graph CNN, and U-Net architectures.

16. The computer-implemented method of claim 1, wherein the mesh element list specifies whether an aspect of the first digital representation is an edge element, a vertex element, a face element, or a voxel element.

17. The computer-implemented method of claim 1, wherein the neural network is initially trained at least in part using transfer learning.

18. The computer-implemented method of claim 1, wherein the fully trained neural network can be used to at least partially initially train a second neural network using transfer learning.

19. A computer-implemented method for automatically identifying one or more mesh elements for use in digital oral care for further processing using a trained neural network, the method comprising: receiving, by one or more computer processors, digital patient data representing dentition of a patient, wherein the digital patient data is a 3-dimensional (3D) representation and is not generated by mapping one or more 2-dimensional (2D) images onto the 3D representation; generating, by the one or more computer processors, a list of grid elements associated with the digital patient data; Using, by the one or more computer processors, a fully trained neural network that has been fully trained to identify one or more aspects of the one or more digital 3D oral care representations on which to perform additional processing, the additional processing being performed using steps comprising: receiving, by one or more computer processors, a first digital 3D oral care representation of a patient's dentition, wherein the first digital 3D oral care representation is not generated by mapping one or more 2-dimensional (2D) images onto the 3D oral care representation; generating, by the one or more computer processors, a list of mesh elements associated with the first digital 3D oral care representation; using, by the one or more computer processors, the initially trained neural network to identify one or more aspects of the one or more digital 3D oral care representations for which additional processing is to be performed; automatically training, by the one or more computer processors, the neural network based on the use, wherein the training of the neural network is modified by performing operations comprising: identifying, by the neural network, one or more aspects of the first digital 3D oral care representation for which additional processing is to be performed based on the list of mesh elements; generating a predicted 3D oral care representation by marking the one or more aspects for which additional treatment is to be performed; generating an accuracy score specifying a difference between one or more predicted 3D oral care representations and one or more corresponding reference 3D oral care representations that identify the one or more aspects of the first digital 3D oral care representation for which additional processing is to be performed; and modifying at least one aspect of the neural network based on the accuracy score; and A 3-dimensional (3D) output of the patient data is generated, the 3-dimensional (3D) output including the identified elements on which additional processing is to be performed.

20. The computer-implemented method of claim 19, wherein identifying the aspect of the digital patient data comprises assigning one or more tags that specify whether the aspect of the digital patient data should be additionally processed.

Images