Generation of a dental restoration appliance and validation

Abstract

Systems and techniques are disclosed for training one or more machine learning models to generate a digital representation of a dental prosthetic tooth geometry, including generating one or more digital representations defining a prosthetic state of a first digital representation, one or more differences between the one or more predicted representations of the repair state and one or more reference representations of the repair state are determined, and the machine learning model is modified based on the determined differences.

Description

Technical Field

[0001] The present disclosure relates to various improved machine learning techniques for use in digital oral care, including the disciplines of digital dentistry and digital orthodontics. Background Art

[0002] Dentists often use dental appliances to reshape or restore a patient's dental anatomy, or to move teeth using orthodontic appliances. These appliances are typically constructed from a model of the patient's dental anatomy that is modified to the desired end state. The model can be a physical model or a digital model. Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. Problems with past approaches include loss of mapping accuracy and inefficiencies in the data processing that produces the 2D to 3D conversion.

[0003] For example, according to existing implementations, the projection operations performed by existing systems may result in 3D mesh elements receiving conflicting labels due to two or more projection operations. This may require the execution of additional machine learning models to disambiguate those conflicting labels, thereby increasing the complexity and error of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An example processing unit operating in accordance with the techniques of this disclosure is shown.

[0005] Figure 2 Example general techniques for training a generator or other neural network according to various aspects of the present disclosure are shown.

[0006] Figure 3 Example general techniques for using a trained generator or other neural network in accordance with various aspects of the present disclosure are shown.

[0007] Figure 4 Another example general technique for training a generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0008] Figure 5 Another example general technique for using a trained generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0009] Figure 6 An example machine learning architecture according to various aspects of the present disclosure is shown.

[0010] Figure 7Example techniques for performing 2D verification on dental data are shown.

[0011] Figure 8 Example general techniques for validating output generated by a machine learning model in accordance with various aspects of the present disclosure are shown.

[0012] Fig. 9 Example techniques for training machine learning models are shown.

[0013] Figures 10 to 12 are example techniques for generating dental restoration designs in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0014] The present disclosure describes various automation techniques that can be implemented throughout the process of manufacturing dental and orthodontic appliances. Therefore, the present disclosure contemplates improvements in the field of digital oral care, including digital dentistry and digital orthodontics. The automated geometry generation techniques of the present invention are intended to simplify the manufacturing process that would otherwise be extremely time-consuming. Another advantage of these automated geometry generation techniques is that the accuracy of dental appliances is improved. In some cases, the algorithm can produce geometries with higher quality and higher accuracy than those produced by human technicians. Although in some cases, human technicians can modify or "fine-tune" the design output by the automated tool, the automated tool improves the quality of the resulting appliance by providing a common construction baseline for multiple technicians. In addition, by studying the output of the automated tools in the present disclosure (e.g., both the tools for geometry generation and the tools for geometry verification), untrained or new human technicians can understand the appropriate techniques for creating dental and orthodontic appliances (generally used as oral care appliances in this article). The transfer of knowledge to other technicians and the standardization of techniques are important benefits of the disclosed technology. Due to all of the above reasons, another advantage is that more accurate geometry and knowledge transfer can improve the restoration results associated with the use of manufactured dental or orthodontic appliances.

[0015] Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. The technology disclosed herein takes a more direct approach in that mesh elements are labeled directly without the need for intermediate 2D images and the projection of information from those 2D images onto a 3D mesh. Thus, for example, 3D mesh elements can be directly labeled for segmentation and mesh cleaning, which is not possible when using existing systems that rely on 2D mapping techniques. This direct element labeling approach improves the accuracy of the underlying machine learning (ML) model and provides greater efficiency with respect to the use of computational resources because the computational overhead of generating images and mapping them back onto 3D geometry can be avoided.

[0016] As used herein, a 3-dimensional ("3D") mesh (or 3D geometry) includes data corresponding to the edges, vertices, and faces of the 3D mesh. These edges, vertices, and faces are also referred to as digital representations of one or more aspects of a 3D mesh. In some examples, the aspects of a 3D mesh may refer to the shape or geometric features of the mesh. In some cases, the aspects of one mesh may be compared to the aspects of another mesh, such as during a validation operation. Although these three types of data are interrelated, they are different. Vertices are points in 3D space that define the boundaries of a mesh. Therefore, if there is no additional information about how the points are connected to each other, these points can be regarded as point clouds. However, in the context of 3D meshes, edges provide structure to point clouds. An edge includes two points and may also be referred to as a line segment. A face includes both edges and vertices. For example, in the case of a triangular mesh, a face includes three vertices, where these vertices are interconnected to form three continuous edges. Although 3D meshes are typically formed using triangles, other specific implementations may use quadrilaterals, pentagons, or some other n-sided polygons to define 3D meshes. Some meshes may contain degenerate elements, such as non-manifold geometries. Non-manifold geometry is a digital geometry that cannot exist in the real world. For example, one definition of a non-manifold is a 3D shape that cannot be unfolded into a 2D surface, so that the unfolded shape has all surface normal vectors pointing in the same direction. An example of a possible non-manifold geometry is that a face or edge is squeezed but not moved, which causes two identical edges to form on top of each other. Typically, such non-manifold geometry is deleted before processing can be performed. Other mesh preprocessing operations are also possible. The 3D data of each example in the present disclosure can be presented to the ML model as a 3D mesh and / or output from the ML model as a 3D mesh. Other 3D data representations include voxels, finite elements, finite differences, discrete elements, and other 3D geometric representations of dental data and / or appliances. Other specific implementations can use non-discrete methods to describe 3D geometries, thereby using mathematical formulas to regenerate the geometry during processing. Such formulas may contain expressions including polynomials, cosines, and / or other trigonometric or algebraic terms. An advantage of non-discrete formats can be to compress data and save storage space. Digital 3D data may require different coordinate systems such as XYZ (Euclidean), cylindrical, radial, and custom coordinate systems.

[0017] That is, a 3D mesh is a data structure that can describe the structure, geometry, and / or shape of an object related to oral care, including but not limited to teeth, hardware elements, or gum tissue of a patient. The geometry of a 3D mesh can define aspects of the physical size, scale, and / or symmetry of the mesh. The structure of a 3D mesh can define the count, distribution, and / or connectivity of mesh elements. A 3D mesh can include one or more mesh elements, such as one or more vertices, edges, faces, and combinations thereof. In some implementations, mesh elements can include voxels, such as in the context of sparse mesh processing operations. Various spatial and structural features can be calculated for these mesh elements and provided to the predictive models of the present disclosure, with the advantage of improving the accuracy of those predictive models. For example, in some implementations, mesh element features can quantify certain aspects of a 3D mesh that is close to or associated with one or more mesh elements, as described elsewhere in the present disclosure.

[0018] Depending on the specific implementation, it may be beneficial to pre-process the information to generate one or more mesh feature elements. That is, each 3D mesh may undergo pre-processing before being input into a prediction architecture (e.g., including at least one of an encoder, a decoder, an autoencoder, a multi-layer perceptron (MLP), a transformer, a pyramid encoder-decoder, a U-Net, or a graph CNN). Such pre-processing may include converting the mesh into a list of mesh elements such as vertices, edges, faces, or voxels in the case of sparse processing. For one or more selected mesh element types (e.g., vertices), a feature vector may be generated. In some examples, a feature vector is generated for each vertex of the mesh. Each feature vector may contain a combination of spatial and / or structural features, as specified by the following table:

[0019] Consistent with Table 1, a voxel may also have features that are computed as an aggregation of other mesh elements (e.g., vertices, edges, and faces) that either intersect the voxel or, in some implementations, are primarily or completely contained within the voxel. Rotating a mesh may not change structural features, but may change spatial features. Also, as described elsewhere, the term "mesh" should be considered in a non-limiting sense to include 3D meshes, 3D point clouds, and 3D voxelized geometries. In some cases, a 3D point cloud may be derived from the vertices of a 3D triangle mesh.

[0020] Techniques that can operate on the feature vectors of the above features include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental restoration appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification. Such feature vectors can be presented to the input of a prediction model. In some embodiments, such feature vectors can be presented to one or more internal layers of a neural network that is part of one or more of those prediction models.

[0021] But a 3D mesh is only one type of 3D representation that can be used. Therefore, it should be understood that, without loss of generality, there are various types of 3D representations contemplated herein. For example, a 3D representation may include, be one or more of the following, or be a part thereof: a 3D polygonal mesh, a 3D point cloud, a 3D voxelized representation (e.g., a collection of voxels), or a 3D representation described by a mathematical equation. Although the term "mesh" is frequently used throughout this disclosure, in some specific implementations, the term should be understood to be interchangeable with other types of 3D representations. A 3D representation may describe an element of the 3D geometry and / or 3D structure of an object. And a patient's dentition may include one or more 3D representations of the patient's teeth, gums, and / or other oral anatomical structures. Depending on the specific implementation, a 3D scanner (such as an intraoral scanner, a computed tomography (CT) scanner, an ultrasound scanner, a magnetic resonance imaging (MRI) machine, or a mobile device capable of performing stereophotogrammetry) may be used to generate an initial 3D representation.

[0022] In light of the above, the techniques described herein relate to operations performed on 3D representations to perform tasks associated with geometry generation and / or verification. For example, the present disclosure relates to improved automated techniques for segmentation generation and verification, coordinate system prediction and verification, clear tray appliance setup verification, dental prosthetic appliance verification, bracket and attachment (or other hardware) placement and verification, 3D printed component verification, prosthetic design generation and verification, fixture model verification, and clear tray appliance trim line verification, to name a few examples. The present disclosure also relates to improved automated techniques for verifying many of those examples.

[0023] In general, the use of side information ensures that the ML model is insensitive to different input orders of 3D elements. A notable exception is the specific implementation of coordinate prediction, which operates on 3D point clouds instead of 3D meshes. These and other differences are described in more detail below.

[0024] Certain examples in this disclosure mention the use of MeshCNN or encoders to process 3D mesh geometry (e.g., encoder structures for 3D verification and bracket / accessory placement, and MeshCNN for marking mesh elements in segmentation and mesh cleaning). Without limitation, each of these examples may employ other kinds of neural networks to process 3D mesh geometry in addition to or in place of the specified neural network. The following neural networks may be interchangeable in various implementations of the 3D mesh geometry examples of this disclosure: ResNet, U-Net, DenseNet, MeshCNN, Graph-CNN, PointNet, Multilayer Perceptron (MLP), PointNet++, PointCNN, and PointGCN. In other cases, an encoder structure may be used.

[0025] In some cases, the system of the present disclosure may be deployed in a clinical environment (such as a dental or orthodontic clinic) for use by clinicians (e.g., doctors, dentists, orthodontists, nurses, hygienists, oral care technicians). Such systems deployed in a clinical environment may enable clinicians to process oral care data (such as dental scans) in a clinical environment or in some cases in a "chairside" environment (e.g., in near "real time" when the patient is located in the clinical environment). A non-limiting list of examples of techniques may include: segmentation, mesh cleaning, coordinate system prediction, CTA trim line generation, restoration design generation, appliance component generation or placement or assembly, generation of other oral care grids, verification of oral care grids, setting prediction, hardware removal from tooth grids, hardware placement on teeth, estimation of missing values, clustering of oral care data, oral care grid classification, setting comparison, metric calculation, or metric visualization. In some cases, the execution of these techniques may enable patient data to be processed, analyzed, and used by clinicians in appliance creation before the patient leaves the clinical environment (this may facilitate treatment planning because feedback may be received from the patient during the treatment planning process).

[0026] The system of the present disclosure can use representation learning to train ML models. The advantages of representation learning include the fact that, in contrast to receiving inputs of variable size or structure, the generative network (e.g., a neural network that predicts a transformation) is guaranteed to receive inputs of known size and / or standard format. Representation learning can produce performance superior to other methods because the noise in the input data can be reduced (e.g., because the representation generation model extracts important aspects of the input mesh or point cloud through loss calculation or the network architecture selected for this purpose). Such loss calculation methods include KL divergence loss, reconstruction loss, or other losses disclosed herein. Representation learning can reduce the size of the data set required to train the model because the representation model learns the representation, while the generative network can focus on learning the generation task. Since meaningful features are available to the generative network, the result can be an improved model generalization. In some cases, transfer learning can first train the representation generation model. The representation generation model can then be used (in whole or in part) to pre-train subsequent models, such as the generative model (e.g., generating transformation predictions).

[0027] In some cases, federated learning may be used to train the techniques of the present disclosure. Federated learning may enable multiple remote clinicians to iteratively improve machine learning models (e.g., validation of 3D oral care representations, mesh segmentation, mesh cleaning, other techniques involving labeling mesh elements, coordinate system prediction, non-organic object placement on teeth, appliance component generation, dental restoration design generation, techniques for placing 3D oral care representations, setting predictions, generation or modification of 3D oral care representations using autoencoders, generation or modification of 3D oral care representations using transformers, generation or modification of 3D oral care representations using diffusion models, 3D oral care representation classification, imputation of missing values) while protecting data privacy (e.g., clinical data may not need to be sent "over the network" to a third party). Data privacy is particularly important for clinical data that is protected by applicable law. Clinicians may receive a copy of the machine learning model, use a local machine learning program to further train the ML model using locally available data from a local clinic, and then send the updated ML model back to a central hub or a third party. A central hub or third party can integrate updated ML models from multiple clinicians into a single updated ML model that benefits from learning from patient data recently collected at various clinical sites. In this way, new ML models can be trained that benefit from additional and updated patient data (potentially from multiple clinical sites) without that patient data actually being sent to a third party. In some cases, training on devices within a local clinic can be performed when the equipment is idle or otherwise during off-hours (e.g., when a patient is not being treated in the clinic). Equipment used in a clinical environment to collect data and / or train ML models for the techniques described herein may include intraoral scanners, CT scanners, X-ray machines, laptops, servers, desktop computers, or handheld devices (such as smartphones with image collection capabilities).

[0028] In addition to federated learning techniques, in some implementations, contrastive learning can be used to at least partially train the ML models described herein. In some cases, contrastive learning can enhance samples in a training dataset to emphasize differences between samples of different categories and / or increase similarities between samples of the same category.

[0029] Figure 1An example processing unit 102 operating in accordance with the techniques of the present disclosure is shown. The processing unit 102 provides a hardware environment for training one or more neural networks described throughout the specification. Generally speaking, and as will be described in more detail elsewhere, training the one or more neural networks is accomplished by providing one or more training data sets. Therefore, the quality and composition of the training data set for the neural network can have a significant impact on any neural network trained thereby. Data set filtering and outlier removal can be advantageously applied to the training of neural networks of various techniques of the present disclosure (e.g., mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, CTA setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixed body model verification and CTA trim line verification, verification using autoencoders, and setting predictions).

[0030] In the depicted example, the processing unit includes a processing circuit system that may include one or more processors 104 and a memory 106, which in some examples provide a computer platform for executing an operating system 116, which may be, for example, a real-time multitasking operating system or other types of operating systems. In turn, the operating system 116 provides a multitasking operating environment for executing one or more software components (such as applications or other training routines). The processor 104 is coupled to one or more I / O interfaces 114, which provide an I / O interface for communicating with devices such as keyboards, controllers, display devices, image capture devices, other computing systems, and the like. In addition, the one or more I / O interfaces 114 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network. Additionally, the processor 104 may be coupled to an electronic display 108.

[0031] In some examples, processor 104 and memory 106 may be separate, discrete components. In other examples, memory 106 may be on-chip memory co-located with processor 104 within a single integrated circuit. Multiple instances of processing circuit systems (e.g., multiple processors 104 and / or memory 106) may exist within processing unit 102 to facilitate parallel execution of applications and / or processes (including applications and / or processes related to machine learning). Multiple instances may be of the same type, such as a multi-processor system or a multi-core processor. Multiple instances may be of different types, such as a multi-core processor with associated multiple graphics processor units (GPUs). In some examples, processor 104 may be implemented as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or equivalent discrete or integrated logic circuit systems, or a combination of any of the foregoing devices or circuit systems.

[0032] Figure 1 The architecture of the processing unit 102 shown in the figure is shown for example purposes only. The processing unit 102 should not be limited to the example architecture shown. In other examples, the processing unit 102 can be configured in various ways. The processing unit 102 can be implemented as any suitable computing system (e.g., at least one server computer, workstation, mainframe, appliance, cloud computing system, and / or other computing system) capable of performing the operations and / or functions described according to at least one aspect of the present disclosure. As an example, the processing unit 102 may represent a cloud computing system, a server computer, a desktop computer, a server farm, and / or a server cluster (or a portion thereof). In other examples, the processing unit 102 may represent or be implemented by at least one virtualized computing instance (e.g., a virtual machine or container) of a data center, a cloud computing system, a server farm, and / or a server cluster. In some examples, the processing unit 102 includes at least one computing device, each computing device having a memory 106 and at least one processor 104.

[0033] Storage unit 134 may be configured to store information within processing unit 102 during operation (e.g., 3D geometry, transformations to be performed on the 3D geometry, etc.). Storage unit 134 may include a computer-readable storage medium or a computer-readable storage device. In some examples, storage unit 134 includes at least short-term memory or long-term memory. Storage unit 134 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetic disk, an optical disk, a flash memory, a magnetic disk, an optical disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable memory (EEPROM).

[0034] In some examples, storage unit 134 is used to store program instructions for execution by processor 104. Storage unit 134 can be used by software or applications running on processing unit 102 to store information during program execution and to store results of program execution. For example, storage unit 134 can store any number of neural networks 110a-110n, including those described herein. According to some implementations, neural networks 110a-110n can be neural networks trained according to the techniques disclosed herein. In other implementations, one or more of neural networks 110a-110n can be untrained or partially trained.

[0035] As will be described in more detail elsewhere, ML models (e.g., one or more neural networks) can be trained in a supervised and unsupervised manner. Supervised models that can be trained to make recommendations described herein include: regression models (such as linear regression), decision trees, random forests, boosting, Gaussian processes, k-nearest neighbors (KNN), logistic regression, naive Bayes, gradient boosting algorithms (e.g., GBM, XGBoost, LightGBM, and CatBoost), support vector machines (SVM), or fully connected neural network models that have been trained for classification. In some cases, a multilayer perceptron (MLP) can be used to predict missing program parameters based on known program parameters.

[0036] Unsupervised models that can be trained to make the recommendations described herein include clustering techniques such as K-means clustering, density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture models, balanced iterative reduction and clustering based on hierarchy (BIRCH), affinity propagation clustering, mean shift clustering, ordering points to identify cluster structures (OPTICS), agglomerative hierarchical clustering, and spectral clustering.

[0037] Whether the training is supervised or unsupervised, there are a variety of optimization methods that can be used to train the neural network of the present disclosure (e.g., update the neural network weights), including gradient descent (which uses first-order derivatives to determine the training gradient and is commonly used in the training of neural networks), Newton's method (which can use second-order derivatives in the loss calculation to find a better training direction than gradient descent, but may require the calculation of the Hessian matrix), and conjugate gradient method (which may converge faster than gradient descent, but does not require the Hessian matrix calculation that may be required by Newton's method). In some specific implementations, in addition to or instead of the aforementioned methods, other methods may be used to update the weights. These other methods include: Levenberg-Marquardt method and simulated annealing. The back propagation algorithm is used to transmit the results of the loss calculation back to the network so that the network weights can be adjusted to perform learning.

[0038] Neural networks facilitate the functionality of many applications of the present disclosure, including, but not limited to: mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, CTA setup verification, bracket / accessory / hardware placement verification, generation of custom oral care appliance components, placement of custom oral care appliance components, verification of custom oral care appliances (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders. The neural networks of the present disclosure may embody portions or all of a variety of different neural network models. Examples include U-Net architecture, multi-layer perceptron (MLP), transformer, pyramid architecture, recurrent neural network (RNN), autoencoder, variational autoencoder, regularized autoencoder, conditional autoencoder, capsule network, capsule autoencoder, stacked capsule autoencoder, denoising autoencoder, sparse autoencoder, conditional autoencoder, long / short term memory (LSTM), gated recurrent unit (GRU), deep belief network (DBN), deep convolutional network (DCN), deep convolutional inverse graphics network (DCIGN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), neural Turing machine (NTM) and generative adversarial network (GAN). In some implementations, an encoder structure or a decoder structure can be used. Each of these models has its own specific advantages. A specific model may be particularly suitable for one or another model.

[0039] In some implementations, the neural networks of the present disclosure may be adapted to operate on 3D point cloud data (alternatively 3D meshes or 3D voxelized representations). Many neural network implementations may be applicable to the processing of 3D representations and may be applied to training predictive and / or generative models for oral care applications, including: PointNet, PointNet++, SO-Net, Spherical Convolution, Monte Carlo Convolution and Dynamic Graph Networks, PointCNN, ResNet, MeshNet, DGCNN, VoxNet, 3D-ShapeNets, Kd-Net, Point GCN, Grid-GCN, KCNet, PD-Flow, PU-Flow, MeshCNN, and DSG-Net. Oral care applications include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, CTA setup verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliance components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification and CTA trim line verification, verification using autoencoder, setup prediction, and generation of dental prosthetic appliances.

[0040] In some specific implementations, some techniques of the present disclosure may use autoencoders. Possible autoencoders include, but are not limited to: AtlasNet, FoldingNet, and 3D-PointCapsNet. Some autoencoders may be implemented at least in part based on PointNet.

[0041] Some techniques of the present disclosure relate to hardware placement. ML models for this purpose can be enhanced by using representation learning. For example, representation learning may involve training a first neural network to learn a representation of teeth, and training the same neural network or a second neural network to learn a representation of hardware, and then using a third neural network to generate a transformation for the hardware to place the hardware on the teeth. In other embodiments, one or more appliance components may be placed relative to one or more teeth. Some embodiments may use U-Net to generate representations. Some embodiments may use autoencoders (such as VAEs or capsule autoencoders) to learn representations of the basic features of one or more meshes related to the field of oral care (including information about the structure of the tooth mesh in some cases). Then, a potential representation (latent vector or latent capsule) may be used as an input to a module that generates one or more transformations for one or more hardware elements or appliance components. In some embodiments, these transformations may place hardware elements or appliance components into the pose required for appliance generation (e.g., a dental prosthetic appliance or an indirect bonding tray). In some embodiments, the transformation may be described by a 9×1 transformation vector (e.g., specifying a translation vector and a quaternion). In other implementations, the transformation can be described by a transformation matrix (e.g., a 4×4 affine transformation matrix). In some implementations, a principal component analysis can be performed on the oral care grid, and the resulting principal components can be used as at least part of a representation of the oral care grid in a later machine learning and / or other predictive or generative process.

[0042] Depending on the particular implementation, additional methods may also be used to improve the performance of the ML model. For example, end-to-end training may be applied to the disclosed techniques involving two or more neural networks, where the two or more neural networks are trained together (e.g., weights are updated simultaneously during the processing of each batch of input oral care data). In some implementations, end-to-end training may be applied to hardware / component placement by simultaneously training a neural network that learns representations of one or more oral care objects and a neural network that processes those representations.

[0043] Another way to improve the ML models described herein is to use transfer learning. In some specific implementations, a network (e.g., U-Net) can be trained for a first task (e.g., coordinate system prediction) and then used to provide one or more starting neural network weights for the training of another neural network, which is trained to perform a second task (e.g., setting prediction). The first network can learn low-level neural network features of an oral care grid and is shown to perform well in the first task. By using the first network as a starting point for training, the second network can experience faster training and / or improved performance. Certain layers can be trained to encode the neural network features of the oral care grid in the training data set. These layers can then be fixed (or receive small fine-tuning during training) and combined with other neural network components (such as additional layers) that are trained for one or more oral care tasks. In this way, a portion of a neural network used for one or more technologies of the present disclosure can receive initial training for another task, which can produce important learning in the trained network layers. This encoded learning can then be built on with further task-specific training. In some implementations, a neural network used to make predictions based on an oral care grid can first be partially trained on one or more general / publicly available datasets and then further trained on oral care data.

[0044] In some embodiments, a neural network previously trained on a first data set (oral care data or other data) may subsequently receive further training on oral care data and be applied to oral care applications (such as mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, CTA setup verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances or components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders). Transfer learning may be used to further train any of the following networks from published literature: GCN (Graph Convolutional Network), PointNet, ResNet, or any other neural network from published literature listed earlier in this section.

[0045] Yet another approach involves adding attention gates to ML models. In general, attention gates may be integrated with one or more of the neural networks of the present disclosure, with the advantage of enabling the associated neural network architecture to focus on one or more input values. In some implementations, attention gates may be integrated with a U-Net architecture, with the advantage of enabling the U-Net to focus on certain inputs. Attention gates may also be integrated with encoders or autoencoders (such as VAEs or capsule autoencoders). Some implementations of the presently disclosed techniques may benefit from one or more attention layers in a transformer, where the transformer is trained to generate a 3D oral care representation.

[0046] Figure 2 is an example technique 200 that can be used to train the ML models described herein. In general, the receiving module 202 is configured to receive patient case data 204. Typically, the patient case data 204 represents a digital representation of a patient's mouth. This may mean, for example, that the receiving module 202 may receive one or more malocclusion arches (e.g., 3D meshes representing upper and lower arches of a patient's teeth, i.e., a dentition of a patient's mouth including multiple aspects of the patient's dental anatomy, which may include teeth and may include gums). Depending on the specific implementation, the malocclusion arches may be arranged in an occlusal position or other orientation. In other implementations, a single arch may be required. For purposes of illustration, additional implementations will be described in more detail below. In other words, the receiving module 202 may receive mesh data corresponding to a 3D mesh of one or more patient's dentitions. It should be understood that both the amount of 3D mesh data and the type of 3D mesh data received by the receiving module 202 as part of the patient case data may vary based on the specific implementation. For example, in an implementation regarding verification of bracket placement, the mesh data received as part of the patient case data 204 may include only 3D mesh data regarding a particular tooth and associated bracket, while in an implementation regarding verification of a 3D printed component, the 3D data received as part of the patient case data 204 may include 3D mesh data associated with a component examined in the form of a CT scan or other diagnostic image, to name a few additional examples. Depending on the particular implementation, the patient case data 204 may also include a 3D representation of the patient's gingival tissue.

[0047] As shown in the example, the receiving module 202 also receives "reference truth" data 206. Generally speaking, these "reference truth" data 206 specify the expected results of applying other technologies disclosed herein, which are mesh segmentation, coordinate system prediction, mesh cleaning, restoration design and bracket / attachment placement, as well as all verification applications of the present disclosure, just to name a few examples. "Reference truth" and "reference" as used herein will be used interchangeably. For example, it should be understood that for the purposes of the present disclosure, a "reference" transformation vector is equivalent to a "reference truth" transformation vector. According to a specific implementation, and as will be described in more detail below, the "reference truth" data 206 may include a "reference truth" unique hot vector describing the expected transformation of a 3D geometry. For another example, the "reference truth" data 206 may include expected labels for various aspects of the 3D geometry. Other examples are also provided below. According to a specific implementation, the "reference truth" data 206 may be predefined or provided as a result of executing one or more other technologies disclosed herein.

[0048] Depending on the particular implementation, the receiving module 202 may also be configured to perform data augmentation on one or more aspects of the received data, including the patient data 204 and the "ground truth" data 206. Data augmentation is described in more detail below.

[0049] The system 100 can be configured to provide each mesh received by the receiving module 202 to the mesh preprocessor module 205, thereby allowing for preprocessing of any 3D mesh data received in the patient case data 206. This preprocessing step allows the system to convert the mesh into a form that allows the input mesh to be "consumed" by a neural network or other ML technique. In one specific implementation, the mesh preprocessor module 205 can be configured to generate a combination of edge, vertex, and face lists. One or more of these generated lists can be provided to both the generator 211 and the mesh feature module 208, as described in more detail below.

[0050] In addition to utilizing the mesh preprocessor module 205, the system 100 may also perform a number of additional operations before and after providing the patient case data 204 to the mesh preprocessor module 205. For example, according to a particular implementation, the system 100 may perform mesh cleaning on the patient case data 204 before providing the patient case data 204 to the mesh preprocessor module 205. Additionally, the system 100 may resample or update any information generated by the mesh preprocessor module 205. For example, in an implementation in which the mesh preprocessor module 205 generates a combination of edge, vertex, and face lists, the system may resample, update, or otherwise modify the labels identified in those lists. Additionally, according to a particular implementation, the system 100 may perform data augmentation on the resampled data.

[0051] The mesh feature module 208 may be configured to receive the list generated by the mesh preprocessor module 205 and generate feature information related thereto, which may be used by the ML model to generate predictions. For example, in one implementation, the mesh feature module 208 may calculate one or more of the following: edge midpoints, edge curvatures, edge normal vectors, edge normalized vectors, edge translation vectors, and other information related to each tooth in the 3D mesh received by the receiving module 202. Depending on the particular implementation, the mesh feature module 208 may or may not be utilized. That is, it should be understood that the calculation of any of the edge midpoints, edge curvatures, edge normal vectors, and edge translation vectors of the 3D mesh data included in the patient data 206 is optional. One advantage of using the mesh feature module 208 is that a system utilizing the mesh feature module 208 may be trained more quickly and accurately, but the technique 200 still performs better than prior art techniques that do not utilize the mesh feature module 208.

[0052] The technology 200 also utilizes a generative adversarial network ("GAN") to achieve certain aspects of the improvement. Generally speaking, a GAN is a ML model in which two neural networks "compete" with each other to provide predictions, evaluate these predictions, and then use the evaluation of the two models to improve each other's training. In some specific implementations, the GAN can be a conditional GAN, in which the generated output is adjusted based on some input data. An example in which conditional GANs have been found to provide benefits is in the field of restoration design. In those specific implementations, these conditional input data can be unstored grids and associated instructions or prescription information. The instructions or prescription information (or other oral care parameters) describing the results of the dental restoration can include text, real numbers, integers, or categorical values. In some specific implementations, and as will be described below, the textual provisions can be a process that uses natural language processing (NLP) to extract key values, such as the added height or added width that has been prescribed for each treated tooth (for example, in the example of dental restoration design, it produces a target geometry for each treated tooth).

[0053] As shown in this example, the two neural networks of the GAN are the generator 211 and the discriminator 235. In other implementations, models other than neural networks can be used for the generator or the discriminator.

[0054] The generator 211 receives input (e.g., one or more 3D meshes included in the patient case data 206). Depending on the particular implementation, the generator 211 uses the received input to determine a predicted output 207 related to the 3D mesh. For example, for segmentation, the generator 211 may be configured to predict a segmentation label, and in an implementation of predicting a clear tray appliance setting, the prediction may include one or more vectors corresponding to one or more transformations applied to the 3D mesh included in the patient case data 206. Other predicted outputs 207 are also possible. In some implementations, the generator 211 may also receive random noise, which may include junk data or other information that may be used to purposefully attempt to confuse the generator 211. Depending on the particular implementation, and as described above, the generator 211 may implement any number of neural networks, including MeshCNN, ResNet, U-Net, and DenseNet. In other cases, the generator may implement an encoder.

[0055] Since the generator 211 can be implemented as one or more neural networks, the generator 211 can include an activation function. The activation function determines whether a neuron in the neural network will fire (e.g., send an output to the next layer). Some activation functions may include: a binary step function and a linear activation function. Other activation functions give the network nonlinear behavior, including: a sigmoid / logistic activation function, a Tanh (hyperbolic tangent) function, a rectified linear unit (ReLU), a leaky ReLU function, a parametric ReLU function, an exponential linear unit (ELU), a softmax function, a swish function, a Gaussian error linear unit (GELU), and a scaled exponential linear unit (SELU). The linear activation function is well suited for some regression applications (and other applications) in the output layer. In the output layer, the sigmoid / logistic activation function is well suited for some binary classification applications (and other applications). The sigmoid activation function is well suited for some multi-class classification applications (and other applications) in the output layer. In the output layer, the sigmoid activation function is well suited for some multi-label classification applications (and other applications). The ReLU activation function works well for some Convolutional Neural Network (CNN) applications in the hidden layers (among other applications). The Tanh and / or sigmoid activation functions work well for some Recurrent Neural Network (RNN) applications, for example, in the hidden layers (among other applications).

[0056] After the generator 211 determines one or more predicted outputs 207, the generator 211 may be trained. Generally speaking, training the generator 211 involves comparing the predicted outputs 207 to corresponding ground truth inputs 208. For example, the predicted output 207 associated with the lower left canine corresponding to number twenty-seven of the International Dental Numbering System is compared to the ground truth output 208 for the same canine. As previously described, the ground truth input is an input that has been verified as a correct label for a particular portion of the 3D mesh data included in the patient case data 206. Depending on the particular implementation, the ground truth input 208 may be derived or otherwise determined from the ground truth data 206, or may be the ground truth data 206.

[0057] The differences between the predicted output 207 and the reference true input 208 can be used to calculate one or more loss values ​​G1216. For example, these differences can be used as part of the loss function calculation, or for the calculation of reconstruction errors. Some specific implementations may involve comparing the volumes and / or areas of two meshes (i.e., representations 207 and 208). Some specific implementations may involve calculating the minimum distance between corresponding vertices / faces / edges / voxels of the two meshes. For a point in one mesh (e.g., a vertex, a midpoint on an edge, or a triangle center), the minimum distance between the point and the corresponding point in the other mesh is calculated. Different methods can be considered in the case where the other mesh has a different number of elements or there is no clear mapping between corresponding points of the two meshes.

[0058] Regardless of the manner in which the difference between the predicted output 207 and the ground truth input is determined, various loss values ​​may be determined as part of the technique 200 or any other technique described herein. These losses include L1 loss, L2 loss, MSE loss, cross entropy loss, etc. Losses may be calculated and used to train neural networks such as multi-layer perceptrons (MLPs), U-Net structures, generators and discriminators (e.g., for GANs), autoencoders, variational autoencoders, regularized autoencoders, masked autoencoders, transformer structures, etc. For example, in learning sequences, some implementations may use triplet loss or contrastive loss.

[0059] The loss can also be used to train the encoder structure and the decoder structure. The KL divergence loss can be used at least in part to train one or more neural networks of the present disclosure, such as a mesh reconstruction autoencoder (e.g., a tooth reconstruction autoencoder with an optional normalization flow, which can be trained to reconstruct specific teeth, such as the lower right central incisor or the upper left canine), which has the advantage of imparting Gaussian behavior to the optimization space. This Gaussian behavior can enable the reconstruction autoencoder to produce better reconstructions (i.e., when the latent vector representation is modified and the modified latent vector is reconstructed using the decoder, the resulting reconstruction is more likely to be a valid instance of the input representation). There are other techniques for calculating the loss that can be described elsewhere in the present disclosure. Such a loss can be based on quantizing the difference between two or more 3D representations.

[0060] The mean squared error (MSE) loss may involve the computation of the average squared distance between two sets, vectors, or data sets. The MSE may generally be minimized. The MSE may be applied to regression problems where the predictions generated by a neural network or other ML model may be real numbers. In some implementations, the neural network may be equipped with one or more linear activation units at the output to generate the MSE predictions. Mean absolute error (MAE) loss and mean absolute percentage error (MAPE) loss are also possible.

[0061] In some implementations, cross entropy can be used to quantify the difference between two or more distributions. In some implementations, cross entropy loss can be used to train the neural network of the present disclosure. In some implementations, cross entropy loss may involve comparing the predicted probability with the benchmark true probability. Other names for cross entropy loss include "logarithmic loss", "logistic loss" and "log loss". A small cross entropy loss may indicate a better (ie, more accurate) model. The cross entropy loss may be logarithmic. In some implementations, cross entropy loss may be applied to binary classification problems. In some implementations, the neural network may be equipped with a sigmoid activation unit at the output to generate a probability prediction. In the case of multi-class classification, cross entropy may also be used. In this case, in some implementations, a neural network that has been trained to perform multi-class predictions may be equipped with one or more softmax activation functions at the output (e.g., where there is an output node for the class to be predicted).

[0062] Other loss calculation techniques that can be applied in the training of the neural network of the present disclosure include one or more of the following: Huber loss, hinge loss, classification hinge loss, cosine similarity, Poisson loss, Logcosh loss, or mean square logarithmic error loss (MSLE). Other loss calculation methods are described herein and can be applied to the training of any neural network described in the present disclosure.

[0063] In some implementations, one or more neural networks of the present disclosure may be trained, at least in part, by a loss based on at least one of: point-wise mesh Euclidean distance (PMD) and earth mover distance (EMD). Some implementations may incorporate a Hausdorff distance (HD) calculation into the loss calculation. Calculating the Hausdorff distance between two or more 3D representations (such as 3D meshes) may provide one or more technical improvements because the HD considers not only the distance between the two meshes, but also the way those meshes are oriented and the relationship between the shapes of the meshes in those orientations (or positions or poses). The Hausdorff distance may improve comparisons of two or more instances of two or more tooth meshes, such as tooth meshes in different poses (e.g., comparisons of predicted settings to baseline true settings, which may be performed in the process of calculating a loss value for training a setting prediction neural network).

[0064] See again Figure 2 , G1 216 may represent the regression loss between the predicted output 207 and the ground truth input 208. That is, according to one implementation, the loss G1 216 reflects the percentage by which the predicted output 207 deviates from the ground truth input 208. That is, the generator loss G1 216 may be an L2 loss, a smoothed L1 loss, or some other type of loss. According to a particular implementation, the L1 loss is defined as , where P represents the predicted output 207 and G represents the ground truth input 208. Depending on the specific implementation, the L2 loss can be defined as , again where P represents the predicted output 207 and G represents the reference true input 208. In addition, and as will be described in more detail below, a loss value G1 216 may be provided to the generator 211 to further train the generator 211, for example, by modifying one or more weights in the neural network of the generator 211 to train the underlying model and improve the model's ability to generate a predicted output 207 that mirrors or substantially mirrors the reference true input 208. Any of these losses may be used to provide a loss value for training the neural network through a suitable training algorithm (such as back propagation). In some cases, an accuracy score may be used for training the neural network. The accuracy score quantifies the difference between the predicted data structure and the reference true data structure. The accuracy score (e.g., in normalized form) may be fed back into the neural network during the training of the network, for example, through back propagation. In the case of segmentation, the accuracy score may count the matching labels between the predicted grid and the reference true grid (i.e., where each grid element has an associated label). The higher the percentage of matching labels, the better the prediction (i.e., when comparing the predicted labels to the reference true labels). A similar accuracy score can be calculated in the case of mesh cleaning, which also predicts labels for mesh elements. The number or percentage of matches between the predicted labels and the ground truth labels can be used as an accuracy score, which can be used to train the neural network driving mesh cleaning (i.e., the accuracy score can be normalized).

[0065] Additionally, depending on the particular implementation, the system 100 can use the predicted output 207 to generate a predicted representation 220. Furthermore, the system 100 can use the reference ground truth input 208 to generate a reference ground truth representation 211. For example, in an implementation related to clear tray appliance generation, the predicted transformation and the reference ground truth transformation can be applied to the patient case data 206 to generate a predicted transformation and a reference ground truth transformation of the patient case data 206.

[0066] Depending on the particular implementation, the predicted representation 220 and the ground truth representation 211 may be labeled or otherwise annotated to indicate whether the representation corresponds to the ground truth data 206. Additionally, depending on the particular implementation, the representation 220 may be assigned a "false" value to indicate that the representation does not correspond to the ground truth label 208, while the representation 221 may be assigned a "true" value.

[0067] According to a particular implementation, representations 220 and 221 are provided as inputs to discriminator 235. Additionally, according to a particular implementation, 3D mesh data in patient case data 206 is also provided to discriminator 235. That is, discriminator 235 may receive various representations of data corresponding to patient case data 206, predicted output 207, ground truth data 206, ground truth input 208, and representations 220 and 221. In general, discriminator 235 is configured to determine when an input is generated from predicted output 207 or when an input is generated from ground truth input 208. The output of discriminator 235 is described in more detail in conjunction with the implementations discussed herein.

[0068] The discriminator 235 may be initially trained in various ways. For example, the discriminator 235 may be configured as an encoder structure, which in some cases (such as those described herein) may be configured to perform verification when used as a generator. For example, the initial encoder included in the discriminator 235 may be configured with random edge weights. Using back propagation, the encoder and thus the discriminator 235 may be continuously improved by modifying the value of the weights to allow the discriminator 235 to more accurately determine which inputs should be identified as "true" reference true representations and which inputs should be identified as "false" reference true representations. In other words, while the discriminator 235 may be initially trained, the discriminator 235 continues to develop / be trained as the technique 200 is executed. And similar to the generator 211, the accuracy of the discriminator 235 improves with each execution of the technique 200. Although, as will be appreciated by one of ordinary skill in the art, the improvement to the discriminator 235 will reach a limit where the accuracy of the discriminator 235 is not statistically improved, the training of the discriminator 235 is considered to be complete at this point. In other words, the system 100 may consider the training of both the generator 211 and the discriminator 235 to be complete when the discriminator 235 has difficulty distinguishing the predicted representation 220 from the reference true representation 221. As used herein, when the training of the generator 211 and the discriminator 235 is complete, they are described as being fully trained.

[0069] After the discriminator 235 generates an output, the technique 200 then compares the output of the discriminator 235 with the input to determine whether the discriminator 235 accurately distinguishes the predicted representation 220 from the reference ground truth representation 221. For example, the output of the discriminator 235 can be compared to the annotation of the representation. If the output and the annotation match, then the discriminator 235 accurately predicts the type of input received by the discriminator 235. Conversely, if the output and the annotation do not match, then the discriminator 235 inaccurately predicts the type of input received by the discriminator 235. In some implementations, and similar to the generator 211, the discriminator 235 can also receive random noise to intentionally attempt to confuse the discriminator 235.

[0070] In addition, depending on the particular implementation, the discriminator 235 may generate additional values ​​that can be used to train various aspects of the system implementing the technique 200. In one example, the discriminator 235 may generate a discriminator loss value 236 that reflects how accurately the discriminator 235 determines whether an input corresponds to the predicted representation 220 and / or the ground truth representation 221. Depending on the particular implementation, the discriminator loss 236 is larger when the prediction of the discriminator 235 is less accurate, and smaller when the prediction of the discriminator 235 is more accurate. In another example, the discriminator 235 may generate a generator loss value G2 238. Depending on the particular implementation, although not directly inversely proportional to the discriminator loss 236, the generator loss value G2 238 generally exhibits an inversely proportional relationship with the discriminator loss 236. That is, when the discriminator loss 236 is larger, the generator loss G2 238 is smaller, and when the discriminator loss 236 is smaller, the generator loss G2 238 is larger. In some implementations, the discriminator loss 236 may be determined using a binary cross entropy loss function calculated for both the "true" model and the "fake" model. In some implementations, the generator loss may consist of two losses: 1) the first loss is the generator loss G2 238 as determined by the discriminator (so binary cross entropy may be used); and 2) the second loss may be implemented by the l1-norm or mean squared error measuring the difference between the desired output and the actual output of the generator 211, for example as specified by the generator loss G1 216.

[0071] In other words, Figure 2 As shown, the generator loss G2 238 can be added to the generator loss G1 216 using a summation operation 240. And the sum of the generator loss G1 216 and G2 238 can be provided to the generator 211 for the purpose of training the generator 211. That is, it should be understood that the calculation of the generator loss G1 216 is for Figure 2 Training of the GAN shown is not required. In some implementations, it is possible to train the generator 211 or the discriminator 235 using only a combination of the generator loss G2 238 and the discriminator loss 236. However, like other optional aspects of the present disclosure, the use of the generation loss G1 216 can be used to train the discriminator 235 more quickly to produce more accurate predictions. Depending on the specific implementation, the system 100 may use other steps or operations as part of the described techniques. For example, as already described but not depicted, an implementation related to the clear tray-type aligner setting may use one or more transformation steps to transform the patient data 206 using a predicted output 207 and a reference real input 208 corresponding to one or more 3D mesh transformations (e.g., scaling, rotation, and / or translation operations).

[0072] Depending on the particular implementation, the loss G1 216 and the loss G2 238 may also include one or more reasoning metrics specifying one or more differences between the predicted output 207 and the ground truth input 208 and / or the predicted representation 202 and the ground truth representation 221. That is, as an optional step, the system 100 may generate these reasoning metrics to further refine the training of one or more neural networks or ML models. These reasoning metrics may include: intersection over union metric, average boundary distance metric, boundary percentage metric, and over segmentation rate, to name a few examples.

[0073] In general, the intersection-over-union metric specifies the percentage of correctly predicted edges, faces, and vertices within a mesh after a certain operation (such as segmentation) is completed. The average boundary distance specifies the distance between the predicted output 207 (or predicted representation 220) and the reference true input 208 (or reference true representation 221) for a 3D representation (such as a 3D mesh). The boundary percentage specifies the percentage of the mesh boundary length of a 3D mesh (such as a segmented 3D mesh) where the distance between the reference true input 208 (or reference true representation) and the predicted output 207 (or predicted representation 220) is below a threshold. For example, the threshold may determine whether one or more predicted outputs 207 (such as a small line segment between each pair of boundary points) are close enough to the reference true input 208. In the case where the technique 200 is used to implement a segmentation process, if the distance is below the threshold, the system 100 may mark a particular line segment as a perfect boundary segment. The percentage represents the ratio of segments that are within the predicted boundary to segments that are within the reference true boundary. And the over-segmentation rate specifies the percentage of the boundary length of the teeth that are over-segmented. Depending on the specific implementation, one or more inference metrics can be used to additionally train the generator 211 or the discriminator 235, or both.

[0074] The technology of the present disclosure may include operations such as 3D convolution, 3D pooling, 3D deconvolution, and 3D up-pooling. 3D convolution can help segmentation processing, such as when downsampling a 3D representation (such as a 3D mesh or point cloud). 3D deconvolution is, for example, the inverse operation of 3D convolution in U-Net. 3D pooling can help segmentation processing, such as in a generalized neural network feature map. 3D up-pooling is, for example, the inverse operation of 3D pooling in U-Net. These operations can be implemented by one or more layers in the prediction or generation neural network described herein. These operations can be directly applied to aspects of the 3D representation such as mesh elements, which may include mesh edges or mesh faces. These operations provide technical improvements over other methods because these operations are invariant to mesh rotation, scaling, and translation changes. Generally speaking, these operations depend on edge (or face) connectivity, so as long as the edge (or face) connectivity is maintained, these operations will not be affected by mesh changes in 3D space. That is, these operations can be applied to an oral care mesh and produce the same output regardless of the orientation, position, or scale of the oral care mesh, which can improve data accuracy. MeshCNN is a general deep neural network library for 3D triangulated meshes that can be used for tasks such as 3D shape classification or mesh element labeling (e.g., for segmentation or mesh cleaning). MeshCNN implements these operations on mesh edges. Other toolkits and specific implementations can operate on edges or faces.

[0075] The technique 200 can be used to train ML models for many digital dental and digital orthodontic applications. Table 2 shows how the technique 200 can receive different data 204 and 206 for certain digital dental applications, and the form that the prediction output 207 can take depending on the particular implementation.

[0076] ML models (such as those described herein) can be trained to generate transformations to place prefabricated components (e.g., from a component library) for creating dental restoration appliances. Such dental restoration appliances can be used to shape dental composites in a patient's mouth while curing the composites (e.g., using a curing light), ultimately creating veneers on one or more teeth of the patient. 3M FILTEK Matrix is ​​an example of such a product. Dental restoration appliance components (e.g., library components) that can be placed using the technology of the present disclosure include: drainage holes (e.g., which can allow composites to flow out of the appliance), rear snap clips (e.g., which can enable the appliance to be grasped or manipulated), door hinges (e.g., which can enable the door to be rotated open or closed), door snaps (e.g., which can secure the door in a closed position), incisal alignment features (e.g., which can help align the appliance), center clips (e.g., which can enable the appliance to be aligned), custom labels, manufacturing box frames, diastema matrix handles, and the like. Further details on placing features and generating features can be found in PCT patent application WO2021 / 240290A1, the entirety of which is incorporated herein by reference.

[0077] For example, in a segmentation implementation, each patient case in the data set 204 consists of a pre-segmented dental arch. In some implementations, the technique 200 can be used to segment each tooth in the dental arch and label the tooth with its identity (i.e., performing traditional tooth segmentation). In some implementations, the technique 200 can be used to separate the face and tongue of the dental arch (i.e., performing face-tongue segmentation). In some implementations, the technique 200 can be used to separate the gingival portion of the dental arch from the teeth (i.e., performing gingival segmentation). In some implementations, the technique can be used to segment extraneous material directly from the gums (i.e., performing trim line segmentation). Some segmentation implementations may use MeshCNN to predict mesh element labels. Some implementations may train a U-Net structure to generate a representation of a 3D mesh and may also be simultaneously trained to predict mesh element labels. Other implementations may use other models to predict mesh element labels.

[0078] As described elsewhere in this specification, the receiving module 202 receives patient case data. In the depicted example, the receiving module 202 may receive patient case data 204 including dental arch data after one or more mesh cleaning operations have been performed on the patient's 3D dental arch geometry. For example, in one example, this may produce one or more cleaned dental arch geometries. The mesh cleaning operations may use one or more models of MeshCNN, U-Net, or other models to predict mesh element labels.

[0079] According to a particular implementation, the 3D dental arch geometry may include a 3D mesh geometry for the patient's gingival tissue, while in other implementations, the 3D dental arch geometry may omit the 3D dental arch geometry for the patient's gingival tissue. In addition, the receiving module 202 may be configured to also receive a reference true label as a reference true label 206, which describes a verified or otherwise known accurate label (e.g., labels "correct" and "incorrect") of a mesh element associated with a segmentation result performed on the 3D geometry. According to a particular implementation, the labels described with respect to the segmentation operation are used to specify a specific set of mesh elements (such as "edge" elements, "face" elements, "vertex" elements, etc.) for specific aspects of the 3D geometry. For example, a single triangular polygon of a 3D mesh includes 3 edge elements, 3 vertex elements, and 1 face element. Therefore, it should be understood that a segmented dental geometry consisting of many polygons may have a large number of labels associated with the segmented dental geometry.

[0080] In addition, the received geometry may have one or more labels that are applied to the corresponding geometry to generate representations 220 and 221. For example, in one embodiment, at each iteration of generator 211, generator 211 may output a label for each mesh element found in the input dental arch. Each of these labels marks the corresponding mesh element (e.g., an edge) as belonging to a gingival or tooth structure in the input mesh. In the case where the mesh element belongs to a tooth, the identity of the tooth is also specified. For example, an edge may be assigned a label to indicate that the mesh element belongs to the gingiva. Another mesh element may be assigned a label to indicate that the mesh element belongs to the upper right third molar. Yet another mesh element may be assigned a label to indicate that the mesh element belongs to the lower left central incisor. Other labels are also possible.

[0081] Once trained, the generator 211 may be used to generate accurate prediction outputs 207 for the patient case data 206 received by the receiving module 202 . Figure 3 An example technique 300 for generating predicted labels 207 is shown in FIG. Generally speaking, technique 300 uses the same computer modules and components to perform many of the same steps as technique 200. That is, as can be seen from this example, technique 300 does not train generator 211, but relies on the training in technique 200 to generate predicted output 307. In addition, technique 300 does not include a discriminator. As discussed above with respect to Figure 2 It should be understood from the discussion of that when generator 211 is trained, predicted output 207 will eventually be equal to or substantially equal to predicted output 307.

[0082] Some of the techniques described in Table 2 (and elsewhere in this disclosure) can benefit from training a representation learning model. In some implementations, such a representation model can be used to implement Figure 2, Figure 3 , Figure 4 and Figure 5 211 in the generator. In some embodiments, the representation learning model may include a first module and a second module, the first module may be trained to generate representations of received 3D oral care representations (e.g., teeth, gums, hardware and / or appliance components), and the second module may be trained to receive those 3D representations and generate one or more output oral care representations. In some cases, such output oral care representations may include transformations that may be applied to hardware or appliance components for placement relative to one or more teeth. In some cases, such output oral care representations may include one or more coordinate system axis definitions. In some cases, such output oral care representations may include meshes or labels on mesh elements corresponding to teeth, gums, or other aspects of the dentition (e.g., utilizing mesh cleaning, mesh segmentation, or dental restoration design).

[0083] In some implementations, a first module of a representation learning model may be trained to generate a 3D representation of one or more teeth (and / or gums or hardware) suitable for providing to a second module, wherein the second module is trained to output one or more predicted transforms (or other oral care representations). In some implementations, one or more layers including convolution kernels (e.g., with kernel size 5 or some other size) and pooling operations (e.g., average pooling, max pooling, or some other pooling method) may be trained to create representations for one or more received oral care 3D representations in the first module. In some implementations, one or more U-Nets may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more autoencoders may be trained to generate representations for one or more received oral care 3D representations in the first module (e.g., wherein a 3D encoder of the autoencoder is trained to convert one or more tooth 3D representations into one or more latent representations, such as latent vectors or latent capsules, wherein such latent representations may be reconstructed into replicas of one or more input tooth meshes via a 3D decoder of the autoencoder). In some implementations, one or more 3D encoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more pyramid encoder-decoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. Other methods of encoding representations are also possible.

[0084] The representation of one or more teeth may be input to a second module of a representation learning model, such as an encoder structure, a multi-layer perceptron (MLP), a transformer (e.g., including at least one of a 3D encoder and a 3D decoder, which may be configured with a self-attention mechanism that enables the network to focus training on key inputs), an autoencoder (e.g., a variational autoencoder or a capsule autoencoder), which has been trained to output one or more representations (e.g., a transformation to place an oral care grid, such as the grid in the example of hardware and appliance component placement technology). In some embodiments, the transformation may include one or more 4×4 matrices, Euler angles, or quaternions. The second module may be trained at least in part by calculating one or more loss values ​​(such as L1 loss, L2 loss, MSE loss, reconstruction loss), or one or more of other loss calculation methods found elsewhere in the present disclosure. Such loss functions may quantify the difference between one or more generated representations and one or more reference representations (e.g., a benchmark true transformation known to have good function). In some implementations, either or both of module one and module two may receive one or more mesh element features associated with one or more oral care meshes (e.g., a mesh element feature vector may be calculated for one or more mesh elements of an input tooth, gum, hardware article, or appliance assembly). The advantages of receiving mesh element features generally relate to improving the underlying system. For example, such implementations allow the first module to more accurately represent the received 3D representation and allow the second module to generate a more accurate output 3D representation (e.g., a transformation, a representation of a tooth anatomy, or a label on a mesh element).

[0085] Figure 4 Depicted is a technique 400 for training an ML model according to certain aspects of the present disclosure. In general, the technique 4400 uses the above Figure 2 That is, now describe Figure 4Certain additional aspects of . For example, depending on a particular implementation, it may be inappropriate or incorrect to apply the prediction output directly to the patient data to generate the prediction representation. For example, in a segmentation-based implementation, it is appropriate to apply one or more prediction labels to generate the prediction representation 220 because, for example, the underlying representation of the patient data is not modified. In contrast, in other implementations, the prediction output 407 may be one or more vectors describing one or more transformations, and it may be necessary to apply incremental processing steps to apply those transformations to the patient data. For example, when the prediction output 207 is the prediction output vector 407, the grid transformer 418 may be used to apply the one or more prediction vectors to the patient data to generate the prediction representation 420. Similarly, when the reference input 207 is the reference input vector 407, the grid transformer 426 may be used to apply the prediction vector to the patient data to generate the prediction representation 421. Transformers 418 and 426 may use conventional techniques to apply the corresponding vectors to the patient data 204, translate, scale, and rotate the patient data 204 to generate the prediction representation 420 and the reference representation 421, respectively.

[0086] One specific example involves coordinate system generation. Digital dental and digital orthodontic applications may require the definition of coordinate systems in order to operate on 3D mesh models of teeth and gums. Some coordinate systems may be defined relative to the entire dental arch of teeth and are referred to as global coordinate systems. Some coordinate systems may be defined relative to individual teeth and are referred to as local coordinate systems.

[0087] In general, a tooth coordinate system comprises a set of XYZ axes that are used to facilitate mathematical transformations and other operations on a tooth mesh. A tooth coordinate system operates relative to the tooth with the origin located at a carefully selected central position relative to the tooth mesh. The local coordinate system of a tooth is contrasted with a global coordinate system, which has its origin located relative to the center of the entire dental arch. The global coordinate system is used to facilitate mathematical transformations and other operations on the entire dental arch. The correct choice of a tooth coordinate system is critical to the correct function of operations in the design of dental and orthodontic appliances relative to the tooth.

[0088] In a specific implementation related to coordinate system prediction, each patient case in the dataset 204 includes: 1) a set of segmented teeth in a dental arch; and 2) a set of transformations for describing a coordinate system relative to each of these teeth. In the depicted example, the generator 211 can be configured to generate one or more prediction vectors 4407. In addition, the reference real input 208 is in Figure 4407 and 408 are represented as reference truth vector 408. As already mentioned, both vectors 407 and 408 represent transformations to be performed on patient case data 204 in order to generate one or more predicted representations 420 and reference truth representations 421, respectively. Vectors 407 and 408 can be of any size, but it has been observed that vectors with 4×4 dimensions are well suited for technique 400.

[0089] According to the depicted example, the technique 400 transforms the patient case data 204 using mesh transformers 418 and 426, generating prediction representations 420 and 421, respectively. In addition, and consistent with other aspects of the present disclosure, for each prediction transformation (e.g., as defined by the prediction vector 407), the system 100 calculates the LossG1 216 between the generated prediction vector 407 and the corresponding reference true vector 408. The LossG1 216 is fed back to update the weights of the generator 211. In addition, as already described, both the generated vector 407 and the reference true vector 408 are provided to the discriminator 235 (along with the relevant patient data 204, such as a dental mesh). The discriminator 235 attempts to label the vectors 407 and 408, thereby distinguishing true (reference true) from false (generated).

[0090] Depending on the specific implementation, the generator 211 may be replaced by an encoder, which may be viewed as Figure 4 The first half of the U-Net structure depicted in . Specifically, the encoder may include any number of mesh convolution operators 402 and any number of mesh pooling operators 404, but typically does not include an on-mesh pooling operator 406 or a mesh deconvolution operator. That is, the mesh convolution operator 402 generates a high-dimensional feature for each mesh element by collecting neighbor information of the element based on topology (i.e., based on mesh surface connectivity information). The mesh pooling operator 404 at each layer of the encoder simplifies the input mesh to a coarser resolution by reducing the count of mesh elements and aggregating neighboring features of each element. The high-dimensional features aggregated at the last layer are further processed by multiple fully connected layers and are ultimately transformed into a final regression output (e.g., a transformation matrix corresponding to a tooth coordinate system for tooth movement in 3D).

[0091] In some implementations, the techniques disclosed herein can simultaneously predict two orthogonal coordinate axes. From these two orthogonal coordinate axes, a third coordinate axis can be calculated, for example, using the Gram–Schmidt process.

[0092] Depending on the particular implementation, the coordinate system prediction operates on a six-dimensional representation. In addition, while it is possible to use the technique 400 to perform coordinate system prediction on a point cloud (e.g., a 3D point cloud), it is advantageous to perform coordinate system prediction on a 3D geometry such as a 3D mesh. This is because, in general, a 3D mesh (as opposed to a 3D point cloud) is more accurate in its ability to capture the local surface structure of an object. For example, two surfaces may be very close in Euclidean space, but very far apart from each other in a mesh topology (or in geodesic space). Therefore, a 3D mesh is a better choice for representing a surface.

[0093] In addition, for edges and vertices, vertex elements in a 3D mesh may have infinitely (theoretically) connected adjacent vertices, while edge elements in a 3D mesh have a fixed number of adjacent edges (e.g., 4 adjacent edges). The border edge can be given two virtual edges, making the number four. The use of a grid makes grid convolutions in 3D more straightforward. The fixed number of adjacent edges also makes the grid convolution output relatively more stable during training. From a grid topology perspective, the number of edges in a 3D grid is typically greater than the number of vertices (e.g., typically 3 times). In a sense, the grid resolution can be increased by using edges for prediction because there are many more edges than vertices in a typical grid. In addition, it should be understood that neural networks generally benefit from training on a large number of elements. Therefore, by using a 3D grid, the resulting reasoning is improved, and the beneficial effects are passed on to later post-processing steps, resulting in an overall more accurate system.

[0094] Similar to Figure 2 and Figure 3 Once trained, the generator 211 can be used to generate an accurate prediction vector 407 for the patient data 204 received by the receiving module 202. An example technique for generating the prediction vector 407 is Figure 5 4. The technique 500 shown in FIG. 4 shares many of the same features as the techniques 300 and / or 400 described above.

[0095] Figure 6 is an illustration of an example ML architecture 600 that may be used by the system 100 to design and manufacture dental appliances for restoring a patient's dental anatomy in accordance with various aspects of the present disclosure. For example, many of the techniques described herein rely on some form of the architecture 600 as the basis for the ML models described herein.

[0096] In the depicted example, the ML model 600 is a U-Net architecture. The eponymous architecture is configured as one or more grid convolution operators 602a-602n, grid pooling operators 604a-604n, grid unpooling operators 406a-406n, and grid deconvolution operators arranged in an inverted pyramid or "U" configuration. As used herein, it should be understood that the term "operator" is synonymous with and used interchangeably with the terms "node" and "layer", which are also used to describe similar operations in ML parlance.

[0097] In general, the U-Net architecture 600 involves mesh pooling and mesh unpooling operations, which facilitate the process of extracting mesh element neighbor information. Each successive pooling layer helps the model learn neighbor geometry information by reducing the resolution relative to the previous layer. Each successive mesh unpooling layer helps the model expand this summarized neighbor information back to a higher resolution. A series of mesh pooling layers followed by a series of mesh unpooling layers will enable efficient and accurate training of the U-Net, and enable the U-Net to output features for each element containing both local and global geometry information.

[0098] According to a specific implementation, one purpose of the U-Net architecture 600 is to compute a high-dimensional feature vector for the input mesh. For example, according to a specific implementation, the U-Net architecture 600 computes a feature vector for each mesh element (e.g., a 128-element feature vector for each edge, vertex, or face element). The vector exists in a high-dimensional space that is capable of representing the local geometry of the edge within the context of the local teeth, and also represents the global geometry of the two dental arches. The encoder uses the high-dimensional features of the elements within each tooth to make predictions. This combination of local and global information contributes to the accuracy of the predictions. The combination of local and global information enables the U-Net architecture 600 to take into account geometric constraints. For example, during the treatment of clear tray-type aligners, it is undesirable for teeth to collide in 3D space. The combination of local and global information enables the U-Net architecture 600 to generate transformations that reduce or eliminate the incidence of conflicts, and thus produce higher accuracy relative to the prior art. In the event of a collision, the technology of WO2020 / 136587A1 “Methods to automatically remove collisions between digital mesh objects and smoothly move mesh objects between spatial arrangements” can be used to detect and remove collisions between tooth meshes of a dental arch. Consistent with the present disclosure, geometric quantities, such as penetration depth, penetration direction, and counts of overlapping mesh elements (such as vertices), can be calculated, consistent with the detection and removal of tooth mesh collisions. Consistent with the present disclosure, the mesh shape and / or position can be perturbed or changed to reduce or eliminate the incidence of collisions, which in some cases may still exist after operation of the neural network structure of the present disclosure. Generally speaking, information provided to the ML model 600 is first processed by propagating “downward” through operators 602a, 604a, 602b, 604n, etc. until the information reaches the bottom operator (represented here by the mesh convolution operator 602c). This information then propagates "upwards" through operators 606a, 602d, 606n, etc. until it is output by a final grid convolution operator 402n, which may be used by various aspects of the present disclosure, as will be described in more detail below.

[0099] Figure 6 The example U-Net architecture shown is depicted as having a total of nine layers (or nine operators), but it should be understood and appreciated that the U-Net architecture can be configured with any number of convolutional layers, any number of grid pooling layers, and any number of grid unpooling layers to achieve desired results.

[0100] In general, each of the operators 602a-602n, 604a-604n, and 606a-606n can be configured using conventional techniques to modify received inputs related to 3D mesh data (including, for example, mesh size and pose, as embodied by edge lengths, edge curvatures, edge normals, edge midpoints, and other edge data) to produce specific outputs suitable for each of the operators 602a-602n, 604a-604n, and 606a-606n, as will be described in more detail below.

[0101] According to a particular implementation, the mesh convolution operators 602a-602n disclosed in the present disclosure may be configured to be agnostic to the size and pose (e.g., position and / or orientation) of the input 3D mesh according to a particular implementation. The advantage of this agnostic approach is that, as opposed to an input mesh of a fixed size and / or orientation, the mesh cleaning operator can be used to process an original input mesh of arbitrary orientation.

[0102] However, in other implementations, size and pose information is required, such as in the context of regression operations. In implementations where size and pose information is desired, convolution operations may alternatively be configured to be agnostic to size and pose information. For example, convolution filters used as part of the convolution operators 602a-602nML model may be specifically configured to be sensitive to size and pose information, where such a system should not be agnostic to that information. In other implementations, there may be specific aspects of operations that benefit from size and pose information. One specific example is for 3D mesh segmentation, which benefits from a size and pose agnostic mode in some applications (e.g., segmentation of the gingiva, which is used to find a general area of ​​an intraoral scan containing teeth), but does not benefit in other applications (e.g., tooth segmentation, which benefits from information about the left and right sides of the mesh). Therefore, it should be understood that within a particular type of task (e.g., a segmentation task), various aspects of the ML model may be configured to be size and pose agnostic for those operations that benefit, and other aspects of the ML model may be configured to be size and pose sensitive for those operations.

[0103] The mesh pooling operators 604a-604n are configured to resample the input mesh to a lower resolution. As a result, the mesh is continuously refined and resampled to a lower resolution through each successive layer of the mesh pooling operators 604a-604n. This allows downsampling or shrinking of the mesh input. For example, downsampling of information in 3D space may take a 3×3×3 set of information and combine it into a single 1×1×1 representation. For example, in the context of 3D mesh information, the four adjacent edges of a given edge will be combined into a single edge at the next resolution level. The mesh resolution (mesh surface area) after downsampling will be reduced by a factor of 4.

[0104] One of the many advantages of this approach is that the grid pooling operators 604a-604n cause each feature to collect information about its neighbors and summarize that information into a form that is passed to the next layer. Thus, as the grid information moves through the U-Net architecture 600, the output of the lowest level convolution operation 602 (such as 602c in the depicted example) takes the form of a downsampled grid that reveals global information about the original input grid. In other words, the output of the lowest level convolution operation 602 is considered to constitute fully summarized information and it can be used in accordance with various techniques of the present disclosure. For example, the downsampled output of the lowest level grid convolution operation 602 can be used in classification operations (e.g., for 3D verification) and regression operations (e.g., for coordinate system prediction), to name just a few examples.

[0105] In addition, the fully summarized information may undergo further processing by additional operators (e.g., depicted as operators 602n, 604n, and 606n). For example, the fully summarized information output by operator 402c may be processed by mesh unpooling operators 606a and 606n to increase the resolution of the mesh information. As depicted by the example ML model 600, there is a 1:1 relationship between mesh pooling operators 604a and 604b and mesh unpooling operators 606a and 606n. That is, after a sufficient number of mesh unpooling operations (e.g., performed by operators 606a-606n) equal to the number of mesh pooling operations (e.g., performed by operators 604a-604b) have been performed, sufficient information to allow other automatic techniques to identify the element categories (e.g., edges, faces, vertices) in the 3D mesh is surfaced. For example, this allows an automated system to perform mesh segmentation on the output of the convolution operator 602n (eg, perform tooth segmentation, gingival segmentation, facial-lingual segmentation, etc.).

[0106] Figure 7 is an example technique 700 for automatically validating a neural network trained using the techniques described herein.

[0107] Generally speaking, ML models can be trained to verify data sets to be used for digital dentistry or digital orthodontics. In some specific implementations, ML models such as neural networks can be used to verify 2D raster image views of 3D data. An example neural network is a convolutional neural network (CNN). Multiple views of 3D data can be generated. CNN is used to classify each view (e.g., as correct or incorrect), and the verification results of multiple those 2D raster views can be used to verify the correctness of the 3D data. In other specific implementations, the neural network can be a general deep neural network for 3D triangle meshes, such as MeshCNN. MeshCNN is an open source neural network implementation. MeshCNN uses geometric deep learning (or GDL) technology, which involves the first method of performing mesh processing, which operates on edges (or other mesh elements, such as vertices or faces) to implement mesh convolution, mesh pooling, mesh anti-pooling, mesh deconvolution and other 3D-specific deep learning techniques. The open source Minkowski engine includes a neural network with GDL capabilities, which additionally provides GDL operations for sparse convolution. Sparse convolution is a convolution technique that has a different representation value than the mesh convolution operation (i.e., voxels) found in MeshCNN. Voxels are used in sparse convolution operations. Voxels are the 3D geometric equivalent of pixels in 2D images. Sparse convolution techniques exploit the sparsity of data to make 3D volume processing more efficient in many cases. This increase in efficiency is important because some problems can be difficult to solve. In other words, the GDL technique can be applied to every GDL example of the present disclosure, including all 3D verification techniques, mesh segmentation, mesh cleaning, mesh coordinate system prediction, restoration prediction, restoration appliance component placement and generation, and bracket and accessory placement.

[0108] In these implementations, MeshCNN can be used to directly verify the correctness of 3D without having to rely on 2D raster image views of 3D data. In some implementations, the results of one of those verification operations can be fed back into the automated process to improve further iterations of the process of generating those 3D data. In other implementations, the results of one of those verification operations can be reported or displayed to a technician, who can then continue to correct problems with those 3D data. In other implementations, ML models such as neural networks can be used to directly verify 2D data (such as photos of dental or orthodontic appliances). In some implementations, the data to be verified may describe the dental geometry of the patient, possibly including teeth and / or gums. In other implementations, the data to be verified may describe dental or orthodontic appliances or components thereof. In some implementations, the verification invention described in the present disclosure can be integrated into an automated test suite (e.g., unit testing and regression testing for software and algorithms). In other words, although neural networks are the preferred ML method, other ML techniques may be used appropriately.

[0109] For example, MeshCNN can be trained on two (or more) categories of data, such as a 3D mesh corresponding to the RAW category (the "raw" output from segmentation) and a 3D mesh from the TECH category (a mesh modified or corrected by a technician). MeshCNN will become able to distinguish between these two categories and can be used in environments where teeth must be segmented for applications such as dental or orthodontic appliances. In some cases, the RAW category may correspond to a suboptimal state and the TECH category may correspond to an optimal state. In the 3D case, the MeshCNN or encoder may be trained to distinguish these categories. In the 2D case, the CNN may be trained to distinguish these categories. This method may also be applied to other verification operations of the present disclosure. The operational verification engine used in the deployment is designed to detect defects in 3D geometric structures (e.g., dental or orthodontic geometric structures). This operational verification system can be trained on RAW and TECH data categories as an alternative to the CORRECT and UNCORRECT categories that the verification engine may encounter in the field throughout operational use. This involves each of the verification applications described in the present disclosure (e.g., segmentation verification, mesh cleaning verification, coordinate system verification, dental restoration appliance component verification, 3D printed part verification, trim line verification, fixture model verification, and restoration design verification).

[0110] Now go to Figure 7 In the depicted example, in step 702, a system (such as system 100) receives one or more 3D oral care representations, such as a 3D mesh of a patient's dentition (which may include information related to the patient's teeth, gingival tissue, and other aspects of the patient's oral anatomy), as well as other information. The received 3D mesh may vary according to a particular purpose. For example, in a specific implementation regarding mesh segmentation, the received 3D information may be related to a dental arch of a patient's mouth, which may include a 3D representation of teeth and / or gingival tissue, a specific implementation for verification of hardware or appliance component placement. The received 3D mesh may include a 3D representation of specific teeth and associated hardware. In a specific implementation regarding verification of a 3D printed part, the received 3D mesh may include 3D mesh data associated with a part examined in the form of a CT scan or other diagnostic imaging, to name just a few additional examples.

[0111] In step 703, the system 100 may receive a fully trained neural network, such as the fully trained generator 211 described above.

[0112] In step 704, the system 100 may optionally process the received 3D oral care representation to prepare for subsequent steps. For example, in one embodiment, the system 100 may generate or otherwise place components for dental restoration appliances on corresponding teeth in the 3D mesh that must be verified. In another embodiment, the system 100 may place brackets or accessories (or other hardware, such as buttons or hooks attached to teeth, to which resistance bands may be attached and attached to teeth) relative to specific teeth in the 3D oral care representation. In a related embodiment, the system 100 may predict a coordinate system of one or more teeth (e.g., including one or more local coordinate axes of each tooth). In yet other embodiments, the 3D oral care representation may be processed to facilitate identification or marking of mesh elements in a 3D mesh (or 3D point cloud) of a patient's dentition. Examples of where this may be useful include segmentation applications (e.g., tooth segmentation), mesh cleaning, or automatic restoration design generation. In another embodiment, and relative to segmentation, specific teeth may be marked as being correctly segmented or incorrectly segmented. Other types of verification regarding other aspects of the present disclosure are also possible. In other words, depending on the details of a particular implementation, there may be many ways to train a neural network that can validate a 3D oral care representation.

[0113] In step 706, the system 100 may use a 3D modeling tool to generate multiple 2D raster views for each tooth. According to a specific implementation, a 3D modeling tool such as GEOMAGIC may be used, for example, through an automated script. In some examples, other 3D modeling and rendering engines may be used. As used herein, a view may be defined as a specific orientation of a camera within a modeling tool, providing a specific representation of a 3D mesh having a 3-dimensional space represented in the modeling tool. In other words, at step 706, the camera within the modeling tool may be positioned so that each tooth in the 3D mesh is observed from a slightly different angle or vantage point within the modeling tool. The number of views generated may vary according to a specific implementation or a specific use case. For example, according to one specific implementation, fifteen different views of the 3D mesh are generated, although any number of views may be generated for a particular tooth. Therefore, if fifteen views are generated at step 706, a total of 480 2D images for the patient's mouth may be generated at step 706 for a patient having thirty-two teeth, just to give one example.

[0114] Depending on the particular implementation, the 2D raster image generated in step 706 can be used as a comparator when performing other techniques described herein. For example, with respect to tooth segmentation, the segmented tooth mesh (e.g., generated in step 704) can be overlaid on top of the 3D mesh data received in step 702. Then, aspects of the 2D raster image that align with the scan data can be identified. For example, in one implementation, the result of the overlay is a red portion of the geometry corresponding to the segmented tooth mesh and a blue portion of the geometry corresponding to the scan data.

[0115] One advantage of applying a visualization such as that described above is that such visualization allows a human user to identify potential misclassifications of the training data. Additionally, applying what is essentially a binary process to teeth, a binary classification ML model (as described elsewhere in this specification) can be trained to provide accurate predictions. It should be understood that, without loss of generality, each of the 2D and 3D validation examples of the present disclosure can operate under n-class classification, for example, where there are multiple "correct" validation results and multiple "incorrect" validation results.

[0116] In step 708, the system 100 may accumulate or otherwise aggregate 2D views over multiple patient cases. For example, according to one implementation, 60 patient cases may be used. In other words, if 480 2D images are generated for each patient, then in an implementation using 60 patient cases, the training data may include 28,800 different 2D images, as just one example.

[0117] In step 710, the system 100 may train the neural network received in step 703 to validate the accumulated views of one or more cases. For example, when it comes to validating a digitally generated setup for orthodontic alignment treatment, running the fully trained neural network may assign one or more standard scores that specify whether one or more aspects of the received views of the generated setup are correctly formed.

[0118] In step 712, the system 100 outputs both the test results and the resulting neural network. For example, depending on the particular implementation, the output may specify whether the received 3D mesh passes the validation check. If the received 3D mesh does not pass the validation check, the output may also include a correction to the received information describing one or more corrective measures. For example, if the 3D mesh represents a scan of a 3D printed part, the corrective measures may describe how to modify the already manufactured 3D printed part to fit the patient's dental anatomy. Various conditions can be measured or analyzed in this manner. For example, the technology can measure whether the generated settings are correctly formed with respect to measurement criteria for alignment, marginal ridges, buccal and lingual inclinations, occlusal relationships, occlusal contacts, overjet (or overbite), interproximal contacts, and root angulation, to name just a few examples. In other examples, the corrective measures may provide guidance on how to correct the function of the 3D printer (e.g., address a partially blocked nozzle that causes a deformed 3D printed part).

[0119] Although neural networks are used to describe technique 700, one or more steps of technique 700 may also be performed using ML models other than neural networks, such as support vector machines (SVNs), random forests, K-nearest neighbors (KNNs), and other ML models. In order to understand how such other ML models may be used, the data may be divided into two categories of data, "TECH" (class 01) and "RAW" (class 00) data. The TECH class is data obtained by manual intervention of professional technicians. The RAW class is data output from automated tools. TECH class data may generally represent a more correct data set than RAW class data because the TECH class data has been fixed / improved / fine-tuned by expert technicians. The following method is related to a non-neural network method for distinguishing between the TECH class (class 01) and the RAW class (class 00).

[0120] For an effective texture feature-based validation classifier, combining segmentation markers via color with tooth / gum geometry can produce different kinds of artifacts for each class. There are multiple existing texture feature descriptors that can be used as part of texture feature-based validation, including HOG, SURF, SIFT, GLOH, FREAK, and Kadir-Brady. These texture-based validation classifiers can be used by less complex ML models, like some image enhancements can improve the classifiers, such as increasing the contrast between the tooth and gum segmentations so that the feature vectors find more differences around the tooth / gum line when comparing computer- and technician-generated segmentations. Each of the validation applications of the present disclosure may describe a specific implementation involving operations based on texture features.

[0121] For example, verification based on texture features using SIFT classification may include an optional step of converting training images to grayscale, and the steps of finding SIFT keypoints on each image, generating descriptors for these keypoints, selecting only the top N descriptors (where N is the minimum number of descriptors found in all training sample input images), and training a support vector machine (SVM) model on the image descriptors. As an example, other specific implementations may, for example, replace the training of an SVM model on the image descriptors with fitting a k-nearest neighbor (KNN) classifier on the image descriptors.

[0122] That is, while more simplified non-neural network ML models can be used, there are various advantages to using a neural network approach. For example, neural networks can be designed with a sufficiently large number of parameters (i.e., weights) to encode solutions to complex problems, such as understanding 2D raster image views and 3D geometric shapes (i.e., 3D meshes). In addition, texture features may not detect all relevant attributes of an image, such as attributes that indicate defects or errors that the verification process wants to detect.

[0123] Figure 8 An example general technique 800 for validating output generated by an ML model in accordance with various aspects of the present disclosure is shown. The validation ML model may be trained to process the following non-limiting list of 3D representations: 1) mesh element labels for segmentation or mesh cleaning; 2) coordinate system axes of teeth (e.g., as encoded by transforms); 3) dental restoration design; orthodontic settings; 4) custom lingual brackets; 5) bonding pads for brackets (which can be generated for a specific tooth by outlining the perimeter on the tooth, specifying the thickness to form a shell, and then subtracting the teeth via Boolean operations); 6) CTA; 7) location or shape of trim lines (e.g., CTA trim lines); 8) shape or structure or posture of attachments; 9) occlusal ramps or slits; 10) 3D printed appliances (local thickness, rib geometry, flap positioning, etc.); 11) 3D models of the patient's teeth and gums showing trim lines (e.g., fixture models), data or structures related to implant placement; 12) hardware placement; 13) other types of dental restoration designs (e.g., veneers, crowns, or bridges); 14) and other 3D printed parts belonging to oral care procedures or other fields.

[0124] The technique 800 may use the step of receiving a 3D mesh of one or more teeth and additional optional data related to a dental procedure. This information may be provided to one or more anomaly detection networks for verification. In some embodiments, this may include generating one or more 2D raster views of the 3D mesh. Next, the system 100 may use a neural network to analyze each aspect of the 2D and / or 3D representation to present a pass / fail determination regarding those aspects. If a sufficient number of aspects receive a passing accuracy score, the representation is considered to have passed, at which point the system 100 may provide geometry for use in other dental procedures. If a sufficient number of aspects do not receive a passing accuracy score, the system 100 may generate information about why one or more aspects of the representation failed, and in some embodiments, automatically train one or more neural networks based on the results, and then perform the method 800 again with additional training of the neural network to see if a passing score can be achieved. In various embodiments, this 2D verification method may be applied to each of the various verification applications described in the present disclosure.

[0125] The technique 800 can be performed in near real time, allowing dental professionals and other professionals to perform scans and other dental procedures while the patient is in the chair, thereby improving the effectiveness of dental treatment and providing a more pleasant experience for the patient. For example, the verification method can be applied to the patient's intraoral scan data immediately after the intraoral scan is performed. The advantage is that if there is a problem with the scan data, the dentist can be notified, and if a rescan is necessary, the patient can do so (and in fact, without even leaving the chair). Detected mesh errors include holes in the mesh, incompletely scanned teeth, missing teeth, foreign matter obscuring teeth, and / or upper / lower dental arches being incorrectly identified / interchanged. The results of the verification can be displayed to the dentist (or technician) using one or more heat maps, possibly superimposed on the dental model. Problematic areas of the mesh can be highlighted in a patchwork manner using different color coding. The disclosure regarding mesh cleaning describes mesh defects detected during the mesh cleaning verification process. The application of this near real-time method can also benefit from performing inspections to detect these conditions, so that the intraoral scan can be re-performed under different conditions (e.g., more careful techniques employed by the technician or doctor). In this case, the need for subsequent mesh cleanup operations can be reduced or eliminated.

[0126] Specific errors or flaws in the scan are highlighted using color, bounding boxes, arrows, or other graphical elements and displayed to the dentist / technician. For example, if the verification engine determines that a portion of a tooth is missing from the mesh, a bounding box may be drawn onto a visualization of the mesh over the area of ​​the missing or incomplete tooth. A text report on the quality of the scan may be prepared and sent via SMS, email, or other electronic means, or displayed to the dentist / technician in the dentist's office. In some cases, there may be an LCD display located near the scanner that displays the verification report to the dentist. As another example, the verification engine may apply a parting surface to the tooth, resulting in each edge / vertex / face element in the tooth mesh being labeled as A) facial or B) lingual: 1) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the facial direction (e.g., 1.0 mm or 0.5 mm); 2) a facial portion of a tooth where the parting surface is correct; 3) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the lingual direction (e.g., 1.0 mm or 0.5 mm). Depending on the particular implementation, there may be more than one label. For example, some implementations may use both element labels and result labels. The element label describes whether the edge / vertex / face element is on the facial side of the tooth mesh or on the lingual side of the tooth mesh. The result label indicates whether the parting surface near the tooth is 1) too facial, 2) correct, or 3) too lingual, to name just one example.

[0127] According to the technology of the present disclosure, an ML model can be trained on examples of 3D oral care representations, wherein benchmark real data is provided to the ML model and a loss function is used to quantify the difference between the predicted examples and the benchmark real examples. The loss value can then be used to update the validation ML model (e.g., update the weights of a neural network). Such validation techniques can determine whether a trial 3D oral care representation is acceptable or suitable for creating oral care appliances. In some cases, "acceptable" may mean that the trial 3D oral care representation conforms to the distribution of benchmark real examples used in training the ML validation model. In some cases, "acceptable" may mean that the trial 3D oral care representation is correctly shaped or correctly positioned relative to one or more aspects of the dental anatomy.

[0128] In the example of a generated appliance component (e.g., a dental restoration appliance component, such as a mold parting surface), the technology can determine whether the component intersects the correct landmarks or other parts of the tooth anatomy (e.g., incisal edges and cusps for mold parting surfaces). The technology can also determine one or more of the following: 1) whether the CTA trim line intersects the gingiva in a manner that reflects the distribution of the benchmark truth; 2) whether the library component is correctly placed relative to one or more target teeth (e.g., a snap clip placed relative to posterior teeth or a center clip placed relative to incisors), or correctly placed relative to one or more landmarks on the target teeth; 3) whether the hardware elements are placed on the face of the tooth with edges that reflect the distribution of the benchmark truth examples; 4) whether the mesh element labels used for segmentation (or mesh cleaning) operations conform to the label distribution in the benchmark truth examples; and 5) whether the shape and / or structure of the dental restoration tooth design conforms to the distribution of tooth designs in the benchmark truth training examples, to name a few examples. Other verification conditions and / or rules are possible for verification of various 3D oral care representations.

[0129] Fig. 9 An example technique 900 for training an ML model (e.g., classifying a 3D mesh for the purpose of 3D mesh or point cloud verification) is shown. The verification systems and techniques of the present disclosure may assign one or more labels to one or more aspects of a representation to be verified (e.g., correctly formed, incorrectly formed, etc.). The verification systems and techniques of the present disclosure may benefit from the computation of mesh element features. 3D oral care mesh verification may be applied to segmentation, mesh cleaning, coordinate system prediction, dental restoration design, CTA setup verification, CTA trim line verification, fixture model verification, archform verification, orthodontic hardware placement verification, appliance component placement verification, 3D printed part verification, chairside scan verification, and other verification techniques described herein. In the event that a 3D verification check produces a failed output, one or more instructions or feedback data may be transmitted to an algorithm, process, or model that creates a 3D oral care representation, so that further iterations of 3D oral care representation generation may improve the design and hopefully alleviate the condition that caused the verification check to fail. In some implementations, a neural network trained to classify a 3D mesh (or point cloud) for verification may take mesh element features as input (e.g., a mesh element feature vector may be calculated for one or more mesh elements in the mesh or point cloud to be verified). In some cases, a mesh element feature vector may accompany each mesh element as input to the verification neural network. In some cases, the verification neural network may form a reformatted (or sometimes reduced dimension) representation of the input mesh or point cloud. The mesh element features may improve this reformatted (or reduced dimension) representation by providing additional information about the shape and / or structure of the input mesh or point cloud. By using the mesh element features, the precision and accuracy of the resulting verified data is improved.

[0130] Figures 10 to 12 Example techniques 1000-1200 for generating a dental restoration design (e.g., designing a shape that a restored tooth is intended to have after a dental restoration treatment) according to various aspects of the present disclosure, respectively. An autoencoder may be trained to generate a 3D oral care representation of a latent form (e.g., such as a tooth mesh or point cloud). An autoencoder may include a 3D encoder (which converts a 3D oral care representation into a latent form) and / or a 3D decoder (which reconstructs the latent form into a replica of an input 3D oral care representation). Although the present disclosure relates to 3D encoders and 3D decoders, the term 3D should be interpreted in a non-limiting manner to cover multi-dimensional operating modes. For example, the system of the present disclosure may train a multi-dimensional encoder and / or a multi-dimensional decoder.

[0131] Fig.10 A technique for automatically generating geometric shapes for dental restoration designs is shown. An autoencoder (such as Fig.10 and Fig.11) can be trained to reconstruct 3D oral care representations such as teeth (e.g., crowns, roots, or both). An autoencoder (such as a variational autoencoder, a masked autoencoder, or a capsule autoencoder) can be trained to transform a tooth mesh into a latent form. The latent form can be reconstructed as a replica of an input tooth mesh. A reconstruction error calculation can be used to measure the similarity of the reconstructed teeth to the input teeth. If the reconstruction error is below a threshold, the latent form of the tooth (e.g., a latent vector or a latent capsule) can be considered to be a good representation of the input tooth mesh (e.g., although the latent form is a reduced dimension of the input tooth mesh, the latent form contains enough information to reconstruct the input tooth mesh). The 3D encoder portion of the autoencoder can be trained to convert the input tooth mesh into a latent form, and the 3D decoder portion of the autoencoder can be trained to reconstruct the latent form into a reconstructed tooth mesh (e.g., its size, shape, structure, and / or other characteristics can be substantially similar to the input tooth mesh). The encoder and decoder modules can be trained at least in part using the loss values ​​described herein. Specifically, reconstruction loss and KL divergence loss can be used to calculate the loss. The KL divergence loss can make the mathematical space of the loss at least approximately Gaussian, which can produce useful results. One such result is that the potential form (e.g., potential vector) can be modified (e.g., such as by modifying one or more elements of the grid potential vector, or by appending a vector of instructions from a clinician) and the modified potential vector can be reconstructed into a valid tooth grid. If the KL-divergence loss is used in the training of the autoencoder, the change to the potential vector does not necessarily produce a valid reconstruction. In other words, the use of KL divergence in training the autoencoder can simplify the process of customizing the output of the reconstruction autoencoder. The customization can include text instructions from a clinician, such as text instructions that have been first converted into a reduced dimensional form (and / or reformatted) using a text embedding neural network. This customization can include image data, such as describing the color, texture, or other attributes of the teeth (e.g., as depicted in a photograph of a reference tooth). This customization can include real-valued inputs into the autoencoder, or categorical inputs into the autoencoder, either or both of which can be cascaded with the tooth potential vector. The tooth latent vector may be concatenated with a customization vector (e.g., embedded text instructions, or real-valued or categorical oral care parameters), and the concatenation of these vectors may be fed into a decoder for reconstruction. The output of the decoder may be a reconstructed tooth mesh with one or more customized properties relative to the input tooth mesh (e.g., the shape may be more symmetrical, the height may be adjusted to accommodate the clinician's instructions, or the style of the teeth may be customized in other ways). A mesh element feature vector may be calculated for one or more of the mesh elements in the input tooth mesh and fed into Fig.10 or Fig.11to an encoder in order to give the encoder information about the shape and / or structure of the input tooth mesh. Fig.12 A restoration design generation workflow using a U-Net, ResNet, or other neural network to generate a dental restoration design is shown.

[0132] The system of the present disclosure may calculate the reconstruction error as a combination of the L1 loss and the MSE loss, as shown in the following pseudo code: reconstruction_error = 0.5*L1(all_points_target,all_points_predicted) + 0.5*MSE(all_points_target,all_points_predicted). In the above pseudo code, "all_points_target" may include a point cloud corresponding to a reference real dental restoration design (or a reference real example of some other 3D oral care representation), and "all_points_predicted" may include a point cloud corresponding to a generated example of a dental restoration design (or a generated example of some other kind of 3D oral care representation).

[0133] The following describes various aspects of a tooth mesh reconstruction autoencoder (e.g., a variational autoencoder optionally utilizing a normalization flow) that can be used in accordance with the techniques of the present disclosure. A continuous normalization flow (CNF) can include a series of reversible mappings that can transform probability distributions. The CNF can be implemented by a series of blocks in a decoder of the autoencoder. Such blocks can constrain complex probability distributions, thereby enabling the decoder to learn to map simple distributions to more complex distributions and back, thereby bringing technical improvements related to data accuracy, such that the tooth shape distribution after reconstruction is more representative of the tooth shape distribution in the training data set. The reversibility of the CNF provides a technical advantage of improving mathematical efficiency during training, thereby providing technical improvements related to resource usage.

[0134] Mesh correspondences may be calculated to improve the accuracy of the autoencoder's reconstruction of the mesh. Mesh correspondence calculation may find matching points between the surfaces of the input mesh and the template (reference) mesh. The mesh correspondence calculation module may generate point-to-point (or mesh element-to-mesh element) correspondences between the input tooth mesh and the template tooth mesh by mapping each point from the input mesh to at least one point in the template tooth mesh (or by mapping each mesh element from the input mesh to at least one mesh element in the template tooth mesh). Correspondences between mesh elements of the input mesh and mesh elements of a reference or template mesh with known structure may be calculated. In one example, a range of entries in the vector may correspond to the mesio-lingual tooth tip; another range of elements may correspond to the distal-lingual tooth tip; another range of elements may correspond to the mesial surface of the tooth; another range of elements may correspond to the lingual surface of the tooth, and so on. In the case of a tooth mesh reconstruction autoencoder (such as a VAE), in some implementations, the autoencoder may be trained only on a subset of teeth (e.g., only molars or only the upper left first molar). In other implementations, the autoencoder may be trained on a larger subset of teeth in the mouth or on all teeth. In some implementations, an input vector (e.g., a flag vector) may be provided to the autoencoder that may define or otherwise indicate to the autoencoder which type of dental mesh may have been received as input by the autoencoder. The data accuracy improvement of this method is to use mesh correspondences in mesh reconstruction to reduce sampling errors, improve alignment, and improve mesh generation quality. In some implementations, an iterative closest point (ICP) algorithm may be run between the input dental mesh and the template dental mesh during the calculation of mesh correspondences to align the input dental mesh with the template dental mesh. Correspondences may be calculated to establish vertex-to-vertex relationships (between the input dental mesh and the reconstructed dental mesh) for calculating reconstruction errors. In some implementations, during the calculation of mesh correspondences, an inverse difference transform may be applied to at least approximately align the input dental mesh with the template dental mesh. In some implementations, both ICP and inverse mal transforms may be applied. These operations may result in improved data accuracy because they improve the accuracy of the correspondence calculations and the resulting reconstructions.

[0135] Fig.11 and Fig.12 Describes the use of GANs (such as Figure 2The invention relates to a method for generating a tooth restoration design using a GAN (shown in Figure 1 ) which is trained to generate tooth restoration designs. The tooth restoration design requires prescription information to specify the expected properties of the final tooth geometry (e.g., the width and height of the restored tooth). These techniques can be used to incorporate prescription information from medical professionals (e.g., in the form of text data) into the training of the ML model disclosed herein. Therefore, the text can be used to influence the characteristics of the model output. This can be done by generating a text embedding space, and samples from the space will be represented as vectors with the same dimensions as the text embedding space. This space will define a low-dimensional space capable of representing all the significant information in the text corpus and will be easier to use than a larger space that explicitly represents each single possible word. Once the low-dimensional space is trained to represent all the key emotions behind the sentences in the text corpus, the low-dimensional space can be used to help train a generative tooth model, the output of which can be manipulated by the text input so that the output teeth exhibit some characteristics conveyed to the model as text input.

[0136] According to a specific implementation, the training can be achieved by training the model using a dataset of paired text and tooth mesh samples. The text corresponding to the tooth mesh can be converted into an embedding vector using a previously trained text embedding. In some examples, image information (e.g., about the color of the teeth in the image) can be incorporated in a manner similar to the text information, where an encoder is used to reduce the image to a vector (i.e., similar to the way a certain text can be reduced to a vector). Once in the embedding space, the text and / or image vectors can be cascaded with the unmodified tooth mesh latent space vector. The cascaded vector is located at the center of the variational self-encoder structure. Now, the input consisting of text (and possibly image) and tooth mesh data can be provided to the decoder, thereby producing a tooth mesh as an output, wherein the output tooth mesh reflects the target geometry from the prescription (within tolerance). During the training process, back propagation can be used to update the network to minimize the difference between the reference true (modified) tooth mesh and the tooth mesh generated according to the decoder output. According to a specific implementation (e.g., as shown in Figure 27), the text encoder can be used to encode the prescription into a text embedding vector that contains all the necessary information about the patient's prescription. This vector is then concatenated with the grid encoder output, which similarly contains the necessary information about the structure and geometry of the input grid. The resulting vector is then passed to the decoder to reconstruct the desired output grid. Note that the grid encoder-decoder architecture can also be replaced with a U-Net architecture, in which case the text embedding vector can be concatenated with the coarsest level features in the U-Net contraction unit (U-Net encoder) before passing them together to the U-Net expansion unit (U-Net decoder).

[0137] Other methods are also possible. For example, Fig.12 As shown in , the text embedding features can be cascaded with the features of the input grid at the input of the grid processing network. In this specific implementation, the ML model can be U-Net, ResNet and other similar models.

[0138] In other examples, examples for restoring tooth generation (e.g., GAN, U-Net, or VAE methods) may be adapted to generate roots of teeth. In one example, the dental arch is scanned and the resulting mesh is segmented to obtain a mesh for each tooth. Each tooth mesh contains mesh elements from the tooth that are visible from outside the full dental arch mesh, but the tooth mesh does not contain mesh elements of any internal tooth structure that has not been scanned, such as the root of the tooth. The root is located inside the alveolar bone and is used to fix the tooth to the alveolar bone. Some processes in digital dentistry and digital orthodontics can be assisted by generating a root structure that is attached to the tooth along the open gingival contour of the tooth. The GAN, VAE, or U-Net method from the restoration design generation invention may be adapted to generate a root geometry that extends along the gingival direction (relative to the tooth) to complete or close the shape of the tooth. VAE is a variational autoencoder that includes an encoder, a latent space representation (i.e., can be represented as a vector), and a decoder. In the present disclosure, this latent space can be trained to learn an efficient encoding for 3D mesh geometry, which in some cases can incorporate data from natural language processing (NLP) input. In some cases, a conditional VAE can be used.

[0139] Various aspects of the present disclosure may be used for different purposes in one or more digital dental fields, including segmentation, coordinate systems, mesh cleaning, setting of clear tray-type aligners, dental prosthesis, brackets and accessories, 3D printed parts, prosthetic designs, and fixture models. These fields may involve the generation of one or more (2D or 3D) representations and the verification of one or more (2D or 3D) representations. One or more of these fields may be combined, for example, certain techniques may combine the following concepts: 1) segmentation, 2) calculation of the geometry of dental prosthesis, and 3) mesh verification. For example, the results of face-tongue segmentation may be used by an algorithm for generating a mold parting surface, with the goal of improving the resulting mold parting surface (i.e., relative to the mold parting surface that would be generated without the benefit of the previous face-tongue segmentation). The resulting mold parting surface may then be checked by a verification module (i.e., using 2D or 3D processing). If the verification module determines that the generated mold parting surface is poor, the algorithm that generated the mold parting surface can be rerun, potentially using actionable feedback from the verification engine (e.g., tips on how to adjust the mold parting surface on a tooth-by-tooth basis, whether the parting surface should move in the facial or lingual direction near each tooth). If the verification module determines that the generated mold parting surface is acceptable, the mold parting surface is output.

[0140] Although this specification sets forth many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular implementation. Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations, even when initially claimed, in some cases one or more features from a claimed combination may be deleted from that combination, and a claimed combination may refer to a subcombination or a variation of a subcombination.

[0141] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described components and systems may generally be integrated together in a single system or distributed in multiple systems.

Claims

1. A computer-implemented method for training one or more machine learning models to generate a digital representation of a dental restoration tooth geometry, the method comprising: receiving, by one or more computer processors, a first digital representation defining an unrestored state of a patient's tooth; receiving, by the one or more computer processors, data related to an outcome of the dental restoration; using, by the one or more computer processors, a machine learning model that has been initially trained to generate a digital representation of a restoration state defining the first digital representation based on the data relating to an outcome of the dental restoration; and automatically training, by the one or more computer processors, the machine learning model based on the use, wherein the training of the machine learning model is modified by performing operations comprising: generating one or more digital representations defining a repair status of said first digital representation; determining, by the one or more computer processors, one or more differences between the one or more predicted representations of the repair state and the one or more reference representations of the repair state; as well as modifying the machine learning model based on the determined differences; The dental restoration tooth geometry is used to manufacture dental restoration appliances.

2. The computer-implemented method of claim 1, wherein the machine learning model is trained to generate a restoration design for a specific tooth.

3. The computer-implemented method of claim 1, wherein a template tooth is received by the one or more processors and at least one correspondence between at least one aspect of the template tooth and at least one aspect of the first digital representation is calculated.

4. The computer-implemented method of claim 1, further comprising applying data augmentation to the first digital representation of the patient's teeth.

5. The computer-implemented method of claim 1 , wherein the one or more machine learning models comprise one or more neural networks.

6. The computer-implemented method of claim 5, wherein one of the one or more neural networks is a generator and one of the one or more neural networks is a discriminator for at least partially training the generator.

7. The computer-implemented method of claim 5, wherein a neural network of the one or more neural networks is selected from the group consisting of an encoder, an autoencoder, a variational autoencoder, a capsule autoencoder, a conditional variational autoencoder, a text encoder, an image encoder, and a decoder.

8. The computer-implemented method of claim 1, wherein the data relating to the outcome of the dental restoration comprises instructions describing the outcome of the dental restoration.

9. The computer-implemented method of claim 1, wherein the first digital representation comprises at least one mesh element selected from the group consisting of one or more edges, one or more faces, one or more vertices, or one or more voxels.

10. The computer-implemented method of claim 9, wherein at least one mesh element feature is calculated for at least one of the mesh elements.

11. The computer-implemented method of claim 1 , wherein the machine learning model is initially trained using historical digital representations of restored dental geometries.

12. A computer-implemented method according to claim 1, wherein the machine learning model is trained iteratively, and when the accuracy of the machine learning model reaches a predetermined threshold, the machine learning model is considered to be fully trained.

13. The computer-implemented method of claim 7, wherein the autoencoder comprises at least an encoder and a decoder, and the autoencoder has been trained to reconstruct a 3D oral care representation of teeth.

14. The computer-implemented method of claim 13, wherein the encoder is trained to convert the first digital representation into a latent representation.

15. The computer-implemented method of claim 13, wherein the decoder is trained to reconstruct the latent form as a facsimile of the first digital representation.

16. The computer-implemented method of claim 13, wherein a reconstruction error is calculated to quantify a difference between the first digital representation and a facsimile of the first digital representation.

17. The computer-implemented method of claim 13, wherein the autoencoder is trained at least in part by computing at least one of a KL divergence loss or a reconstruction loss.

18. The computer-implemented method of claim 14, further comprising combining the latent representation with at least one additional latent representation.

19. A computer-implemented method for generating a digital representation of a dental restoration geometry using a trained neural network, the method comprising: receiving, by one or more computer processors, first patient data defining an unrestored state of a patient's teeth; receiving, by the one or more computer processors, second patient data related to an outcome of the dental restoration; using, by the one or more computer processors, the fully trained machine learning model to generate, based on the second patient, a digital representation defining a repair state of the first patient data using steps comprising: receiving, by one or more computer processors, a first digital representation defining an unrestored state of a patient's tooth; receiving, by the one or more computer processors, data related to an outcome of the dental restoration; using, by the one or more computer processors, a machine learning model that has been initially trained to generate a digital representation of a restoration state defining the first digital representation based on the data relating to an outcome of the dental restoration; and automatically training, by the one or more computer processors, the machine learning model based on the use, wherein the training of the machine learning model is modified by performing operations comprising: generating one or more digital representations defining a repair status of said first digital representation; determining, by the one or more computer processors, one or more differences between one or more predicted representations of the repair state and one or more reference representations of the repair state; as well as modifying the machine learning model based on the determined differences; as well as A 3-dimensional (3D) output defining a repair state of the first patient data is generated.

20. The computer-implemented method of claim 19, wherein a template tooth is received by the one or more processors and at least one correspondence between at least one aspect of the template tooth and at least one aspect of the first digital representation is calculated.

Images