Coordinate system prediction in digital dentistry and digital orthodontics and verification of prediction thereof

Abstract

Systems and techniques are disclosed for training one or more encoders to automatically generate a coordinate system for use in digital dentistry, including predicting one or more predicted transforms with respect to one or more coordinate axes, determining a loss value specifying a difference between the one or more predicted transforms and one or more corresponding reference transforms, and modifying at least one aspect of the encoder structure based on the loss.

Description

Technical Field

[0001] The present disclosure relates to various improved machine learning techniques for use in digital oral care, including the disciplines of digital dentistry and digital orthodontics. Background Art

[0002] Dentists often use dental appliances to reshape or restore a patient's dental anatomy, or to move teeth using orthodontic appliances. These appliances are typically constructed from a model of the patient's dental anatomy that is modified to the desired end state. The model can be a physical model or a digital model. Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. Problems with past approaches include loss of mapping accuracy and inefficiencies in the data processing that produces the 2D to 3D conversion.

[0003] For example, according to existing implementations, the projection operations performed by existing systems may result in 3D mesh elements receiving conflicting labels due to two or more projection operations. This may require the execution of additional machine learning models to disambiguate those conflicting labels, thereby increasing the complexity and error of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An example processing unit operating in accordance with the techniques of this disclosure is shown.

[0005] Figure 2 Example general techniques for training a generator or other neural network according to various aspects of the present disclosure are shown.

[0006] Figure 3 Example general techniques for using a trained generator or other neural network in accordance with various aspects of the present disclosure are shown.

[0007] Figure 4 Another example general technique for training a generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0008] Figure 5 Another example general technique for using a trained generator or other neural network in accordance with various aspects of the present disclosure is shown.

[0009] Figure 6 An example machine learning architecture according to various aspects of the present disclosure is shown.

[0010] Figure 7Example techniques for data augmentation are shown.

[0011] Figure 8 Example techniques for performing 2D verification on dental data are shown.

[0012] Fig. 9 An example technique for training an encoder is shown.

[0013] Fig.10 An example technique for teeth segmentation is shown.

[0014] Fig.11 Example general techniques for validating output generated by a machine learning model in accordance with various aspects of the present disclosure are shown.

[0015] Fig.12 Example techniques for training machine learning models are shown.

[0016] Fig.13 Depicted are two types of 2D raster image views that can be used to train a neural network to perform validation on a predicted coordinate system.

[0017] Figures 14 to 16 Shown with Fig.13 The same two types of data, both of which are separate, and both coordinate systems are superimposed on the same tooth mesh. DETAILED DESCRIPTION

[0018] The present disclosure describes various automation techniques that can be implemented throughout the process of manufacturing dental and orthodontic appliances. Therefore, the present disclosure contemplates improvements in the field of digital oral care, including digital dentistry and digital orthodontics. The automated geometry generation techniques of the present invention are intended to simplify the manufacturing process that would otherwise be extremely time-consuming. Another advantage of these automated geometry generation techniques is that the accuracy of dental appliances is improved. In some cases, the algorithm can produce geometries with higher quality and higher accuracy than those produced by human technicians. Although in some cases, human technicians can modify or "fine-tune" the design output by the automated tool, the automated tool improves the quality of the resulting appliance by providing a common construction baseline for multiple technicians. In addition, by studying the output of the automated tools in the present disclosure (e.g., both the tools for geometry generation and the tools for geometry verification), untrained or new human technicians can understand the appropriate techniques for creating dental and orthodontic appliances (generally used as oral care appliances in this article). The transfer of knowledge to other technicians and the standardization of techniques are important benefits of the disclosed technology. Due to all of the above reasons, another advantage is that more accurate geometry and knowledge transfer can improve the restoration results associated with the use of manufactured dental or orthodontic appliances.

[0019] Historically, systems have performed operations on two-dimensional (2D) images of tooth tissue (or dental or orthodontic appliances) and then projected the resulting data from those 2D images back onto corresponding three-dimensional (3D) mesh geometry (e.g., to label portions of the mesh). Some of these systems are configured to operate on photographs, while others are configured to operate on height maps. The technology disclosed herein takes a more direct approach in that mesh elements are labeled directly without the need for intermediate 2D images and the projection of information from those 2D images onto a 3D mesh. Thus, for example, 3D mesh elements can be directly labeled for segmentation and mesh cleaning, which is not possible when using existing systems that rely on 2D mapping techniques. This direct element labeling approach improves the accuracy of the underlying machine learning (ML) model and provides greater efficiency with respect to the use of computational resources because the computational overhead of generating images and mapping them back onto 3D geometry can be avoided.

[0020] As used herein, a 3-dimensional ("3D") mesh (or 3D geometry) includes data corresponding to the edges, vertices, and faces of the 3D mesh. These edges, vertices, and faces are also referred to as digital representations of one or more aspects of a 3D mesh. In some examples, the aspects of a 3D mesh may refer to the shape or geometric features of the mesh. In some cases, the aspects of one mesh may be compared to the aspects of another mesh, such as during a validation operation. Although these three types of data are interrelated, they are different. Vertices are points in 3D space that define the boundaries of a mesh. Therefore, if there is no additional information about how the points are connected to each other, these points can be regarded as point clouds. However, in the context of 3D meshes, edges provide structure to point clouds. An edge includes two points and may also be referred to as a line segment. A face includes both edges and vertices. For example, in the case of a triangular mesh, a face includes three vertices, where these vertices are interconnected to form three continuous edges. Although 3D meshes are typically formed using triangles, other specific implementations may use quadrilaterals, pentagons, or some other n-sided polygons to define 3D meshes. Some meshes may contain degenerate elements, such as non-manifold geometries. Non-manifold geometry is a digital geometry that cannot exist in the real world. For example, one definition of a non-manifold is a 3D shape that cannot be unfolded into a 2D surface, so that the unfolded shape has all surface normal vectors pointing in the same direction. An example of a possible non-manifold geometry is that a face or edge is squeezed but not moved, which causes two identical edges to form on top of each other. Typically, such non-manifold geometry is deleted before processing can be performed. Other mesh preprocessing operations are also possible. The 3D data of each example in the present disclosure can be presented to the ML model as a 3D mesh and / or output from the ML model as a 3D mesh. Other 3D data representations include voxels, finite elements, finite differences, discrete elements, and other 3D geometric representations of dental data and / or appliances. Other specific implementations can use non-discrete methods to describe 3D geometries, thereby using mathematical formulas to regenerate the geometry during processing. Such formulas may contain expressions including polynomials, cosines, and / or other trigonometric or algebraic terms. An advantage of non-discrete formats can be to compress data and save storage space. Digital 3D data may require different coordinate systems such as XYZ (Euclidean), cylindrical, radial, and custom coordinate systems.

[0021] That is, a 3D mesh is a data structure that can describe the structure, geometry, and / or shape of an object related to oral care, including but not limited to teeth, hardware elements, or gum tissue of a patient. The geometry of a 3D mesh can define aspects of the physical size, scale, and / or symmetry of the mesh. The structure of a 3D mesh can define the count, distribution, and / or connectivity of mesh elements. A 3D mesh can include one or more mesh elements, such as one or more vertices, edges, faces, and combinations thereof. In some implementations, mesh elements can include voxels, such as in the context of sparse mesh processing operations. Various spatial and structural features can be calculated for these mesh elements and provided to the predictive models of the present disclosure, with the advantage of improving the accuracy of those predictive models. For example, in some implementations, mesh element features can quantify certain aspects of a 3D mesh that is close to or associated with one or more mesh elements, as described elsewhere in the present disclosure.

[0022] Depending on the specific implementation, it may be beneficial to pre-process the information to generate one or more mesh feature elements. That is, each 3D mesh may undergo pre-processing before being input into a prediction architecture (e.g., including at least one of an encoder, a decoder, an autoencoder, a multi-layer perceptron (MLP), a transformer, a pyramid encoder-decoder, a U-Net, or a graph CNN). Such pre-processing may include converting the mesh into a list of mesh elements such as vertices, edges, faces, or voxels in the case of sparse processing. For one or more selected mesh element types (e.g., vertices), a feature vector may be generated. In some examples, a feature vector is generated for each vertex of the mesh. Each feature vector may contain a combination of spatial and / or structural features, as specified by the following table:

[0023] Consistent with Table 1, a voxel may also have features that are computed as an aggregation of other mesh elements (e.g., vertices, edges, and faces) that either intersect the voxel or, in some implementations, are primarily or completely contained within the voxel. Rotating a mesh may not change structural features, but may change spatial features. Also, as described elsewhere, the term "mesh" should be considered in a non-limiting sense to include 3D meshes, 3D point clouds, and 3D voxelized geometries. In some cases, a 3D point cloud may be derived from the vertices of a 3D triangle mesh.

[0024] Techniques that can operate on the feature vectors of the above features include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental restoration appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification. Such feature vectors can be presented to the input of a prediction model. In some embodiments, such feature vectors can be presented to one or more internal layers of a neural network that is part of one or more of those prediction models.

[0025] But a 3D mesh is only one type of 3D representation that can be used. Therefore, it should be understood that, without loss of generality, there are various types of 3D representations contemplated herein. For example, a 3D representation may include, be one or more of the following, or be a part thereof: a 3D polygonal mesh, a 3D point cloud, a 3D voxelized representation (e.g., a collection of voxels), or a 3D representation described by a mathematical equation. Although the term "mesh" is frequently used throughout this disclosure, in some specific implementations, the term should be understood to be interchangeable with other types of 3D representations. A 3D representation may describe an element of the 3D geometry and / or 3D structure of an object. And a patient's dentition may include one or more 3D representations of the patient's teeth, gums, and / or other oral anatomical structures. Depending on the specific implementation, a 3D scanner (such as an intraoral scanner, a computed tomography (CT) scanner, an ultrasound scanner, a magnetic resonance imaging (MRI) machine, or a mobile device capable of performing stereophotogrammetry) may be used to generate an initial 3D representation.

[0026] In light of the above, the techniques described herein relate to operations performed on 3D representations to perform tasks associated with geometry generation and / or verification. For example, the present disclosure relates to improved automated techniques for segmentation generation and verification, coordinate system prediction and verification, clear tray appliance setup verification, dental prosthetic appliance verification, bracket and attachment (or other hardware) placement and verification, 3D printed component verification, prosthetic design generation and verification, fixture model verification, and clear tray appliance trim line verification, to name a few examples. The present disclosure also relates to improved automated techniques for verifying many of those examples.

[0027] In general, the use of side information ensures that the ML model is insensitive to different input orders of 3D elements. A notable exception is the specific implementation of coordinate prediction, which operates on 3D point clouds instead of 3D meshes. These and other differences are described in more detail below.

[0028] Certain examples in this disclosure mention the use of MeshCNN or encoders to process 3D mesh geometry (e.g., encoder structures for 3D verification and bracket / accessory placement, and MeshCNN for marking mesh elements in segmentation and mesh cleaning). Without limitation, each of these examples may employ other kinds of neural networks to process 3D mesh geometry in addition to or in place of the specified neural network. The following neural networks may be interchangeable in various implementations of the 3D mesh geometry examples of this disclosure: ResNet, U-Net, DenseNet, MeshCNN, Graph-CNN, PointNet, Multilayer Perceptron (MLP), PointNet++, PointCNN, and PointGCN. In other cases, an encoder structure may be used.

[0029] In some cases, the system of the present disclosure may be deployed in a clinical environment (such as a dental or orthodontic clinic) for use by clinicians (e.g., doctors, dentists, orthodontists, nurses, hygienists, oral care technicians). Such systems deployed in a clinical environment may enable clinicians to process oral care data (such as dental scans) in a clinical environment or in some cases in a "chairside" environment (e.g., in near "real time" when the patient is located in the clinical environment). A non-limiting list of examples of techniques may include: segmentation, mesh cleaning, coordinate system prediction, CTA trim line generation, restoration design generation, appliance component generation or placement or assembly, generation of other oral care grids, verification of oral care grids, setting prediction, hardware removal from tooth grids, hardware placement on teeth, estimation of missing values, clustering of oral care data, oral care grid classification, setting comparison, metric calculation, or metric visualization. In some cases, the execution of these techniques may enable patient data to be processed, analyzed, and used by clinicians in appliance creation before the patient leaves the clinical environment (this may facilitate treatment planning because feedback may be received from the patient during the treatment planning process).

[0030] The system of the present disclosure can use representation learning to train ML models. The advantages of representation learning include the fact that, in contrast to receiving inputs of variable size or structure, the generative network (e.g., a neural network that predicts a transformation) is guaranteed to receive inputs of known size and / or standard format. Representation learning can produce performance superior to other methods because the noise in the input data can be reduced (e.g., because the representation generation model extracts important aspects of the input mesh or point cloud through loss calculation or the network architecture selected for this purpose). Such loss calculation methods include KL divergence loss, reconstruction loss, or other losses disclosed herein. Representation learning can reduce the size of the data set required to train the model because the representation model learns the representation, while the generative network can focus on learning the generation task. Since meaningful features are available to the generative network, the result can be an improved model generalization. In some cases, transfer learning can first train the representation generation model. The representation generation model can then be used (in whole or in part) to pre-train subsequent models, such as the generative model (e.g., generating transformation predictions).

[0031] In some implementations, the techniques of the present disclosure may be trained to predict one or more local orthogonal coordinate axes of a tooth (e.g., predicting one or more of the X, Y, and Z orthogonal axes of a tooth). In other implementations, the techniques of the present disclosure may be trained to predict one or more dental arch morphology coordinate axes. Position may include a tuple [l, d, e] relative to a reference dental arch morphology spline S, which approximates the shape of the dental arch. Rotation may include a tuple [a, b, g], which represents α, β, and γ rotations. α describes a rotation around the l axis. β describes a rotation around the d axis. γ describes a rotation around the e axis. A full tuple describing position and rotation may include [l, d, e, a, b, g]. p is a point with an arc length of l along S. d is the distance between the tooth origin t and the reference dental arch morphology spline S. The tooth origin t is obtained by translating a distance "d" upward along the d axis and then translating a distance "e" along the e axis. The e-axis is perpendicular to the d-axis and the l-axis and can be defined as the direction out of or into the page. e represents the height. l represents the length of the cross-arch morphology spline. d represents the distance from the arch morphology spline. This arch morphology coordinate system is described by the same applicant's U.S. published patent application US2021 / 0259808A1, the entire contents of which are incorporated herein by reference.

[0032] In some cases, federated learning may be used to train the techniques of the present disclosure. Federated learning may enable multiple remote clinicians to iteratively improve machine learning models (e.g., validation of 3D oral care representations, mesh segmentation, mesh cleaning, other techniques involving labeling mesh elements, coordinate system prediction, non-organic object placement on teeth, appliance component generation, dental restoration design generation, techniques for placing 3D oral care representations, setting predictions, generation or modification of 3D oral care representations using autoencoders, generation or modification of 3D oral care representations using transformers, generation or modification of 3D oral care representations using diffusion models, 3D oral care representation classification, imputation of missing values) while protecting data privacy (e.g., clinical data may not need to be sent "over the network" to a third party). Data privacy is particularly important for clinical data that is protected by applicable law. Clinicians may receive a copy of the machine learning model, use a local machine learning program to further train the ML model using locally available data from a local clinic, and then send the updated ML model back to a central hub or a third party. A central hub or third party can integrate updated ML models from multiple clinicians into a single updated ML model that benefits from learning from patient data recently collected at various clinical sites. In this way, new ML models can be trained that benefit from additional and updated patient data (potentially from multiple clinical sites) without that patient data actually being sent to a third party. In some cases, training on devices within a local clinic can be performed when the equipment is idle or otherwise during off-hours (e.g., when a patient is not being treated in the clinic). Equipment used in a clinical environment to collect data and / or train ML models for the techniques described herein may include intraoral scanners, CT scanners, X-ray machines, laptops, servers, desktop computers, or handheld devices (such as smartphones with image collection capabilities).

[0033] In addition to federated learning techniques, in some implementations, contrastive learning can be used to at least partially train the ML models described herein. In some cases, contrastive learning can enhance samples in a training dataset to emphasize differences between samples of different categories and / or increase similarities between samples of the same category.

[0034] Figure 1An example processing unit 102 operating according to the technology of the present disclosure is shown. The processing unit 102 provides a hardware environment for training one or more neural networks described throughout the specification. In general, and as will be described in more detail elsewhere, training the one or more neural networks is accomplished by providing one or more training data sets. Therefore, the quality and composition of the training data set for the neural network can have a significant impact on any neural network trained thereby. Dataset filtering and outlier removal can be advantageously applied to the training of neural networks of various technologies of the present disclosure (e.g., mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setting verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances (e.g., verification of the shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixed body model verification and CTA trim line verification, verification using autoencoders, and setting prediction).

[0035] In the depicted example, the processing unit includes a processing circuit system that may include one or more processors 104 and a memory 106, which in some examples provide a computer platform for executing an operating system 116, which may be, for example, a real-time multitasking operating system or other types of operating systems. In turn, the operating system 116 provides a multitasking operating environment for executing one or more software components (such as applications or other training routines). The processor 104 is coupled to one or more I / O interfaces 114, which provide an I / O interface for communicating with devices such as keyboards, controllers, display devices, image capture devices, other computing systems, and the like. In addition, the one or more I / O interfaces 114 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network. Additionally, the processor 104 may be coupled to an electronic display 108.

[0036] In some examples, processor 104 and memory 106 may be separate, discrete components. In other examples, memory 106 may be on-chip memory co-located with processor 104 within a single integrated circuit. Multiple instances of processing circuit systems (e.g., multiple processors 104 and / or memory 106) may exist within processing unit 102 to facilitate parallel execution of applications and / or processes (including applications and / or processes related to machine learning). Multiple instances may be of the same type, such as a multi-processor system or a multi-core processor. Multiple instances may be of different types, such as a multi-core processor with associated multiple graphics processor units (GPUs). In some examples, processor 104 may be implemented as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or equivalent discrete or integrated logic circuit systems, or a combination of any of the foregoing devices or circuit systems.

[0037] Figure 1 The architecture of the processing unit 102 shown in the figure is shown for example purposes only. The processing unit 102 should not be limited to the example architecture shown. In other examples, the processing unit 102 can be configured in various ways. The processing unit 102 can be implemented as any suitable computing system (e.g., at least one server computer, workstation, mainframe, appliance, cloud computing system, and / or other computing system) capable of performing the operations and / or functions described according to at least one aspect of the present disclosure. As an example, the processing unit 102 may represent a cloud computing system, a server computer, a desktop computer, a server farm, and / or a server cluster (or a portion thereof). In other examples, the processing unit 102 may represent or be implemented by at least one virtualized computing instance (e.g., a virtual machine or container) of a data center, a cloud computing system, a server farm, and / or a server cluster. In some examples, the processing unit 102 includes at least one computing device, each computing device having a memory 106 and at least one processor 104.

[0038] Storage unit 134 may be configured to store information within processing unit 102 during operation (e.g., 3D geometry, transformations to be performed on the 3D geometry, etc.). Storage unit 134 may include a computer-readable storage medium or a computer-readable storage device. In some examples, storage unit 134 includes at least short-term memory or long-term memory. Storage unit 134 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetic disk, an optical disk, a flash memory, a magnetic disk, an optical disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable memory (EEPROM).

[0039] In some examples, storage unit 134 is used to store program instructions for execution by processor 104. Storage unit 134 can be used by software or applications running on processing unit 102 to store information during program execution and to store results of program execution. For example, storage unit 134 can store any number of neural networks 110a-110n, including those described herein. According to some implementations, neural networks 110a-110n can be neural networks trained according to the techniques disclosed herein. In other implementations, one or more of neural networks 110a-110n can be untrained or partially trained.

[0040] As will be described in more detail elsewhere, ML models (e.g., one or more neural networks) can be trained in a supervised and unsupervised manner. Supervised models that can be trained to make recommendations described herein include: regression models (such as linear regression), decision trees, random forests, boosting, Gaussian processes, k-nearest neighbors (KNN), logistic regression, naive Bayes, gradient boosting algorithms (e.g., GBM, XGBoost, LightGBM, and CatBoost), support vector machines (SVM), or fully connected neural network models that have been trained for classification. In some cases, a multilayer perceptron (MLP) can be used to predict missing program parameters based on known program parameters.

[0041] Unsupervised models that can be trained to make the recommendations described herein include clustering techniques such as K-means clustering, density-based spatial clustering of applications with noise (DBSCAN), Gaussian mixture models, balanced iterative reduction and clustering based on hierarchy (BIRCH), affinity propagation clustering, mean shift clustering, ordering points to identify cluster structures (OPTICS), agglomerative hierarchical clustering, and spectral clustering.

[0042] Whether the training is supervised or unsupervised, there are a variety of optimization methods that can be used to train the neural network of the present disclosure (e.g., update the neural network weights), including gradient descent (which uses first-order derivatives to determine the training gradient and is commonly used in the training of neural networks), Newton's method (which can use second-order derivatives in the loss calculation to find a better training direction than gradient descent, but may require the calculation of the Hessian matrix), and conjugate gradient method (which may converge faster than gradient descent, but does not require the Hessian matrix calculation that may be required by Newton's method). In some specific implementations, in addition to or instead of the aforementioned methods, other methods may be used to update the weights. These other methods include: Levenberg-Marquardt method and simulated annealing. The back propagation algorithm is used to transmit the results of the loss calculation back to the network so that the network weights can be adjusted to perform learning.

[0043] Neural networks facilitate the functionality of many applications of the present disclosure, including, but not limited to: mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of custom oral care appliance components, placement of custom oral care appliance components, verification of custom oral care appliances (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders. The neural networks of the present disclosure may embody portions or all of a variety of different neural network models. Examples include U-Net architecture, multi-layer perceptron (MLP), transformer, pyramid architecture, recurrent neural network (RNN), autoencoder, variational autoencoder, regularized autoencoder, conditional autoencoder, capsule network, capsule autoencoder, stacked capsule autoencoder, denoising autoencoder, sparse autoencoder, conditional autoencoder, long / short term memory (LSTM), gated recurrent unit (GRU), deep belief network (DBN), deep convolutional network (DCN), deep convolutional inverse graphics network (DCIGN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), neural Turing machine (NTM) and generative adversarial network (GAN). In some implementations, an encoder structure or a decoder structure can be used. Each of these models has its own specific advantages. A specific model may be particularly suitable for one or another model.

[0044] In some implementations, the neural networks of the present disclosure may be adapted to operate on 3D point cloud data (alternatively 3D meshes or 3D voxelized representations). Many neural network implementations may be applicable to the processing of 3D representations and may be applied to training predictive and / or generative models for oral care applications, including: PointNet, PointNet++, SO-Net, Spherical Convolution, Monte Carlo Convolution and Dynamic Graph Networks, PointCNN, ResNet, MeshNet, DGCNN, VoxNet, 3D-ShapeNets, Kd-Net, Point GCN, Grid-GCN, KCNet, PD-Flow, PU-Flow, MeshCNN, and DSG-Net. Oral care applications include, but are not limited to: mesh reconstruction autoencoder, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of custom oral care appliance components, placement of custom oral care appliance components, verification of custom oral care appliance components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification and CTA trim line verification, verification using autoencoders, setup prediction, and generation of dental prosthetic appliances.

[0045] In some specific implementations, some techniques of the present disclosure may use autoencoders. Possible autoencoders include, but are not limited to: AtlasNet, FoldingNet, and 3D-PointCapsNet. Some autoencoders may be implemented at least in part based on PointNet.

[0046] Some techniques of the present disclosure relate to coordinate system prediction. The predicted coordinate system may include a frame in a global coordinate system or a local coordinate system. ML models for this purpose may be enhanced by using representation learning. For example, representation learning may involve training a first configuration of a neural network (e.g., U-Nets, transformers, autoencoders, or convolutional and pooling layer networks, etc.) to learn a representation of one or more teeth, and then using a second configuration of the neural network (e.g., multilayer perceptrons, autoencoders, transformers, etc.) to predict information about one or more coordinate axes, such as one or more local tooth coordinate system axes (e.g., 3 coordinate system axes for individual teeth). The predicted information may include at least one of one or more transformations or one or more vectors that can be converted into transformations. At least one of two or more direction vectors or one or more position vectors may be calculated in a single execution of the second configuration. The direction vector or position vector may be used as input to generate at least one of three or more coordinate axes or a coordinate system origin. In some cases, the second configuration may be trained to predict two (or more) direction vectors (e.g., orthogonal vectors pointing in directions 90 degrees apart from each other) and one (or more) position vectors defining the origin of the local coordinate system. A Graham-Schmidt process (or a variation of Graham-Schmidt or another mathematical technique) may then be performed to predict three (or more) orthogonal local coordinate axes based on the two direction vectors. In some implementations, the first configuration of the neural network may take mesh element features as input to improve the data precision and accuracy of the generated representation. For example, a mesh element feature vector may be calculated for each mesh element of an input dental mesh (or point cloud). The mesh element feature values ​​within the mesh element feature vector provide the first configuration of the neural network with valuable information about the shape and / or structure of the input dental mesh (or point cloud). The mesh element feature vector may include at least one of spatial mesh element features or structural mesh element features.

[0047] In some embodiments, representation learning can be used to place orthodontic hardware relative to the patient's teeth. In other embodiments, one or more appliance components can be placed relative to one or more teeth. Some embodiments may use U-Net to generate representations. Some embodiments may use autoencoders (such as VAEs or capsule autoencoders) to learn representations of the basic features of one or more meshes related to the field of oral care (including information about the structure of the tooth mesh in some cases). This representation (latent vector or latent capsule) can then be used as input to a module that generates one or more transformations for one or more hardware elements or appliance components. In some embodiments, these transformations can place the hardware element or appliance component into the pose required for appliance generation (e.g., a dental restoration appliance or an indirect bonding tray). In some embodiments, the transformation can be described by a 9x1 transformation vector (e.g., specifying a translation vector and a quaternion). In other embodiments, the transformation can be described by a transformation matrix (e.g., a 4x4 affine transformation matrix). In some implementations, principal component analysis can be performed on the oral care grid, and the resulting principal components can be used as at least part of a representation of the oral care grid in later machine learning and / or other predictive or generative processes.

[0048] Depending on the particular implementation, additional methods may also be used to improve the performance of the ML model. For example, end-to-end training may be applied to the disclosed techniques involving two or more neural networks, where the two or more neural networks are trained together (e.g., weights are updated simultaneously during the processing of each batch of input oral care data). In some implementations, end-to-end training may be applied to hardware / component placement by simultaneously training a neural network that learns representations of one or more oral care objects and a neural network that processes those representations.

[0049] Another way to improve the ML models described herein is to use transfer learning. In some specific implementations, a network (e.g., U-Net) can be trained for a first task (e.g., coordinate system prediction) and then used to provide one or more starting neural network weights for the training of another neural network, which is trained to perform a second task (e.g., setting prediction). The first network can learn low-level neural network features of an oral care grid and is shown to perform well in the first task. By using the first network as a starting point for training, the second network can experience faster training and / or improved performance. Certain layers can be trained to encode the neural network features of the oral care grid in the training data set. These layers can then be fixed (or receive small fine-tuning during training) and combined with other neural network components (such as additional layers) that are trained for one or more oral care tasks. In this way, a portion of a neural network used for one or more technologies of the present disclosure can receive initial training for another task, which can produce important learning in the trained network layers. This encoded learning can then be built on with further task-specific training. In some implementations, a neural network used to make predictions based on an oral care grid can first be partially trained on one or more general / publicly available datasets and then further trained on oral care data.

[0050] In some embodiments, a neural network previously trained on a first data set (oral care data or other data) may subsequently receive further training on oral care data and be applied to oral care applications (such as mesh reconstruction autoencoders, mesh segmentation, mesh segmentation verification, coordinate system prediction, coordinate system verification, mesh cleaning, mesh cleaning verification, chairside intraoral dental scan verification, clear tray appliance (CTA) setup verification, bracket / accessory / hardware placement verification, generation of customized oral care appliance components, placement of customized oral care appliance components, verification of customized oral care appliances or components (e.g., verification of shape or placement of dental prosthetic appliance components), restoration design generation, restoration design generation verification, fixture model verification, and CTA trim line verification, as well as verification using autoencoders). Transfer learning may be used to further train any of the following networks from published literature: GCN (graph convolutional network), PointNet, ResNet, or any other neural network from published literature listed earlier in this section.

[0051] Yet another approach involves adding attention gates to ML models. In general, attention gates may be integrated with one or more of the neural networks of the present disclosure, with the advantage of enabling the associated neural network architecture to focus on one or more input values. In some implementations, attention gates may be integrated with a U-Net architecture, with the advantage of enabling the U-Net to focus on certain inputs. Attention gates may also be integrated with encoders or autoencoders (such as VAEs or capsule autoencoders). Some implementations of the presently disclosed techniques may benefit from one or more attention layers in a transformer, where the transformer is trained to generate a 3D oral care representation.

[0052] Figure 2 is an example technique 200 that can be used to train the ML models described herein. In general, the receiving module 202 is configured to receive patient case data 204. Typically, the patient case data 204 represents a digital representation of a patient's mouth. This may mean, for example, that the receiving module 202 may receive one or more malocclusion arches (e.g., 3D meshes representing upper and lower arches of a patient's teeth, i.e., a dentition of a patient's mouth including multiple aspects of the patient's dental anatomy, which may include teeth and may include gums). Depending on the specific implementation, the malocclusion arches may be arranged in an occlusal position or other orientation. In other implementations, a single arch may be required. For purposes of illustration, additional implementations will be described in more detail below. In other words, the receiving module 202 may receive mesh data corresponding to a 3D mesh of one or more patient's dentitions. It should be understood that both the amount of 3D mesh data and the type of 3D mesh data received by the receiving module 202 as part of the patient case data may vary based on the specific implementation. For example, in an implementation regarding verification of bracket placement, the mesh data received as part of the patient case data 204 may include only 3D mesh data regarding a particular tooth and associated bracket, while in an implementation regarding verification of a 3D printed component, the 3D data received as part of the patient case data 204 may include 3D mesh data associated with a component examined in the form of a CT scan or other diagnostic image, to name a few additional examples. Depending on the particular implementation, the patient case data 204 may also include a 3D representation of the patient's gingival tissue.

[0053] As shown in the example, the receiving module 202 also receives "reference truth" data 206. Generally speaking, these "reference truth" data 206 specify the expected results of applying other technologies disclosed herein, which are mesh segmentation, coordinate system prediction, mesh cleaning, restoration design and bracket / attachment placement, as well as all verification applications of the present disclosure, just to name a few examples. "Reference truth" and "reference" as used herein will be used interchangeably. For example, it should be understood that for the purposes of the present disclosure, a "reference" transformation vector is equivalent to a "reference truth" transformation vector. According to a specific implementation, and as will be described in more detail below, the "reference truth" data 206 may include a "reference truth" unique hot vector describing the expected transformation of a 3D geometry. For another example, the "reference truth" data 206 may include expected labels for various aspects of the 3D geometry. Other examples are also provided below. According to a specific implementation, the "reference truth" data 206 may be predefined or provided as a result of executing one or more other technologies disclosed herein.

[0054] Depending on the particular implementation, the receiving module 202 may also be configured to perform data augmentation on one or more aspects of the received data, including the patient data 204 and the "ground truth" data 206. Data augmentation is described in more detail below.

[0055] The system 100 can be configured to provide each mesh received by the receiving module 202 to the mesh preprocessor module 205, thereby allowing for preprocessing of any 3D mesh data received in the patient case data 206. This preprocessing step allows the system to convert the mesh into a form that allows the input mesh to be "consumed" by a neural network or other ML technique. In one specific implementation, the mesh preprocessor module 205 can be configured to generate a combination of edge, vertex, and face lists. One or more of these generated lists can be provided to both the generator 211 and the mesh feature module 208, as described in more detail below.

[0056] In addition to utilizing the mesh preprocessor module 205, the system 100 may also perform a number of additional operations before and after providing the patient case data 204 to the mesh preprocessor module 205. For example, according to a particular implementation, the system 100 may perform mesh cleaning on the patient case data 204 before providing the patient case data 204 to the mesh preprocessor module 205. Additionally, the system 100 may resample or update any information generated by the mesh preprocessor module 205. For example, in an implementation in which the mesh preprocessor module 205 generates a combination of edge, vertex, and face lists, the system may resample, update, or otherwise modify the labels identified in those lists. Additionally, according to a particular implementation, the system 100 may perform data augmentation on the resampled data.

[0057] The mesh feature module 208 may be configured to receive the list generated by the mesh preprocessor module 205 and generate feature information related thereto, which may be used by the ML model to generate predictions. For example, in one implementation, the mesh feature module 208 may calculate one or more of the following: edge midpoints, edge curvatures, edge normal vectors, edge normalized vectors, edge translation vectors, and other information related to each tooth in the 3D mesh received by the receiving module 202. Depending on the particular implementation, the mesh feature module 208 may or may not be utilized. That is, it should be understood that the calculation of any of the edge midpoints, edge curvatures, edge normal vectors, and edge translation vectors of the 3D mesh data included in the patient data 206 is optional. One advantage of using the mesh feature module 208 is that a system utilizing the mesh feature module 208 may be trained more quickly and accurately, but the technique 200 still performs better than prior art techniques that do not utilize the mesh feature module 208.

[0058] The technique 200 also utilizes a generative adversarial network ("GAN") to achieve certain aspects of the improvements. Generally speaking, a GAN is a type of ML model in which two neural networks "compete" with each other to provide predictions, these predictions are evaluated, and then the evaluations of the two models are used to improve each other's training. In some implementations, the GAN can be a conditional GAN, in which the generated output is adjusted based on some input data. One example in which conditional GANs have been found to provide benefits is in the field of restoration design. In those implementations, these adjusted input data can be an unrestored mesh and associated textual specifications. In some implementations, and as will be described below, the textual specifications can be a process that uses natural language processing (NLP) to extract key values, such as an added height or added width that has been specified for each treated tooth (e.g., in the example of dental restoration design, which produces a target geometry for each treated tooth).

[0059] As shown in this example, the two neural networks of the GAN are the generator 211 and the discriminator 235. In other implementations, models other than neural networks can be used for the generator or the discriminator.

[0060] The generator 211 receives input (e.g., one or more 3D meshes included in the patient case data 206). Depending on the particular implementation, the generator 211 uses the received input to determine a predicted output 207 related to the 3D mesh. For example, for segmentation, the generator 211 may be configured to predict a segmentation label, and in an implementation of predicting a clear tray appliance setting, the prediction may include one or more vectors corresponding to one or more transformations applied to the 3D mesh included in the patient case data 206. Other predicted outputs 207 are also possible. In some implementations, the generator 211 may also receive random noise, which may include junk data or other information that may be used to purposefully attempt to confuse the generator 211. Depending on the particular implementation, and as described above, the generator 211 may implement any number of neural networks, including MeshCNN, ResNet, U-Net, and DenseNet. In other cases, the generator may implement an encoder.

[0061] Since the generator 211 can be implemented as one or more neural networks, the generator 211 can include an activation function. The activation function determines whether a neuron in the neural network will fire (e.g., send an output to the next layer). Some activation functions may include: a binary step function and a linear activation function. Other activation functions give the network nonlinear behavior, including: a sigmoid / logistic activation function, a Tanh (hyperbolic tangent) function, a rectified linear unit (ReLU), a leaky ReLU function, a parametric ReLU function, an exponential linear unit (ELU), a softmax function, a swish function, a Gaussian error linear unit (GELU), and a scaled exponential linear unit (SELU). The linear activation function is well suited for some regression applications (and other applications) in the output layer. In the output layer, the sigmoid / logistic activation function is well suited for some binary classification applications (and other applications). The sigmoid activation function is well suited for some multi-class classification applications (and other applications) in the output layer. In the output layer, the sigmoid activation function is well suited for some multi-label classification applications (and other applications). The ReLU activation function works well for some Convolutional Neural Network (CNN) applications in the hidden layers (among other applications). The Tanh and / or sigmoid activation functions work well for some Recurrent Neural Network (RNN) applications, for example, in the hidden layers (among other applications).

[0062] After the generator 211 determines one or more predicted outputs 207, the generator 211 may be trained. Generally speaking, training the generator 211 involves comparing the predicted outputs 207 to corresponding ground truth inputs 208. For example, the predicted output 207 associated with the lower left canine corresponding to number twenty-seven of the International Dental Numbering System is compared to the ground truth output 208 for the same canine. As previously described, the ground truth input is an input that has been verified as a correct label for a particular portion of the 3D mesh data included in the patient case data 206. Depending on the particular implementation, the ground truth input 208 may be derived or otherwise determined from the ground truth data 206, or may be the ground truth data 206.

[0063] The differences between the predicted output 207 and the reference true input 208 can be used to calculate one or more loss values ​​G1216. For example, these differences can be used as part of the loss function calculation, or for the calculation of reconstruction errors. Some specific implementations may involve comparing the volumes and / or areas of two meshes (i.e., representations 207 and 208). Some specific implementations may involve calculating the minimum distance between corresponding vertices / faces / edges / voxels of the two meshes. For a point in one mesh (e.g., a vertex, a midpoint on an edge, or a triangle center), the minimum distance between the point and the corresponding point in the other mesh is calculated. Different methods can be considered in the case where the other mesh has a different number of elements or there is no clear mapping between corresponding points of the two meshes.

[0064] Regardless of the manner in which the difference between the predicted output 207 and the ground truth input is determined, various loss values ​​may be determined as part of the technique 200 or any other technique described herein. These losses include L1 loss, L2 loss, MSE loss, cross entropy loss, etc. Losses may be calculated and used to train neural networks such as multi-layer perceptrons (MLPs), U-Net structures, generators and discriminators (e.g., for GANs), autoencoders, variational autoencoders, regularized autoencoders, masked autoencoders, transformer structures, etc. For example, in learning sequences, some implementations may use triplet loss or contrastive loss.

[0065] The loss can also be used to train the encoder structure and the decoder structure. The KL divergence loss can be used at least in part to train one or more neural networks of the present disclosure, such as a mesh reconstruction autoencoder, which has the advantage of imparting Gaussian behavior to the optimization space. This Gaussian behavior can enable the reconstruction autoencoder to produce better reconstructions (i.e., when the latent vector representation is modified and the modified latent vector is reconstructed using the decoder, the resulting reconstruction is more likely to be a valid instance of the input representation). There are other techniques for calculating the loss that can be described elsewhere in the present disclosure. This loss can be based on quantizing the difference between two or more 3D representations.

[0066] The mean squared error (MSE) loss may involve the computation of the average squared distance between two sets, vectors, or data sets. The MSE may generally be minimized. The MSE may be applied to regression problems where the predictions generated by a neural network or other ML model may be real numbers. In some implementations, the neural network may be equipped with one or more linear activation units at the output to generate the MSE predictions. Mean absolute error (MAE) loss and mean absolute percentage error (MAPE) loss are also possible.

[0067] In some implementations, cross entropy can be used to quantify the difference between two or more distributions. In some implementations, cross entropy loss can be used to train the neural network of the present disclosure. In some implementations, cross entropy loss may involve comparing the predicted probability with the benchmark true probability. Other names for cross entropy loss include "logarithmic loss", "logistic loss" and "log loss". A small cross entropy loss may indicate a better (ie, more accurate) model. The cross entropy loss may be logarithmic. In some implementations, cross entropy loss may be applied to binary classification problems. In some implementations, the neural network may be equipped with a sigmoid activation unit at the output to generate a probability prediction. In the case of multi-class classification, cross entropy may also be used. In this case, in some implementations, a neural network that has been trained to perform multi-class predictions may be equipped with one or more softmax activation functions at the output (e.g., where there is an output node for the class to be predicted).

[0068] Other loss calculation techniques that can be applied in the training of the neural network of the present disclosure include one or more of the following: Huber loss, hinge loss, classification hinge loss, cosine similarity, Poisson loss, Logcosh loss, or mean square logarithmic error loss (MSLE). Other loss calculation methods are described herein and can be applied to the training of any neural network described in the present disclosure.

[0069] In some implementations, one or more neural networks of the present disclosure may be trained, at least in part, by a loss based on at least one of: point-wise mesh Euclidean distance (PMD) and earth mover distance (EMD). Some implementations may incorporate a Hausdorff distance (HD) calculation into the loss calculation. Calculating the Hausdorff distance between two or more 3D representations (such as 3D meshes) may provide one or more technical improvements because the HD considers not only the distance between the two meshes, but also the way those meshes are oriented and the relationship between the shapes of the meshes in those orientations (or positions or poses). The Hausdorff distance may improve comparisons of two or more instances of two or more tooth meshes, such as tooth meshes in different poses (e.g., comparisons of predicted settings to baseline true settings, which may be performed in the process of calculating a loss value for training a setting prediction neural network).

[0070] See again Figure 2 , G1 216 may represent the regression loss between the predicted output 207 and the ground truth input 208. That is, according to one implementation, the loss G1 216 reflects the percentage by which the predicted output 207 deviates from the ground truth input 208. That is, the generator loss G1 216 may be an L2 loss, a smoothed L1 loss, or some other type of loss. According to a particular implementation, the L1 loss is defined as , where P represents the predicted output 207 and G represents the ground truth input 208. Depending on the specific implementation, the L2 loss can be defined as , again where P represents the predicted output 207 and G represents the reference true input 208. In addition, and as will be described in more detail below, a loss value G1 216 may be provided to the generator 211 to further train the generator 211, for example, by modifying one or more weights in the neural network of the generator 211 to train the underlying model and improve the model's ability to generate a predicted output 207 that mirrors or substantially mirrors the reference true input 208. Any of these losses may be used to provide a loss value for training the neural network through a suitable training algorithm (such as back propagation). In some cases, an accuracy score may be used for training the neural network. The accuracy score quantifies the difference between the predicted data structure and the reference true data structure. The accuracy score (e.g., in normalized form) may be fed back into the neural network during the training of the network, for example, through back propagation. In the case of segmentation, the accuracy score may count the matching labels between the predicted grid and the reference true grid (i.e., where each grid element has an associated label). The higher the percentage of matching labels, the better the prediction (i.e., when comparing the predicted labels to the reference true labels). A similar accuracy score can be calculated in the case of mesh cleaning, which also predicts labels for mesh elements. The number or percentage of matches between the predicted labels and the ground truth labels can be used as an accuracy score, which can be used to train the neural network driving mesh cleaning (i.e., the accuracy score can be normalized).

[0071] Additionally, depending on the particular implementation, the system 100 can use the predicted output 207 to generate a predicted representation 220. Furthermore, the system 100 can use the reference ground truth input 208 to generate a reference ground truth representation 211. For example, in an implementation related to clear tray appliance generation, the predicted transformation and the reference ground truth transformation can be applied to the patient case data 206 to generate a predicted transformation and a reference ground truth transformation of the patient case data 206.

[0072] Depending on the particular implementation, the predicted representation 220 and the ground truth representation 211 may be labeled or otherwise annotated to indicate whether the representation corresponds to the ground truth data 206. Additionally, depending on the particular implementation, the representation 220 may be assigned a "false" value to indicate that the representation does not correspond to the ground truth label 208, while the representation 221 may be assigned a "true" value.

[0073] According to a particular implementation, representations 220 and 221 are provided as inputs to discriminator 235. Additionally, according to a particular implementation, 3D mesh data in patient case data 206 is also provided to discriminator 235. That is, discriminator 235 may receive various representations of data corresponding to patient case data 206, predicted output 207, ground truth data 206, ground truth input 208, and representations 220 and 221. In general, discriminator 235 is configured to determine when an input is generated from predicted output 207 or when an input is generated from ground truth input 208. The output of discriminator 235 is described in more detail in conjunction with the implementations discussed herein.

[0074] The discriminator 235 may be initially trained in various ways. For example, the discriminator 235 may be configured as an encoder structure, which in some cases (such as those described herein) may be configured to perform verification when used as a generator. For example, the initial encoder included in the discriminator 235 may be configured with random edge weights. Using back propagation, the encoder and thus the discriminator 235 may be continuously improved by modifying the value of the weights to allow the discriminator 235 to more accurately determine which inputs should be identified as "true" reference true representations and which inputs should be identified as "false" reference true representations. In other words, while the discriminator 235 may be initially trained, the discriminator 235 continues to develop / be trained as the technique 200 is executed. And similar to the generator 211, the accuracy of the discriminator 235 improves with each execution of the technique 200. Although, as will be appreciated by one of ordinary skill in the art, the improvement to the discriminator 235 will reach a limit where the accuracy of the discriminator 235 is not statistically improved, the training of the discriminator 235 is considered to be complete at this point. In other words, the system 100 may consider the training of both the generator 211 and the discriminator 235 to be complete when the discriminator 235 has difficulty distinguishing the predicted representation 220 from the reference true representation 221. As used herein, when the training of the generator 211 and the discriminator 235 is complete, they are described as being fully trained.

[0075] After the discriminator 235 generates an output, the technique 200 then compares the output of the discriminator 235 with the input to determine whether the discriminator 235 accurately distinguishes the predicted representation 220 from the reference ground truth representation 221. For example, the output of the discriminator 235 can be compared to the annotation of the representation. If the output and the annotation match, then the discriminator 235 accurately predicts the type of input received by the discriminator 235. Conversely, if the output and the annotation do not match, then the discriminator 235 inaccurately predicts the type of input received by the discriminator 235. In some implementations, and similar to the generator 211, the discriminator 235 can also receive random noise to intentionally attempt to confuse the discriminator 235.

[0076] In addition, depending on the particular implementation, the discriminator 235 may generate additional values ​​that can be used to train various aspects of the system implementing the technique 200. In one example, the discriminator 235 may generate a discriminator loss value 236 that reflects how accurately the discriminator 235 determines whether an input corresponds to the predicted representation 220 and / or the ground truth representation 221. Depending on the particular implementation, the discriminator loss 236 is larger when the prediction of the discriminator 235 is less accurate, and smaller when the prediction of the discriminator 235 is more accurate. In another example, the discriminator 235 may generate a generator loss value G2 238. Depending on the particular implementation, although not directly inversely proportional to the discriminator loss 236, the generator loss value G2 238 generally exhibits an inversely proportional relationship with the discriminator loss 236. That is, when the discriminator loss 236 is larger, the generator loss G2 238 is smaller, and when the discriminator loss 236 is smaller, the generator loss G2 238 is larger. In some implementations, the discriminator loss 236 may be determined using a binary cross entropy loss function calculated for both the "true" model and the "fake" model. In some implementations, the generator loss may consist of two losses: 1) the first loss is the generator loss G2 238 as determined by the discriminator (so binary cross entropy may be used); and 2) the second loss may be implemented by the l1-norm or mean squared error measuring the difference between the desired output and the actual output of the generator 211, for example as specified by the generator loss G1 216.

[0077] In other words, Figure 2 As shown, the generator loss G2 238 can be added to the generator loss G1 216 using a summation operation 240. And the sum of the generator loss G1 216 and G2 238 can be provided to the generator 211 for the purpose of training the generator 211. That is, it should be understood that the calculation of the generator loss G1 216 is for Figure 2Training of the GAN shown is not required. In some implementations, it is possible to train the generator 211 or the discriminator 235 using only a combination of the generator loss G2 238 and the discriminator loss 236. However, like other optional aspects of the present disclosure, the use of the generation loss G1 216 can be used to train the discriminator 235 more quickly to produce more accurate predictions. Depending on the specific implementation, the system 100 may use other steps or operations as part of the described techniques. For example, as already described but not depicted, an implementation related to the clear tray-type aligner setting may use one or more transformation steps to transform the patient data 206 using a predicted output 207 and a reference real input 208 corresponding to one or more 3D mesh transformations (e.g., scaling, rotation, and / or translation operations).

[0078] Depending on the particular implementation, the loss G1 216 and the loss G2 238 may also include one or more reasoning metrics specifying one or more differences between the predicted output 207 and the ground truth input 208 and / or the predicted representation 202 and the ground truth representation 221. That is, as an optional step, the system 100 may generate these reasoning metrics to further refine the training of one or more neural networks or ML models. These reasoning metrics may include: intersection over union metric, average boundary distance metric, boundary percentage metric, and over segmentation rate, to name a few examples.

[0079] In general, the intersection-over-union metric specifies the percentage of correctly predicted edges, faces, and vertices within a mesh after a certain operation (such as segmentation) is completed. The average boundary distance specifies the distance between the predicted output 207 (or predicted representation 220) and the reference true input 208 (or reference true representation 221) for a 3D representation (such as a 3D mesh). The boundary percentage specifies the percentage of the mesh boundary length of a 3D mesh (such as a segmented 3D mesh) where the distance between the reference true input 208 (or reference true representation) and the predicted output 207 (or predicted representation 220) is below a threshold. For example, the threshold may determine whether one or more predicted outputs 207 (such as a small line segment between each pair of boundary points) are close enough to the reference true input 208. In the case where the technique 200 is used to implement a segmentation process, if the distance is below the threshold, the system 100 may mark a particular line segment as a perfect boundary segment. The percentage represents the ratio of segments that are within the predicted boundary to segments that are within the reference true boundary. And the over-segmentation rate specifies the percentage of the boundary length of the teeth that are over-segmented. Depending on the specific implementation, one or more inference metrics can be used to additionally train the generator 211 or the discriminator 235, or both.

[0080] The technology of the present disclosure may include operations such as 3D convolution, 3D pooling, 3D deconvolution, and 3D up-pooling. 3D convolution can help segmentation processing, such as when downsampling a 3D representation (such as a 3D mesh or point cloud). 3D deconvolution is, for example, the inverse operation of 3D convolution in U-Net. 3D pooling can help segmentation processing, such as in a generalized neural network feature map. 3D up-pooling is, for example, the inverse operation of 3D pooling in U-Net. These operations can be implemented by one or more layers in the prediction or generation neural network described herein. These operations can be directly applied to aspects of the 3D representation such as mesh elements, which may include mesh edges or mesh faces. These operations provide technical improvements over other methods because these operations are invariant to mesh rotation, scaling, and translation changes. Generally speaking, these operations depend on edge (or face) connectivity, so as long as the edge (or face) connectivity is maintained, these operations will not be affected by mesh changes in 3D space. That is, these operations can be applied to an oral care mesh and produce the same output regardless of the orientation, position, or scale of the oral care mesh, which can improve data accuracy. MeshCNN is a general deep neural network library for 3D triangulated meshes that can be used for tasks such as 3D shape classification or mesh element labeling (e.g., for segmentation or mesh cleaning). MeshCNN implements these operations on mesh edges. Other toolkits and specific implementations can operate on edges or faces.

[0081] The technique 200 can be used to train ML models for many digital dental and digital orthodontic applications. Table 2 shows how the technique 200 can receive different data 204 and 206 for certain digital dental applications, and the form that the prediction output 207 can take depending on the particular implementation.

[0082] ML models (such as those described herein) can be trained to generate transformations to place prefabricated components (e.g., from a component library) for creating dental restoration appliances. Such dental restoration appliances can be used to shape dental composites in a patient's mouth while curing the composites (e.g., using a curing light), ultimately creating veneers on one or more teeth of the patient. 3M FILTEK Matrix is ​​an example of such a product. Dental restoration appliance components (e.g., library components) that can be placed using the technology of the present disclosure include: drainage holes (e.g., which can allow composites to flow out of the appliance), rear snap clips (e.g., which can enable the appliance to be grasped or manipulated), door hinges (e.g., which can enable the door to be rotated open or closed), door snaps (e.g., which can secure the door in a closed position), incisal alignment features (e.g., which can help align the appliance), center clips (e.g., which can enable the appliance to be aligned), custom labels, manufacturing box frames, diastema matrix handles, and the like. Further details on placing features and generating features can be found in PCT patent application WO2021 / 240290A1, the entirety of which is incorporated herein by reference.

[0083] For example, in a segmentation implementation, each patient case in the data set 204 consists of a pre-segmented dental arch. In some implementations, the technique 200 can be used to segment each tooth in the dental arch and label the tooth with its identity (i.e., performing traditional tooth segmentation). In some implementations, the technique 200 can be used to separate the face and tongue of the dental arch (i.e., performing face-tongue segmentation). In some implementations, the technique 200 can be used to separate the gingival portion of the dental arch from the teeth (i.e., performing gingival segmentation). In some implementations, the technique can be used to segment extraneous material directly from the gums (i.e., performing trim line segmentation). Some segmentation implementations may use MeshCNN to predict mesh element labels. Some implementations may train a U-Net structure to generate a representation of a 3D mesh and may also be simultaneously trained to predict mesh element labels. Other implementations may use other models to predict mesh element labels.

[0084] As described elsewhere in this specification, the receiving module 202 receives patient case data. In the depicted example, the receiving module 202 may receive patient case data 204 including dental arch data after one or more mesh cleaning operations have been performed on the patient's 3D dental arch geometry. For example, in one example, this may produce one or more cleaned dental arch geometries. The mesh cleaning operations may use one or more models of MeshCNN, U-Net, or other models to predict mesh element labels.

[0085] According to a particular implementation, the 3D dental arch geometry may include a 3D mesh geometry for the patient's gingival tissue, while in other implementations, the 3D dental arch geometry may omit the 3D dental arch geometry for the patient's gingival tissue. In addition, the receiving module 202 may be configured to also receive a reference true label as a reference true label 206, which describes a verified or otherwise known accurate label (e.g., labels "correct" and "incorrect") of a mesh element associated with a segmentation result performed on the 3D geometry. According to a particular implementation, the labels described with respect to the segmentation operation are used to specify a specific set of mesh elements (such as "edge" elements, "face" elements, "vertex" elements, etc.) for specific aspects of the 3D geometry. For example, a single triangular polygon of a 3D mesh includes 3 edge elements, 3 vertex elements, and 1 face element. Therefore, it should be understood that a segmented dental geometry consisting of many polygons may have a large number of labels associated with the segmented dental geometry.

[0086] In addition, the received geometry may have one or more labels that are applied to the corresponding geometry to generate representations 220 and 221. For example, in one embodiment, at each iteration of generator 211, generator 211 may output a label for each mesh element found in the input dental arch. Each of these labels marks the corresponding mesh element (e.g., an edge) as belonging to a gingival or tooth structure in the input mesh. In the case where the mesh element belongs to a tooth, the identity of the tooth is also specified. For example, an edge may be assigned a label to indicate that the mesh element belongs to the gingiva. Another mesh element may be assigned a label to indicate that the mesh element belongs to the upper right third molar. Yet another mesh element may be assigned a label to indicate that the mesh element belongs to the lower left central incisor. Other labels are also possible.

[0087] Once trained, the generator 211 may be used to generate accurate prediction outputs 207 for the patient case data 206 received by the receiving module 202 . Figure 3 An example technique 300 for generating predicted labels 207 is shown in FIG. Generally speaking, technique 300 uses the same computer modules and components to perform many of the same steps as technique 200. That is, as can be seen from this example, technique 300 does not train generator 211, but relies on the training in technique 200 to generate predicted output 307. In addition, technique 300 does not include a discriminator. As discussed above with respect to Figure 2 It should be understood from the discussion of that when generator 211 is trained, predicted output 207 will eventually be equal to or substantially equal to predicted output 307.

[0088] Some of the techniques described in Table 2 (and elsewhere in this disclosure) can benefit from training a representation learning model. In some implementations, such a representation model can be used to implement Figure 2, Figure 3 , Figure 4 and Figure 5 Generator 211 in Fig. 9 Module 911 and / or Fig.10 Module 1011 in. In some embodiments, the representation learning model may include a first module and a second module, the first module may be trained to generate representations of received 3D oral care representations (e.g., teeth, gums, hardware and / or appliance components), and the second module may be trained to receive those 3D representations and generate one or more output oral care representations. In some cases, such output oral care representations may include transformations that may be applied to hardware or appliance components for placement relative to one or more teeth. In some cases, such output oral care representations may include one or more coordinate system axis definitions. In some cases, such output oral care representations may include meshes or labels on mesh elements corresponding to teeth, gums, or other aspects of the dentition (e.g., using mesh cleaning, mesh segmentation, or dental restoration design).

[0089] In some implementations, a first module of a representation learning model may be trained to generate a 3D representation of one or more teeth (and / or gums or hardware) suitable for providing to a second module, wherein the second module is trained to output one or more predicted transforms (or other oral care representations). In some implementations, one or more layers including convolution kernels (e.g., with kernel size 5 or some other size) and pooling operations (e.g., average pooling, max pooling, or some other pooling method) may be trained to create representations for one or more received oral care 3D representations in the first module. In some implementations, one or more U-Nets may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more autoencoders may be trained to generate representations for one or more received oral care 3D representations in the first module (e.g., wherein a 3D encoder of the autoencoder is trained to convert one or more tooth 3D representations into one or more latent representations, such as latent vectors or latent capsules, wherein such latent representations may be reconstructed into replicas of one or more input tooth meshes via a 3D decoder of the autoencoder). In some implementations, one or more 3D encoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. In some implementations, one or more pyramid encoder-decoder structures may be trained to generate representations for one or more received oral care 3D representations in the first module. Other methods of encoding representations are also possible.

[0090] The representation of one or more teeth may be input to a second module of a representation learning model, such as an encoder structure, a multi-layer perceptron (MLP), a transformer (e.g., including at least one of a 3D encoder and a 3D decoder, which may be configured with a self-attention mechanism that enables the network to focus training on key inputs), an autoencoder (e.g., a variational autoencoder or a capsule autoencoder), which has been trained to output one or more representations (e.g., a transformation to place an oral care grid, such as the grid in the example of hardware and appliance component placement technology). In some embodiments, the transformation may include one or more 4×4 matrices, Euler angles, or quaternions. The second module may be trained at least in part by calculating one or more loss values ​​(such as L1 loss, L2 loss, MSE loss, reconstruction loss), or one or more of other loss calculation methods found elsewhere in the present disclosure. Such loss functions may quantify the difference between one or more generated representations and one or more reference representations (e.g., a benchmark true transformation known to have good function). In some implementations, either or both of module one and module two may receive one or more mesh element features associated with one or more oral care meshes (e.g., a mesh element feature vector may be calculated for one or more mesh elements of an input tooth, gum, hardware article, or appliance assembly). The advantages of receiving mesh element features generally relate to improving the underlying system. For example, such implementations allow the first module to more accurately represent the received 3D representation and allow the second module to generate a more accurate output 3D representation (e.g., a transformation, a representation of a tooth anatomy, or a label on a mesh element).

[0091] Figure 4 Depicted is a technique 400 for training an ML model according to certain aspects of the present disclosure. In general, the technique 400 uses the Figure 2 That is, now describe Figure 4Certain additional aspects of . For example, depending on a particular implementation, it may be inappropriate or incorrect to apply the prediction output directly to the patient data to generate the prediction representation. For example, in a segmentation-based implementation, it is appropriate to apply one or more prediction labels to generate the prediction representation 220 because, for example, the underlying representation of the patient data is not modified. In contrast, in other implementations, the prediction output 407 may be one or more vectors describing one or more transformations, and it may be necessary to apply incremental processing steps to apply those transformations to the patient data. For example, when the prediction output 207 is the prediction output vector 407, the grid transformation module 418 may be used to apply the one or more prediction vectors to the patient data to generate the prediction representation 420. Similarly, when the reference input 207 is the reference input vector 407, the grid transformation module 426 may be used to apply the prediction vectors to the patient data to generate the prediction representation 421. The transformers 418 and 426 may use conventional techniques to apply the corresponding vectors to the patient data 204, translate, scale, and rotate the patient data 204 to generate the prediction representation 420 and the reference representation 421, respectively.

[0092] One specific example involves coordinate system generation. Digital dental and digital orthodontic applications may require the definition of coordinate systems in order to operate on 3D mesh models of teeth and gums. Some coordinate systems may be defined relative to the entire dental arch of teeth and are referred to as global coordinate systems. Some coordinate systems may be defined relative to individual teeth and are referred to as local coordinate systems.

[0093] In general, a tooth coordinate system comprises a set of XYZ axes that are used to facilitate mathematical transformations and other operations on a tooth mesh. A tooth coordinate system operates relative to the tooth with the origin located at a carefully selected central position relative to the tooth mesh. The local coordinate system of a tooth is contrasted with a global coordinate system, which has its origin located relative to the center of the entire dental arch. The global coordinate system is used to facilitate mathematical transformations and other operations on the entire dental arch. The correct choice of a tooth coordinate system is critical to the correct function of operations in the design of dental and orthodontic appliances relative to the tooth.

[0094] In a specific implementation related to coordinate system prediction, each patient case in the dataset 204 includes: 1) a set of segmented teeth in a dental arch; and 2) a set of transformations for describing a coordinate system relative to each of the teeth. In the depicted example, the generator 211 can be configured to generate one or more prediction vectors 407. In addition, the reference real input 208 is in Figure 4407 and 408 are represented as reference truth vector 408. As already mentioned, both vectors 407 and 408 represent transformations to be performed on patient case data 204 in order to generate one or more predicted representations 420 and reference truth representations 421, respectively. Vectors 407 and 408 can be of any size, but it has been observed that vectors with 4x4 dimensions are well suited for technique 400.

[0095] According to the depicted example, the technique 400 transforms the patient case data 204 using mesh transformation modules 418 and 426, generating prediction representations 420 and 421, respectively. In addition, and consistent with other aspects of the present disclosure, for each prediction transformation (e.g., as defined by the prediction vector 407), the system 100 calculates the LossG1 216 between the generated prediction vector 407 and the corresponding reference true vector 408. The LossG1 216 is fed back to update the weights of the generator 211. In addition, as already described, both the generated vector 407 and the reference true vector 408 are provided to the discriminator 235 (along with the relevant patient data 204, such as a dental mesh). The discriminator 235 attempts to label the vectors 407 and 408, thereby distinguishing true (reference true) from false (generated).

[0096] Depending on the specific implementation, the generator 211 may be replaced by an encoder, which may be viewed as Figure 4 The first half of the U-Net structure depicted in . Specifically, the encoder may include any number of mesh convolution operators 402 and any number of mesh pooling operators 404, but typically does not include an on-mesh pooling operator 406 or a mesh deconvolution operator. That is, the mesh convolution operator 402 generates a high-dimensional feature for each mesh element by collecting neighbor information of the element based on topology (i.e., based on mesh surface connectivity information). The mesh pooling operator 404 at each layer of the encoder simplifies the input mesh to a coarser resolution by reducing the count of mesh elements and aggregating neighboring features of each element. The high-dimensional features aggregated at the last layer are further processed by multiple fully connected layers and are ultimately transformed into a final regression output (e.g., a transformation matrix corresponding to a tooth coordinate system for tooth movement in 3D).

[0097] In some implementations, the techniques disclosed herein can simultaneously predict two orthogonal coordinate axes. From these two orthogonal coordinate axes, a third coordinate axis can be calculated, for example, using the Gram–Schmidt process.

[0098] Depending on the particular implementation, the coordinate system prediction operates on a six-dimensional representation. In addition, while it is possible to use the technique 400 to perform coordinate system prediction on a point cloud (e.g., a 3D point cloud), it is advantageous to perform coordinate system prediction on a 3D geometry such as a 3D mesh. This is because, in general, a 3D mesh (as opposed to a 3D point cloud) is more accurate in its ability to capture the local surface structure of an object. For example, two surfaces may be very close in Euclidean space, but very far apart from each other in a mesh topology (or in geodesic space). Therefore, a 3D mesh is a better choice for representing a surface.

[0099] In addition, for edges and vertices, vertex elements in a 3D mesh may have infinitely (theoretically) connected adjacent vertices, while edge elements in a 3D mesh have a fixed number of adjacent edges (e.g., 4 adjacent edges). The border edge can be given two virtual edges, making the number four. The use of a grid makes grid convolutions in 3D more straightforward. The fixed number of adjacent edges also makes the grid convolution output relatively more stable during training. From a grid topology perspective, the number of edges in a 3D grid is typically greater than the number of vertices (e.g., typically 3 times). In a sense, the grid resolution can be increased by using edges for prediction because there are many more edges than vertices in a typical grid. In addition, it should be understood that neural networks generally benefit from training on a large number of elements. Therefore, by using a 3D grid, the resulting reasoning is improved, and the beneficial effects are passed on to later post-processing steps, resulting in an overall more accurate system.

[0100] Similar to Figure 2 and Figure 3 Once trained, the generator 211 can be used to generate an accurate prediction vector 407 for the patient data 204 received by the receiving module 202. An example technique for generating the prediction vector 407 is Figure 5 4. The technique 500 shown in FIG. 4 shares many of the same features as the techniques 300 and / or 400 described above.

[0101] Figure 6 is an illustration of an example ML architecture 600 that may be used by the system 100 to design and manufacture dental appliances for restoring a patient's dental anatomy in accordance with various aspects of the present disclosure. For example, many of the techniques described herein rely on some form of the architecture 600 as the basis for the ML models described herein.

[0102] In the depicted example, the ML model 600 is a U-Net architecture. The eponymous architecture is configured as one or more grid convolution operators 602a-602n, grid pooling operators 604a-604n, grid unpooling operators 406a-406n, and grid deconvolution operators arranged in an inverted pyramid or "U" configuration. As used herein, it should be understood that the term "operator" is synonymous with and used interchangeably with the terms "node" and "layer", which are also used to describe similar operations in ML parlance.

[0103] In general, the U-Net architecture 600 involves mesh pooling and mesh unpooling operations, which facilitate the process of extracting mesh element neighbor information. Each successive pooling layer helps the model learn neighbor geometry information by reducing the resolution relative to the previous layer. Each successive mesh unpooling layer helps the model expand this summarized neighbor information back to a higher resolution. A series of mesh pooling layers followed by a series of mesh unpooling layers will enable efficient and accurate training of the U-Net, and enable the U-Net to output features for each element containing both local and global geometry information.

[0104] According to a specific implementation, one purpose of the U-Net architecture 600 is to compute a high-dimensional feature vector for the input mesh. For example, according to a specific implementation, the U-Net architecture 600 computes a feature vector for each mesh element (e.g., a 128-element feature vector for each edge, vertex, or face element). The vector exists in a high-dimensional space that is capable of representing the local geometry of the edge within the context of the local teeth, and also represents the global geometry of the two dental arches. The encoder uses the high-dimensional features of the elements within each tooth to make predictions. This combination of local and global information contributes to the accuracy of the predictions. The combination of local and global information enables the U-Net architecture 600 to take into account geometric constraints. For example, during the treatment of clear tray-type aligners, it is undesirable for teeth to collide in 3D space. The combination of local and global information enables the U-Net architecture 600 to generate transformations that reduce or eliminate the incidence of conflicts, and thus produce higher accuracy relative to the prior art. In the event of a collision, the technology of WO2020 / 136587A1 “Methods to automatically remove collisions between digital mesh objects and smoothly move mesh objects between spatial arrangements” can be used to detect and remove collisions between tooth meshes of a dental arch. Consistent with the present disclosure, geometric quantities, such as penetration depth, penetration direction, and counts of overlapping mesh elements (such as vertices), can be calculated, consistent with the detection and removal of tooth mesh collisions. Consistent with the present disclosure, the mesh shape and / or position can be perturbed or changed to reduce or eliminate the incidence of collisions, which in some cases may still exist after operation of the neural network structure of the present disclosure. Generally speaking, information provided to the ML model 600 is first processed by propagating “downward” through operators 602a, 604a, 602b, 604n, etc. until the information reaches the bottom operator (represented here by the mesh convolution operator 602c). This information then propagates "upwards" through operators 606a, 602d, 606n, etc. until it is output by a final grid convolution operator 402n, which may be used by various aspects of the present disclosure, as will be described in more detail below.

[0105] Figure 6 The example U-Net architecture shown is depicted as having a total of nine layers (or nine operators), but it should be understood and appreciated that the U-Net architecture can be configured with any number of convolutional layers, any number of grid pooling layers, and any number of grid unpooling layers to achieve desired results.

[0106] In general, each of the operators 602a-602n, 604a-604n, and 606a-606n can be configured using conventional techniques to modify received inputs related to 3D mesh data (including, for example, mesh size and pose, as embodied by edge lengths, edge curvatures, edge normals, edge midpoints, and other edge data) to produce specific outputs suitable for each of the operators 602a-602n, 604a-604n, and 606a-606n, as will be described in more detail below.

[0107] According to a particular implementation, the mesh convolution operators 602a-602n disclosed in the present disclosure may be configured to be agnostic to the size and pose (e.g., position and / or orientation) of the input 3D mesh according to a particular implementation. The advantage of this agnostic approach is that, as opposed to an input mesh of a fixed size and / or orientation, the mesh cleaning operator can be used to process an original input mesh of arbitrary orientation.

[0108] However, in other implementations, size and pose information is required, such as in the context of regression operations. In implementations where size and pose information is desired, convolution operations may alternatively be configured to be agnostic to size and pose information. For example, convolution filters used as part of the convolution operators 602a-602nML model may be specifically configured to be sensitive to size and pose information, where such a system should not be agnostic to that information. In other implementations, there may be specific aspects of operations that benefit from size and pose information. One specific example is for 3D mesh segmentation, which benefits from a size and pose agnostic mode in some applications (e.g., segmentation of the gingiva, which is used to find a general area of ​​an intraoral scan containing teeth), but does not benefit in other applications (e.g., tooth segmentation, which benefits from information about the left and right sides of the mesh). Therefore, it should be understood that within a particular type of task (e.g., a segmentation task), various aspects of the ML model may be configured to be size and pose agnostic for those operations that benefit, and other aspects of the ML model may be configured to be size and pose sensitive for those operations.

[0109] The mesh pooling operators 604a-604n are configured to resample the input mesh to a lower resolution. As a result, the mesh is continuously refined and resampled to a lower resolution through each successive layer of the mesh pooling operators 604a-604n. This allows downsampling or shrinking of the mesh input. For example, downsampling of information in 3D space may take a 3×3×3 set of information and combine it into a single 1×1×1 representation. For example, in the context of 3D mesh information, the four adjacent edges of a given edge will be combined into a single edge at the next resolution level. The mesh resolution (mesh surface area) after downsampling will be reduced by a factor of 4.

[0110] One of the many advantages of this approach is that the grid pooling operators 604a-604n cause each feature to collect information about its neighbors and summarize that information into a form that is passed to the next layer. Thus, as the grid information moves through the U-Net architecture 600, the output of the lowest level convolution operation 602 (such as 602c in the depicted example) takes the form of a downsampled grid that reveals global information about the original input grid. In other words, the output of the lowest level convolution operation 602 is considered to constitute fully summarized information and it can be used in accordance with various techniques of the present disclosure. For example, the downsampled output of the lowest level grid convolution operation 602 can be used in classification operations (e.g., for 3D verification) and regression operations (e.g., for coordinate system prediction), to name just a few examples.

[0111] In addition, the fully summarized information may undergo further processing by additional operators (e.g., depicted as operators 602n, 604n, and 606n). For example, the fully summarized information output by operator 402c may be processed by mesh unpooling operators 606a and 606n to increase the resolution of the mesh information. As depicted by the example ML model 600, there is a 1:1 relationship between mesh pooling operators 604a and 604b and mesh unpooling operators 606a and 606n. That is, after a sufficient number of mesh unpooling operations (e.g., performed by operators 606a-606n) equal to the number of mesh pooling operations (e.g., performed by operators 604a-604b) have been performed, sufficient information to allow other automatic techniques to identify the element categories (e.g., edges, faces, vertices) in the 3D mesh is surfaced. For example, this allows an automated system to perform mesh segmentation on the output of the convolution operator 602n (eg, perform tooth segmentation, gingival segmentation, facial-lingual segmentation, etc.).

[0112] Figure 7 is an illustration of an example process 700 that may be used by system 100 to design and manufacture a dental appliance for restoring a patient's dental anatomy in accordance with various aspects of the present disclosure. Figure 7 An example process 700 for data augmentation that may be performed by the system 100 to advantageously increase the size of a training data set is depicted.

[0113] One of the main drawbacks of ML systems is that the accuracy of the model is limited by the training data. For example, low-quality data will produce a low-quality predictive model. Similarly, when analyzing real-world problems, a lack of data can inadvertently bias the model, reducing its overall accuracy.

[0114] Using process 700 , system 100 may avoid some of these drawbacks by generating additional training examples by adding one or more of random rotations, random translations, random scaling, and random perturbations of the 3D mesh.

[0115] For example, in step 702, the system 100 may receive a 3D mesh. Next, at step 704, the system 100 may generate a copy of the mesh data. Depending on the particular implementation, the copy may be stored in any of the storage devices 178. Next, in steps 706-712, the system may optionally perform operations including applying an incremental rotation to the mesh (step 706), applying an incremental translation to the mesh (step 708), applying an incremental scaling to the mesh (step 710), and randomly perturbing one or more mesh elements (step 712).

[0116] Depending on the specific implementation, incremental rotation, translation, skew, scaling (in any or all of the XYZ axes), and perturbations may be performed on the mesh using predetermined values ​​or randomly selected within a range. For example, when it comes to applying a perturbation to mesh elements, the system 100 may apply Gaussian noise with defined values ​​of 0 mean and 0.1 standard deviation to the positions of one or more vertices in the 3D mesh. Mesh elements that may be perturbed include edges, faces, and vertices. Other mesh elements are also possible. In some implementations, one element may be perturbed. In other implementations, multiple elements (adjacent or non-adjacent elements) may be perturbed. For example, a tooth tip on a tooth may be scaled to increase or decrease the protrusion of the tooth tip in the incisal direction. For another example, a tooth may be added to or removed from a dental arch.

[0117] In other specific implementations, operations from genetic algorithms may be introduced to assist the data enhancement process. The basics of genetic algorithms are well known to those skilled in the art. An optimization algorithm searches the space of possible solutions to a problem over many "generations". The fitness function describes the "fitness" or value of each possible solution. Poor solutions are removed from the population, and highly adaptive solutions are saved for further processing in the next "generation" or iteration of the algorithm. Genetic algorithms use mutation operators such as mutation and crossover to search the space of possible data structures to find one or more data structures with high "fitness" or utility. The perturbations already described are consistent with mutation operations. Crossover can be applied to a 3D mesh by creating a new mesh from two or more "parent" or source meshes. Data enhancement operations can introduce diversity to a training data set by creating a new tooth mesh containing mesh elements of other tooth meshes (for example, after a portion of a first tooth mesh is removed, the corresponding portion of a second tooth mesh is introduced and fused with the first tooth mesh).

[0118] It should also be understood that the system 100 can be configured to randomly select which of the one or more optional steps 706-712 to perform on the replicated mesh. For example, the system 100 can randomly select to perform steps 706 and 712 in one execution of the method 700, and can randomly select to perform only step 710, to name a few examples. In this way, the system 100 can generate a large number of training alternatives based on a single received 3D mesh.

[0119] According to a particular implementation of the present disclosure, process 700 may be used on 3D meshes for training ML models used in mesh segmentation, coordinate system prediction, mesh cleaning, restoration design, and bracket / accessory placement, as well as all validation applications of the present disclosure, to name a few examples.

[0120] Figure 8 is an example technique 800 for automatically validating a neural network trained using the techniques described herein.

[0121] Generally speaking, ML models can be trained to verify data sets to be used for digital dentistry or digital orthodontics. In some specific implementations, ML models such as neural networks can be used to verify 2D raster image views of 3D data. An example neural network is a convolutional neural network (CNN). Multiple views of 3D data can be generated. CNN is used to classify each view (e.g., as correct or incorrect), and the verification results of multiple those 2D raster views can be used to verify the correctness of the 3D data. In other specific implementations, the neural network can be a general deep neural network for 3D triangle meshes, such as MeshCNN. MeshCNN is an open source neural network implementation. MeshCNN uses geometric deep learning (or GDL) technology, which involves the first method of performing mesh processing, which operates on edges (or other mesh elements, such as vertices or faces) to implement mesh convolution, mesh pooling, mesh anti-pooling, mesh deconvolution and other 3D-specific deep learning techniques. The open source Minkowski engine includes a neural network with GDL capabilities, which additionally provides GDL operations for sparse convolution. Sparse convolution is a convolution technique that has a different representation value than the mesh convolution operation (i.e., voxels) found in MeshCNN. Voxels are used in sparse convolution operations. Voxels are the 3D geometric equivalent of pixels in 2D images. Sparse convolution techniques exploit the sparsity of data to make 3D volume processing more efficient in many cases. This increase in efficiency is important because some problems can be difficult to solve. In other words, the GDL technique can be applied to every GDL example of the present disclosure, including all 3D verification techniques, mesh segmentation, mesh cleaning, mesh coordinate system prediction, restoration prediction, restoration appliance component placement and generation, and bracket and accessory placement.

[0122] In these implementations, MeshCNN can be used to directly verify the correctness of 3D without having to rely on 2D raster image views of 3D data. In some implementations, the results of one of those verification operations can be fed back into the automated process to improve further iterations of the process of generating those 3D data. In other implementations, the results of one of those verification operations can be reported or displayed to a technician, who can then continue to correct problems with those 3D data. In other implementations, ML models such as neural networks can be used to directly verify 2D data (such as photos of dental or orthodontic appliances). In some implementations, the data to be verified may describe the dental geometry of the patient, possibly including teeth and / or gums. In other implementations, the data to be verified may describe dental or orthodontic appliances or components thereof. In some implementations, the verification invention described in the present disclosure can be integrated into an automated test suite (e.g., unit testing and regression testing for software and algorithms). In other words, although neural networks are the preferred ML method, other ML techniques may be used appropriately.

[0123] For example, MeshCNN can be trained on two (or more) categories of data, such as a 3D mesh corresponding to the RAW category (the "raw" output from segmentation) and a 3D mesh from the TECH category (a mesh modified or corrected by a technician). MeshCNN will become able to distinguish between these two categories and can be used in environments where teeth must be segmented for applications such as dental or orthodontic appliances. In some cases, the RAW category may correspond to a suboptimal state and the TECH category may correspond to an optimal state. In the 3D case, the MeshCNN or encoder may be trained to distinguish these categories. In the 2D case, the CNN may be trained to distinguish these categories. This method may also be applied to other verification operations of the present disclosure. The operational verification engine used in the deployment is designed to detect defects in 3D geometric structures (e.g., dental or orthodontic geometric structures). This operational verification system can be trained on RAW and TECH data categories as an alternative to the CORRECT and UNCORRECT categories that the verification engine may encounter in the field throughout operational use. This involves each of the verification applications described in the present disclosure (e.g., segmentation verification, mesh cleaning verification, coordinate system verification, dental restoration appliance component verification, 3D printed part verification, trim line verification, fixture model verification, and restoration design verification).

[0124] Now go to Figure 8In the depicted example, in step 802, a system (such as system 100) receives one or more 3D oral care representations, such as a 3D mesh of a patient's dentition (which may include information related to the patient's teeth, gingival tissue, and other aspects of the patient's oral anatomy), as well as other information. The received 3D mesh may vary according to a particular purpose. For example, in a specific implementation regarding mesh segmentation, the received 3D information may be related to a dental arch of a patient's mouth, which may include a 3D representation of teeth and / or gingival tissue, a specific implementation for verification of hardware or appliance component placement. The received 3D mesh may include a 3D representation of specific teeth and associated hardware. In a specific implementation regarding verification of a 3D printed part, the received 3D mesh may include 3D mesh data associated with a part examined in the form of a CT scan or other diagnostic imaging, to name just a few additional examples.

[0125] In step 803, the system 100 may receive a fully trained neural network, such as the fully trained generator 211 described above.

[0126] In step 804, the system 100 may optionally process the received 3D oral care representation to prepare for subsequent steps. For example, in one embodiment, the system 100 may generate or otherwise place components for dental restoration appliances on corresponding teeth in the 3D mesh that must be verified. In another embodiment, the system 100 may place brackets or accessories (or other hardware, such as buttons or hooks attached to teeth, to which resistance bands may be attached and attached to teeth) relative to specific teeth in the 3D oral care representation. In a related embodiment, the system 100 may predict a coordinate system of one or more teeth (e.g., including one or more local coordinate axes of each tooth). In yet other embodiments, the 3D oral care representation may be processed to facilitate identification or marking of mesh elements in a 3D mesh (or 3D point cloud) of a patient's dentition. Examples of where this may be useful include segmentation applications (e.g., tooth segmentation), mesh cleaning, or automatic restoration design generation. In another embodiment, and relative to segmentation, specific teeth may be marked as being correctly segmented or incorrectly segmented. Other types of verification regarding other aspects of the present disclosure are also possible. In other words, depending on the details of a particular implementation, there may be many ways to train a neural network that can validate a 3D oral care representation.

[0127] In step 806, the system 100 may use a 3D modeling tool to generate multiple 2D raster views for each tooth. According to a specific implementation, a 3D modeling tool such as GEOMAGIC may be used, for example, through an automated script. In some examples, other 3D modeling and rendering engines may be used. As used herein, a view may be defined as a specific orientation of a camera within a modeling tool, providing a specific representation of a 3D mesh having a 3-dimensional space represented in the modeling tool. In other words, at step 806, the camera within the modeling tool may be positioned so that each tooth in the 3D mesh is observed from a slightly different angle or vantage point within the modeling tool. The number of views generated may vary according to a specific implementation or a specific use case. For example, according to one specific implementation, fifteen different views of the 3D mesh are generated, although any number of views may be generated for a particular tooth. Therefore, if fifteen views are generated at step 806, a total of 480 2D images for the patient's mouth may be generated at step 806 for a patient with thirty-two teeth, just to give one example.

[0128] Depending on the particular implementation, the 2D raster image generated in step 806 can be used as a comparator when performing other techniques described herein. For example, with respect to tooth segmentation, the segmented tooth mesh (e.g., generated in step 804) can be overlaid on top of the 3D mesh data received in step 802. Then, aspects of the 2D raster image that align with the scan data can be identified. For example, in one implementation, the result of the overlay is a red portion of the geometry corresponding to the segmented tooth mesh and a blue portion of the geometry corresponding to the scan data.

[0129] One advantage of applying a visualization such as that described above is that such visualization allows a human user to identify potential misclassifications of the training data. Additionally, applying what is essentially a binary process to teeth, a binary classification ML model (as described elsewhere in this specification) can be trained to provide accurate predictions. It should be understood that, without loss of generality, each of the 2D and 3D validation examples of the present disclosure can operate under n-class classification, for example, where there are multiple "correct" validation results and multiple "incorrect" validation results.

[0130] In step 808, the system 100 may accumulate or otherwise aggregate 2D views over multiple patient cases. For example, according to one implementation, 60 patient cases may be used. In other words, if 480 2D images are generated for each patient, then in an implementation using 60 patient cases, the training data may include 28,800 different 2D images, as just one example.

[0131] In step 810, the system 100 may train the neural network received in step 803 to validate the accumulated views of one or more cases. For example, when it comes to validating a digitally generated setup for orthodontic alignment treatment, running the fully trained neural network may assign one or more standard scores that specify whether one or more aspects of the received views of the generated setup are correctly formed.

[0132] In step 812, the system 100 outputs both the test results and the resulting neural network. For example, depending on the particular implementation, the output may specify whether the received 3D mesh passes the validation check. If the received 3D mesh does not pass the validation check, the output may also include a correction to the received information describing one or more corrective measures. For example, if the 3D mesh represents a scan of a 3D printed part, the corrective measures may describe how to modify the already manufactured 3D printed part to fit the patient's dental anatomy. Various conditions can be measured or analyzed in this manner. For example, the technology can measure whether the generated settings are correctly formed with respect to measurement criteria for alignment, marginal ridges, buccal and lingual inclinations, occlusal relationships, occlusal contacts, overbite (or overbite), interproximal contacts, and root angulation, to name just a few examples. In other examples, the corrective measures may provide guidance on how to correct the function of the 3D printer (e.g., address a partially blocked nozzle that causes a deformed 3D printed part).

[0133] Although neural networks are used to describe technology 800, one or more steps of technology 800 may also be performed using ML models other than neural networks, such as support vector machines (SVNs), random forests, K-nearest neighbors (KNNs), and other ML models. In order to understand how such other ML models may be used, the data may be divided into two categories of data, "TECH" (class 01) and "RAW" (class 00) data. The TECH class is data obtained by manual intervention by professional technicians. The RAW class is data output from automated tools. TECH class data may generally represent a more correct data set than RAW class data because the TECH class data has been fixed / improved / fine-tuned by expert technicians. The following method is related to a non-neural network method for distinguishing between the TECH class (class 01) and the RAW class (class 00).

[0134] For an effective texture feature-based validation classifier, combining segmentation markers via color with tooth / gum geometry can produce different kinds of artifacts for each class. There are multiple existing texture feature descriptors that can be used as part of texture feature-based validation, including HOG, SURF, SIFT, GLOH, FREAK, and Kadir-Brady. These texture-based validation classifiers can be used by less complex ML models, like some image enhancements can improve the classifiers, such as increasing the contrast between the tooth and gum segmentations so that the feature vectors find more differences around the tooth / gum line when comparing computer- and technician-generated segmentations. Each of the validation applications of the present disclosure may describe a specific implementation involving operations based on texture features.

[0135] For example, verification based on texture features using SIFT classification may include an optional step of converting training images to grayscale, and the steps of finding SIFT keypoints on each image, generating descriptors for these keypoints, selecting only the top N descriptors (where N is the minimum number of descriptors found in all training sample input images), and training a support vector machine (SVM) model on the image descriptors. As an example, other specific implementations may, for example, replace the training of an SVM model on the image descriptors with fitting a k-nearest neighbor (KNN) classifier on the image descriptors.

[0136] That is, while more simplified non-neural network ML models can be used, there are various advantages to using a neural network approach. For example, neural networks can be designed with a sufficiently large number of parameters (i.e., weights) to encode solutions to complex problems, such as understanding 2D raster image views and 3D geometric shapes (i.e., 3D meshes). In addition, texture features may not detect all relevant attributes of an image, such as attributes that indicate defects or errors that the verification process wants to detect.

[0137] Fig. 9 An example technique 900 for validating the output of an ML model in accordance with various aspects of the present disclosure is shown. Generally speaking, and as used herein, "validating" the output of an ML model means automatically checking the quality and correctness of that output, for example, by an ML technique such as a neural network. The term "validation" is used in additional contexts to refer to a smaller data set (perhaps 10% of the total data) that is used to evaluate the progress of an ML model (such as a neural network) as training proceeds. That is, unless specifically referred to as validation data, validating output, as used herein, is generally intended to mean checking the correctness of the output produced by the ML model.

[0138] In general, the method in the depicted example involves an encoder structure (e.g., represented by encoder 911). That is, according to a specific implementation, the generator 211 can be an encoder 911, which can be trained to classify 3D meshes into different categories (i.e., good / bad, or correctly formed / incorrectly formed). The encoder 911 described herein is further configured to output one or more probability vectors (e.g., vector 912). Each element in vector 912 corresponds to a class or category of a label to be applied to the input mesh. The vector element with the highest probability value represents the determination or output of encoder 911. This encoding scheme is called one-hot encoding. During the training of the encoder, each of the training samples (i.e., 3D meshes) has an associated "reference truth" label in the form of a one-hot vector (e.g., vector 914). The vector may contain 1 in the element corresponding to the expected mesh category and 0 in each other element.

[0139] As in Fig. 9 In the present invention, the technique 900 utilizes a receiving module 202 that receives patient case data. According to a particular implementation, the receiving module 202 may receive patient data 204 corresponding to a 3D mesh related to one or more aspects of the dentition of one or more patients. That is, the patient data 204 may vary based on the dental and / or orthodontic treatment to be performed on the patient. According to a particular implementation, the patient data 204 may include one or more dental arches and malocclusions, or arch geometries. Fig. 9 For example, in an implementation regarding verification of bracket placement, the patient data 204 may include only 3D mesh data regarding a particular tooth and associated bracket, while in an implementation regarding verification of a 3D printed part, the patient data 204 may include 3D mesh data associated with a part inspected in the form of a CT scan or other diagnostic image, to name a few additional examples.

[0140] The receiving module 202 also receives ground truth data 208 in the form of one or more vectors 908 corresponding to "ground truth" one-hot vectors. Generally speaking, these "ground truth" vectors 908 specify the expected results of applying other techniques disclosed herein, such as mesh segmentation, coordinate system prediction, mesh cleaning, restoration design, and bracket / attachment placement, as well as all verification applications of the present disclosure, to name a few examples. Depending on the specific implementation, the "ground truth" vectors 908 may be predefined or provided as a result of performing one or more other techniques disclosed herein.

[0141] The result information generated by the mesh feature module 208 and the patient data 204 may be provided to the encoder 911. Also, as described above, the encoder structure 911 is configured to generate one or more predictions represented by the prediction vector 907.

[0142] Once one or more prediction vectors 907 are generated, they are compared to the "ground truth" vector 908. During the neural network training process, a loss value (e.g., loss value G1 216) is calculated for each of the training samples. This loss is generated by a cross entropy calculation of two one-hot vectors: the "ground truth" vector associated with a separate training grid, and the "prediction" vector generated by the encoder 911 in response to the training grid. The loss value is fed back into the encoder 911 to further refine the encoder 911, for example using a back-propagation technique.

[0143] Other training techniques are possible, as are other loss functions. For example, as referenced above Figure 2 As described, the neural network may be trained using one or more generated inference metrics, which may be provided to the neural network after execution of the neural network generates one or more prediction vectors 907 to further refine the weights of the neural network.

[0144] In some implementations, mesh samples used to train a neural network may be replicated, and these replicated meshes may be augmented to increase the supply of training samples and assist in training the neural network. The training process may be aided by exposing the neural network to a wider array of possible inputs. 3D mesh processing operations that may be involved in generating this augmented data set include, but are not limited to: normalization, rotation, translation, and non-uniform scaling. In some implementations, these operations may be applied to the vertices of the mesh to alter the mesh and increase the range of possible meshes that the neural network sees during training. In other implementations, the augmentation step may be applied to the edges or faces of the mesh.

[0145] The authentication technique 900 can be used in various aspects of the present disclosure. For example, Table 3 shows the types of authentication that can be performed using the technique 900 and the patient data 204 received to perform the corresponding authentication.

[0146] For example, with respect to segmentation, the system 100 may be configured to receive a first digital representation of a patient's dentition that has been assigned labels by a neural network consistent with other aspects of the present disclosure. The system 100 may also receive a second digital representation of the patient's teeth that has been assigned predetermined labels. The system 100 may then determine whether the labels on the one or more aspects of the first representation are substantially similar to the labels on the corresponding one or more aspects of the second representation, generate an output describing whether the labels are substantially similar (e.g., in the form of a loss value), and use the process to further train the neural network that provides classification (e.g., using the loss value to train the neural network via backpropagation). Mesh cleaning also involves labeling mesh elements. While segmentation applies labels to mesh elements and copies mesh elements with similar labels to a new mesh (e.g., copying each tooth to its own mesh—a process called tooth cutting), mesh cleaning also applies other operations to the labeled mesh elements. Mesh cleaning may remove all mesh elements with one or more specific labels from a mesh. In other specific implementations, mesh cleaning may perform metric calculations on all mesh elements with a specific label. In yet other implementations, mesh cleaning may apply a transformation operator, such as translation or non-uniform scaling, to all mesh elements having one or more specific tags.

[0147] Similar methods can also be used for other applications. For example, with respect to segmentation verification, the system 100 can be configured to receive one or more dental arches, where the dental arches are 3D meshes, each mesh consisting of elements (such as vertices, edges, and faces), and each mesh element has been assigned a predicted label. The reference true label of each mesh element is also provided to the system 100. The verification neural network can be trained to compare the predicted label with the reference true label. According to one specific implementation, a neural network such as MeshCNN can be used to perform the comparison on the 3D mesh data to classify the mesh. In other specific implementations, an encoder structure can be used to perform the comparison on the 3D mesh data to classify the mesh. In other specific implementations, a neural network (such as a CNN) can be used to perform the comparison to classify 2D raster image views of teeth and gums, wherein the neural network is trained on a first set of images of teeth and gums colored according to the predicted labels on the mesh elements and a second set of images of teeth and gums colored according to the reference true mesh elements. These examples can also be extended to mesh cleanup verification, which also involves training a neural network (2D or 3D) to compare a set of predicted mesh element labels to a set of ground truth mesh element labels. Similar 2D and 3D verification examples are also applicable to applications such as CTA trim line verification, fixture model verification, 3D printed object verification, dental restoration design verification, and bracket and attachment placement verification.

[0148] As another example, with respect to coordinate system verification, the system 100 may be configured to receive one or more first coordinate axes, one or more second coordinate axes, and a representation of a patient's dentition (including but not necessarily limited to a representation of a patient's teeth) to determine whether the first coordinate axis is substantially similar to the second coordinate axis. According to one specific implementation, the comparison may be performed by generating a representation of the teeth that approximates both the first coordinate axis and the second coordinate axis. The representation may be a 2D image, 3D data, and other representations, including combinations thereof.

[0149] As another example, verification can be used to place library components for dental prosthetics or other oral care appliances. The library components come from a fixed set of static designs. The verification neural network checks the position and orientation of the library components relative to the teeth. The input consists of the library component and one or more teeth. Although the generated components can be checked in isolation or relative to one or more teeth, the library components must be checked relative to one or more teeth.

[0150] In some implementations, the output predicted one-hot vector contains two elements, one containing the probability that the input grid receives the predicted "correct" verification result, and the other containing the probability that the input grid receives the predicted "incorrect" verification result. In the "correct" case, the library component is considered to be properly positioned and oriented relative to the associated teeth, and is therefore suitable for the construction of the dental restoration appliance. In the "incorrect" case, the library component placement requires further work, either by a technician or by further iterations of the automated process. Similar verification results also apply to other verification aspects of the present disclosure, depending on the details of those implementations.

[0151] It should also be understood that each of the verification operations described in the present disclosure (and shown in Table 3) shares the possibility of a feedback loop, whereby when a verification result produces a failure result, a notification can be set to the process (e.g., an automated process) that created the geometry to inform the process that the geometry should be recreated. In this way, several iterations of geometry verification and geometry regeneration can be performed to improve the suitability of the geometry for creating a dental or orthodontic appliance. In some specific implementations, an indication can be output by the verification process indicating how to improve the next iteration of geometry creation / regeneration.

[0152] The technology described in the present disclosure can also be modified to implement regression testing on the output of various technologies to verify any changes to various technologies by changes to the neural network or other changes that do not adversely affect the accuracy of the outputs generated by those technologies. For example, each technology such as mesh segmentation, mesh cleaning, coordinate system prediction, restoration design generation, bracket / attachment placement all outputs geometries that can be verified by regression testing or unit testing. Such tests can be run to ensure that the recent source code changes to any source code module required to perform a specific technology have not destroyed or caused important functions in the technology to be incorrect (for example, as defined in source code, object code or other non-transient computer-readable instructions). Ideally, the regression test suite should be run every night to ensure the accuracy of the developed technology. As described above in conjunction with technology 900, this verification can be considered to distinguish between the "pass" geometry generated by the encoder structure 911 and the "fail" geometry generated by it. Generally speaking, the "pass" and "fail" labels involve a level of subjectivity, which may prove to be disadvantageous depending on the specific implementation. On the contrary, regression testing can be used to test the quality of the code, which automatically generates geometries for creating dental or orthodontic appliances.

[0153] In other words, regression testing is used to determine whether recent changes to a system (whether source code, neural network, or input) have a negative impact on the output of the system. For example, a decrease in the predictive quality of a neural network would be considered a negative impact. In the present disclosure, it is necessary to be able to change several lines of automation code and quickly determine whether those changes have any adverse effects on the output of the test cases in our test suite. There may be dozens of test cases. The output of dozens of test cases can be checked manually, but this requires technicians or other personnel to manually check the output of all test cases, which takes a lot of time. The advantage of this specific implementation is that the process is streamlined. Even if 1 test case out of 36 test cases fails to produce acceptable results after the code is changed, the error will be detected using a set of regression tests and the user of the system 100 will be warned accordingly.

[0154] Fig.10 Specifically, the present invention relates to processes and techniques related to tooth segmentation. Generally speaking, tooth segmentation involves converting a scan of a patient's dentition into a 3D representation that contains individualized components of the patient's mouth (e.g., each tooth and associated gingival tissue). The segmented 3D representation can then be used to solve other technical problems described herein, such as generating clear tray-type aligners (to give an example), as well as other technical problems not specifically mentioned herein.

[0155] Therefore, tooth segmentation typically first involves generating an intraoral scan of the patient's dentition. This scan produces a continuous (or homogenous) 3D mesh that covers all relevant teeth and portions of the patient's gingiva as a single 3D representation. In addition, depending on the particular implementation, the patient's upper and lower dental arches are scanned separately and each produces a 3D mesh of the entire dental arch. Because the "raw" scan data (including all scanned teeth and portions of the gingiva) is generally not considered to be that useful in terms of segmented 3D mesh data, automatic tooth segmentation techniques can be used to generate 3D mesh data that, for example, describes individual teeth in the patient's mouth. In general, what is described throughout this disclosure is the use of such segmented 3D data.

[0156] For some implementations, individual teeth may be segmented, resulting in a labeled mesh for each tooth. Other implementations may require segmentation along the gingiva, followed by defining an offset into the gingiva, in order to remove excess mesh material. Other implementations may require segmentation to define an offset trim line into the gingiva, in order to remove excess mesh material. As an example, other implementations may require performing a facial-lingual segmentation, thereby separating the anterior from the posterior portion of the tooth, in order to assist in calculating a mold parting surface (i.e., a generated component used in the production of a dental prosthetic appliance).

[0157] Fig.10 An example technique 1000 for performing mesh segmentation using a trained ML model is illustrated. This implementation uses a U-Net architecture, but other implementations are possible, such as using a MeshCNN. Depending on the particular implementation, the ML model may be a neural network, or another ML model where appropriate. As shown in the depicted example, the technique 1000 may also utilize a receiving module 202 to receive patient data 204, which may include mesh data 1004. In one implementation, the mesh data 1004 may include one or more of the following: 1) one or more segmented entire (or complete) dental arches of a patient, including gingiva; 2) one or more segmented portions of a patient's dental arches, including gingiva; 3) one or more individual segmented teeth of a patient, with or without gingiva. This data is collectively referred to herein as one or more segmented dental arches of a patient's dentition.

[0158] Technique 1000 also utilizes modules from technique 200, including mesh preprocessor 205 and mesh feature module 208. As shown in other techniques, technique 1000 uses U-Net architecture 1011 as a generator, instead of using an encoder structure as a generator, which U-Net architecture may include a neural network to generate prediction output 207, such as one or more prediction labels 1007. In some specific implementations, when 1011 is a U-Net architecture and 1007 is a list of mesh element labels, technique 1000 can be used for mesh segmentation. That is, U-Net architecture 1011 can also be replaced with an encoder structure or other ML models (including neural networks such as MeshCNN and other neural networks). In some specific implementations, the prediction label 1007 can be defined as a one-hot vector. In some specific implementations, when 1011 is an encoder structure and 1007 is a one-hot vector of probabilities, technique 1000 can be used for 3D verification of mesh segmentation operations. In some embodiments, when 1011 is a CNN and 1007 is a one-hot vector of probabilities, technique 1000 can be used for 2D verification of mesh segmentation operations. These embodiments for 3D verification and 2D verification of mesh segmentation are also applicable to other verification examples, such as mesh cleaning verification, coordinate system verification, dental restoration verification, 3D printed part verification, fixed body model verification, CTA trim line verification, dental restoration appliance assembly verification, and verification of the placement of brackets and accessories for orthodontic treatment. For example, according to one embodiment, the output predicted one-hot vector contains two elements, one containing the probability that the input mesh receives the predicted "correct" label, and the other containing the probability that the input mesh receives the predicted "incorrect" label. In one example, the one-hot vector output from the encoder may have the following form: [probability correct, probability incorrect]. Therefore, if the actual vector generated by the encoder is [0.89, 0.11], the meaning of the vector is that the input mesh is correct. In the "correct" case, the mesh segmentation operation is considered successful, and the teeth are accurately separated from the gingiva and from each other to support the operation of producing a dental or orthodontic appliance. In the "incorrect" case, the teeth are not accurately separated from the gingiva, and further work or correction may need to be completed by a technician or by further iterations of the automatic process that originally generated the geometry (e.g., the tooth segmentation algorithm described herein).

[0159] To accommodate subsequent validation iterations, in some implementations, the U-Net is further trained based on the validation results. Additionally, in some implementations, the ML model can check the mesh segmentation work that has been done for each individual tooth, thereby generating localized feedback on the segmentation quality on a tooth-by-tooth basis. Fig.10The example segmentation shown in is considered to be well-formed. That is, the teeth are accurately separated from the gums and from each other. Therefore, if the system 100 were to receive similar mesh data 1004, application of the encoder U-Net architecture 1007 would produce a predicted "pass" or "correct" label. However, if there are a sufficient number of errors in the segmentation results, the system 100 may cause the technique 1000 to be performed one or more times until the accuracy of the U-Net is sufficiently improved so that the U-Net is able to generate a "correct" output.

[0160] Fig.11 An example technique 1100 is shown for validating output from an ML model, such as output from a mesh segmentation operation, or output from another operation involving applying labels to mesh elements, such as mesh cleaning, to name a few examples. Technique 1100 may be performed by system 100. For example, system 100 may perform steps of receiving a digital 3D model of a patient's dentition and one or more neural network parameters and a trained deep learning model. Using pre-processing module 205, system 100 may perform a number of optional steps, including mesh cleaning, generating element labels, and resampling and updating these labels, to name a few examples.

[0161] The system 100 may also run a deep learning model to generate a proposed segmentation as described above. The system 100 may also be configured to perform one or more post-processing steps 1102 on the trained model, such as extracting mesh regions and mesh boundaries. This may produce a final set of 3D meshes that may be analyzed. For example, at step 1104, the system 100 may generate one or more inference evaluation metrics, as described above. These inference evaluation metrics allow the system 100 to generate one or more reports detailing the correctness of the final mesh.

[0162] Fig.11An example general technique 1800 for validating outputs generated by an ML model according to various aspects of the present disclosure is shown. The validation ML model may be trained to process the following non-limiting list of 3D representations: 1) mesh element labels for segmentation or mesh cleaning; 2) coordinate system axes of teeth (e.g., as encoded by transforms); 3) dental restoration design; orthodontic settings; 4) custom lingual brackets; 5) bonding pads for brackets (which can be generated for specific teeth by outlining the perimeter on the teeth, specifying the thickness to form a shell, and then subtracting the teeth through Boolean operations); 6) clear tray appliances (CTAs); 7) location or shape of trim lines (e.g., CTA trim lines); 8) shape or structure or posture of attachments; 9) occlusal ramps or slits; 10) 3D printed appliances (local thickness, rib geometry, flap positioning, etc.); 11) 3D models of the patient's teeth and gums showing trim lines (e.g., fixture models), data or structures related to implant placement; 12) hardware placement; 13) other types of dental restoration designs (e.g., veneers, crowns, or bridges); 14) and other 3D printed parts that are part of the oral care routine or other areas.

[0163] The technique 1100 may use the steps of receiving a 3D mesh of one or more teeth and additional optional data related to a dental procedure. This information may be provided to one or more anomaly detection networks for verification. In some implementations, this may include generating one or more 2D raster views of the 3D mesh. Next, the system 100 may use a neural network to analyze each aspect of the 2D and / or 3D representation to present a pass / fail determination regarding those aspects. If a sufficient number of aspects receive a passing accuracy score, the representation is considered to have passed, at which point the system 100 may provide geometry for use in other dental procedures. If a sufficient number of aspects do not receive a passing accuracy score, the system 100 may generate information about why one or more aspects of the representation failed, and in some implementations, automatically train one or more neural networks based on the results, and then perform the method 1100 again with additional training of the neural network to see if a passing score can be achieved. In various implementations, this 2D verification method may be applied to each of the various verification applications described in the present disclosure.

[0164] The technique 1100 can be performed in near real time, allowing dental professionals and other professionals to perform scans and other dental procedures while the patient is in the chair, thereby improving the effectiveness of dental treatment and providing a more pleasant experience for the patient. For example, the verification method can be applied to the patient's intraoral scan data immediately after the intraoral scan is performed. The advantage is that if there is a problem with the scan data, the dentist can be notified, and if a rescan is necessary, the patient can do so (and in fact, without even leaving the chair). Detected mesh errors include holes in the mesh, incompletely scanned teeth, missing teeth, foreign matter obscuring teeth, and / or upper / lower dental arches being incorrectly identified / interchanged. The results of the verification can be displayed to the dentist (or technician) using one or more heat maps, possibly superimposed on the dental model. Problematic areas of the mesh can be highlighted in a patchwork manner using different color coding. The disclosure regarding mesh cleaning describes mesh defects detected during the mesh cleaning verification process. The application of this near real-time method can also benefit from performing checks to detect these conditions, so that the intraoral scan can be re-performed under different conditions (e.g., more careful techniques employed by the technician or doctor). In this case, the need for subsequent mesh cleanup operations can be reduced or eliminated.

[0165] Specific errors or flaws in the scan are highlighted using color, bounding boxes, arrows, or other graphical elements and displayed to the dentist / technician. For example, if the verification engine determines that a portion of a tooth is missing from the mesh, a bounding box may be drawn onto a visualization of the mesh over the area of ​​the missing or incomplete tooth. A text report on the quality of the scan may be prepared and sent via SMS, email, or other electronic means, or displayed to the dentist / technician in the dentist's office. In some cases, there may be an LCD display located near the scanner that displays the verification report to the dentist. As another example, the verification engine may apply a parting surface to the tooth, resulting in each edge / vertex / face element in the tooth mesh being labeled as A) facial or B) lingual: 1) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the facial direction (e.g., 1.0 mm or 0.5 mm); 2) a facial portion of a tooth where the parting surface is correct; 3) a facial portion of a tooth where the parting surface used to cut the tooth is positioned too far in the lingual direction (e.g., 1.0 mm or 0.5 mm). Depending on the particular implementation, there may be more than one label. For example, some implementations may use both element labels and result labels. The element label describes whether the edge / vertex / face element is on the facial side of the tooth mesh or on the lingual side of the tooth mesh. The result label indicates whether the parting surface near the tooth is 1) too facial, 2) correct, or 3) too lingual, to name just one example.

[0166] According to the technology of the present disclosure, an ML model can be trained on examples of 3D oral care representations, wherein benchmark real data is provided to the ML model and a loss function is used to quantify the difference between the predicted examples and the benchmark real examples. The loss value can then be used to update the validation ML model (e.g., update the weights of a neural network). Such validation techniques can determine whether a trial 3D oral care representation is acceptable or suitable for creating oral care appliances. In some cases, "acceptable" may mean that the trial 3D oral care representation conforms to the distribution of benchmark real examples used in training the ML validation model. In some cases, "acceptable" may mean that the trial 3D oral care representation is correctly shaped or correctly positioned relative to one or more aspects of the dental anatomy.

[0167] In the example of a generated appliance component (e.g., a dental restoration appliance component, such as a mold parting surface), the technology can determine whether the component intersects the correct landmarks or other parts of the tooth anatomy (e.g., incisal edges and cusps for mold parting surfaces). The technology can also determine one or more of the following: 1) whether the CTA trim line intersects the gingiva in a manner that reflects the distribution of the benchmark truth; 2) whether the library component is correctly placed relative to one or more target teeth (e.g., a snap clip placed relative to posterior teeth or a center clip placed relative to incisors), or correctly placed relative to one or more landmarks on the target teeth; 3) whether the hardware elements are placed on the face of the tooth with edges that reflect the distribution of the benchmark truth examples; 4) whether the mesh element labels used for segmentation (or mesh cleaning) operations conform to the label distribution in the benchmark truth examples; and 5) whether the shape and / or structure of the dental restoration tooth design conforms to the distribution of tooth designs in the benchmark truth training examples, to name a few examples. Other verification conditions and / or rules are possible for verification of various 3D oral care representations.

[0168] Fig.12An example technique 2200 for training an ML model (e.g., classifying a 3D mesh for the purpose of 3D mesh or point cloud verification) is shown. The verification systems and techniques of the present disclosure may assign one or more labels to one or more aspects of a representation to be verified (e.g., correctly oriented or placed, incorrectly oriented or placed, etc.). The verification systems and techniques of the present disclosure may benefit from the computation of mesh element features. 3D oral care mesh verification may be applied to segmentation, mesh cleaning, coordinate system prediction, dental restoration design, CTA setup verification, CTA trim line verification, fixture model verification, archform verification, orthodontic hardware placement verification, appliance component placement verification, 3D printed part verification, chairside scan verification, and other verification techniques described herein. In the event that a 3D verification check produces a failed output, then one or more instructions or feedback data may be transmitted to an algorithm, process, or model that creates a 3D oral care representation, such that further iterations of 3D oral care representation generation may improve the design and hopefully mitigate the condition that caused the verification check to fail. In some implementations, a neural network trained to classify a 3D mesh (or point cloud) for verification may take mesh element features as input (e.g., a mesh element feature vector may be calculated for one or more mesh elements in the mesh or point cloud to be verified). In some cases, a mesh element feature vector may accompany each mesh element as input to the verification neural network. In some cases, the verification neural network may form a reformatted (or sometimes reduced dimension) representation of the input mesh or point cloud. The mesh element features may improve this reformatted (or reduced dimension) representation by providing additional information about the shape and / or structure of the input mesh or point cloud. By using the mesh element features, the precision and accuracy of the resulting verified data is improved.

[0169] Figures 13 to 16 are example results generated based on executing various aspects of the present disclosure. For example, Fig.13 Both predicted and reference coordinate axes are illustrated, which can be used to train one or more ML models (e.g., neural networks such as CNNs) to perform validation on the predicted coordinate system for use in various aspects of the present disclosure.

[0170] about Fig.14 , presents a visualization of a set of coordinate system axes associated with tooth LL2. The reference true coordinate system is generated by a skilled technician. The predicted coordinate system is generated using the techniques of the present disclosure. The "combined" figure shows both the reference true and predicted coordinate systems side by side.

[0171] In some cases, a local coordinate system for a 3D oral care representation (such as teeth) may be described by one or more transformations (e.g., an affine transformation matrix, a translation vector, or a quaternion). The system of the present disclosure may be trained for coordinate system prediction using past cohort patient case data. The past patient data may include at least: one or more tooth meshes or one or more reference tooth coordinate systems. An ML model (such as: a U-Net, an encoder, an autoencoder, a pyramid encoder-decoder, a transformer, or another architecture with convolution and pooling layers) may be trained for coordinate system prediction. Representation learning may determine a representation of the teeth (e.g., converting a mesh or point cloud to a potential representation, e.g., using a U-Net, an encoder, a transformer, or another architecture with convolution and pooling layers, etc.), and then using a coordinate system prediction neural network to predict a transformation for the representation (e.g., using a trained multilayer perceptron, transformer, encoder, transformer, etc.), which defines a local coordinate system for the representation (e.g., including one or more coordinate axes). In the case of predicting a coordinate system for a tooth mesh. A mesh element feature vector may be calculated for one or more of the mesh elements of a 3D oral care representation (such as a crown mesh) (using the mesh element features described herein). Such mesh element features may improve the representation of teeth generated in the context of representation learning (e.g., may improve the understanding of the structure and / or shape of the tooth mesh). In some implementations, the mesh element features may also be directly input into a coordinate system prediction neural network along with the tooth representation. The coordinate system prediction neural network may be trained at least in part using transfer learning. The trained coordinate system prediction neural network may also be used in turn as a basis for training another neural network (e.g., such as a setting prediction neural network) using transfer learning. The coordinate system prediction techniques described herein may predict two or more coordinate axes simultaneously. In some implementations of coordinate system prediction, two initial vectors may be predicted simultaneously, and then the x, y, and z orthogonal axes may be calculated from the two initial vectors (e.g., using a Gram Schmidt process). The techniques disclosed herein may be trained to predict coordinate systems for tooth meshes, tooth point clouds, or other representations of teeth.

[0172] about Fig.15 , presents a visualization of a set of coordinate system axes associated with tooth LR5. The reference true coordinate system was generated by a skilled technician. The predicted coordinate system was generated using the techniques of the present disclosure. The "combined" figure shows both the reference true and predicted coordinate systems side by side.

[0173] about Fig.16 , presents a visualization of a set of coordinate system axes associated with tooth LR7. The reference true coordinate system was generated by a skilled technician. The predicted coordinate system was generated using the techniques of the present disclosure. The "combined" figure shows both the reference true and predicted coordinate systems side by side.

[0174] Various aspects of the present disclosure may be used for different purposes in one or more digital dental fields, including segmentation, coordinate systems, mesh cleaning, setting of clear tray-type aligners, dental prosthesis, brackets and accessories, 3D printed parts, prosthetic designs, and fixture models. These fields may involve the generation of one or more (2D or 3D) representations and the verification of one or more (2D or 3D) representations. One or more of these fields may be combined, for example, certain techniques may combine the following concepts: 1) segmentation, 2) calculation of the geometry of dental prosthesis, and 3) mesh verification. For example, the results of face-tongue segmentation may be used by an algorithm for generating a mold parting surface, with the goal of improving the resulting mold parting surface (i.e., relative to the mold parting surface that would be generated without the benefit of the previous face-tongue segmentation). The resulting mold parting surface may then be checked by a verification module (i.e., using 2D or 3D processing). If the verification module determines that the generated mold parting surface is poor, the algorithm that generated the mold parting surface can be rerun, potentially using actionable feedback from the verification engine (e.g., tips on how to adjust the mold parting surface on a tooth-by-tooth basis, whether the parting surface should move in the facial or lingual direction near each tooth). If the verification module determines that the generated mold parting surface is acceptable, the mold parting surface is output.

[0175] Although this specification sets forth many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular implementation. Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations, even when initially claimed, in some cases one or more features from a claimed combination may be deleted from that combination, and a claimed combination may refer to a subcombination or a variation of a subcombination.

[0176] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described components and systems may generally be integrated together in a single system or distributed in multiple systems.

Claims

1. A computer-implemented method for training one or more neural networks to automatically generate a coordinate system for use in digital oral care, the method comprising: receiving, by one or more computer processors, a first digital 3D oral care representation of a patient's teeth; receiving, by the one or more computer processors, one or more reference coordinate axes proximate to one or more teeth in the first 3D oral care representation; using, by the one or more computer processors, a first configuration of one or more neural networks that have been initially trained to generate a modified representation of the first digital 3D oral care representation; using, by the one or more computer processors, a second configuration of one or more neural networks that have been initially trained to predict information about one or more coordinate axes, and wherein the second configuration receives as input the modified representation generated by the first configuration; automatically training, by the one or more computer processors, the second configuration based on using the second configuration, wherein the training of the second configuration is modified by performing operations comprising: predicting, by the second configuration, one or more predicted one or more direction vectors about the one or more coordinate axes; computing, by the one or more computer processors, one or more prediction transforms based on the one or more direction vectors; determining, by the one or more computer processors, a loss value specifying a difference between the one or more predicted transforms and one or more corresponding reference transforms; and At least one aspect of the one or more neural networks included in the second configuration is modified based on the loss value.

2. The computer-implemented method of claim 1, wherein the first number specifies at least one of the patient's dental arches and further includes data corresponding to one or more segmented teeth in at least one of the patient's dental arches. 3 . The computer-implemented method of claim 1 , wherein at least one of the first configuration and the second configuration is initially trained using a historical digital representation comprising one or more coordinate axes.

4. The computer-implemented method of claim 1 , wherein the first representation comprises one or more mesh elements, and the method further comprises: determining a grid element feature vector for at least one of the grid elements; providing, by the one or more computer processors, the grid element feature vector as input to the first configuration; and The modified representation is affected based on the mesh element feature vector.

5. The computer-implemented method of claim 1, wherein at least one neural network of either the first configuration or the second configuration is trained, at least in part, using transfer learning.

6. The computer-implemented method of claim 1, wherein at least one neural network in either the first configuration or the second configuration is used to at least partially train another neural network using transfer learning.

7. The computer-implemented method of claim 4, wherein the mesh element feature vector comprises at least one spatial mesh element feature or at least one structural mesh element feature.

8. The computer-implemented method of claim 7, wherein the mesh elements include at least one of one or more vertices, one or more edges, one or more faces, one or more points of a point cloud, and one or more voxels of the first representation.

9. The computer-implemented method of claim 8, wherein the information related to vertex mesh element characteristics includes at least one or more of an XYZ position or a normal vector.

10. The computer-implemented method of claim 9, wherein the normal vector is a weighted average of normal vectors of at least connected faces of the corresponding vertex.

11. The computer-implemented method of claim 8, wherein the information related to the face mesh element includes at least one or more of an XYZ position of a face centroid, a face area, or a normal vector.

12. The computer-implemented method of claim 8, wherein the information related to the edge mesh element includes at least one or more of an XYZ position of an edge midpoint, an edge length, or a normal vector.

13. The computer-implemented method of claim 12, wherein the normal vector is an average of normal vectors of at least two vertices.

14. A computer implementation according to claim 1, wherein one or more of the loss values ​​forming the basis for the modification are selected from one or more of binary cross entropy loss, mean squared error, L1 loss and L2 loss.

15. The computer-implemented method of claim 1, wherein the one or more coordinate axes are automatically generated in real-time while the patient is present in the clinical setting.

16. The computer-implemented method of claim 1, wherein the first digital representation describes at least one of one or more teeth, gum tissue, and a dental or orthodontic appliance within a patient's mouth.

17. The computer-implemented method of claim 1, wherein the predicted information comprises at least one of one or more transforms or one or more vectors convertible into a transform.

18. The computer-implemented method of claim 17, wherein at least one of two or more direction vectors or one or more position vectors is generated by the second configuration.

19. The computer-implemented method of claim 18, wherein the one or more computer processors use the direction vector or position vector as input to generate at least one of three or more coordinate axes or an origin of the coordinate system.

20. A system, comprising: one or more computer processors; a non-transitory computer-readable storage device having stored thereon first and second configurations of one or more neural networks and instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first digital 3D oral care representation of a patient's teeth; receiving one or more coordinate axes proximate one or more teeth in the first 3D oral care representation; using a first configuration of one or more neural networks that have been initially trained to generate a modified representation of the first digital 3D oral care representation; using a second configuration of one or more neural networks that have been initially trained to predict information about the one or more coordinate axes, and wherein the second configuration receives as input the modified representation generated by the first configuration; Automatically training the second configuration based on using the second configuration, wherein the training of the second configuration is modified by performing operations comprising: predicting one or more prediction transforms about the one or more coordinate axes using the second configuration; determining a loss value specifying a difference between the one or more prediction transforms and one or more corresponding reference transforms; and At least one aspect of the one or more neural networks included in the second configuration is modified based on the loss value.

Images