Machine learning models for the prediction of data structures pertaining to interproximal reduction

Abstract

Systems and methods are disclosed for generating data structures related to interproximal reduction (IPR) in orthodontic treatment using machine learning (ML) models. The systems and methods involve receiving a digital representation of a patient's dentition and providing the digital representation to a partially trained ML model. The partially trained ML model, which includes a first and second module, generates predictions for one or more IPR cut surfaces based on the patient's dentition. The partially trained ML model is further trained by generating predicted IPR cut surfaces, quantifying the difference between the predicted surfaces and corresponding reference surfaces, generating a loss value based on the difference, and modifying the ML model based on the loss value to create a modified ML model. These systems and methods aim to improve the accuracy and efficiency of generating IPR cut surfaces in orthodontic treatment using machine learning techniques.

Description

MACHINE LEARNING MODELS FOR THE PREDICTION OF DATA STRUCTURES PERTAINING TO INTERPROXIMAL REDUCTIONRelated Documents

[0001] The entire disclosure of PCT Application No. PCT / IB2022 / 057373 is incorporated herein by reference. The entire disclosures of each of PCT Applications with Publication Nos. WO2022123402A1, WO2021245480A1, and W02020026117A1 are incorporated herein by reference. The entire disclosure of each of the following Provisional U.S. Patent Applications is incorporated herein by reference: 63 / 432,627; 63 / 366,492; 63 / 366,495; 63 / 352,850; 63 / 366,490; 63 / 366,494; 63 / 370,160; 63 / 366,507; 63 / 352,877; 63 / 366,514; 63 / 366,498; 63 / 366,514; and 63 / 264,914.Technical Field

[0002] This disclosure relates to configurations and training of machine learning (ML) models to improve the accuracy of automatically generated clear tray aligner (CTA) devices used in orthodontic treatments.Summary

[0003] Some existing techniques have attempted to use machine learning to generate the CTA devices, but with mixed results. As a result, there is a need for better machine learning models and training approaches to improve the systems that automate the production of CTAs. Techniques of this disclosure may predict orthodontic setups. These setups improve upon prior techniques in that the techniques not only predict setups transforms for the patient’s teeth, but also predict information pertaining to interproximal reduction (IPR) operations on one or more of the patient’s teeth. The techniques may generate IPR cut surfaces for one or more teeth of the patient. The techniques may predict a magnitude of IPR to be applied to a tooth (e.g., to either the mesial or distal sides of the tooth). The techniques may predict whether one or more teeth in a setup will benefit from IPR (e.g., whether or not IPR should be performed). The techniques may predict one or more stages at which IPR is to be applied for one or more teeth. The techniques may generate an indication of whether one or more teeth are accessible to IPR (e.g., either using oral care metrics, or a trained ML model). Additionally, the techniques may generate orthodontic setups transforms for one or more teeth of the patient’s dentition and may at the same time generate predictions for any of: IPR cut surfaces, IPR magnitude to apply to a side of a tooth, an indication of whether IPR should be performed, an indication of which stage at which IPR is to be performed, or an indication of IPR accessibility.

[0004] Methods and systems are described for generating information pertaining to interproximal reduction (IPR) in orthodontic treatment, through the utilization of machine learning (ML) models. One or more 2D or3D digital representations of a patient's dentition may be provided to a trained ML model that includes both a first ML module for encoding the digital representation into a lower-dimensional latent representation and a second ML module trained to predict data structures pertaining to IPR (e.g., IPR cut surfaces, etc.).

[0005] The digital representation may be provided to a partially trained ML model, which may generate predictions for data structures pertaining to IPR. The partially trained model may undergo further training based on the outcomes of these predictions. The partially trained ML model may compute loss values which quantify the differences between the predicted data structures and reference data structures. The loss may be used to train, at least in part, the partially trained ML model.

[0006] Spatial and / or structural mesh element feature vectors may be computed for one or more of the mesh elements in 3D representations of the patient dentitions. The mesh element feature vectors may be provided to ML models of this disclosure, including the first ML module, to improve the reconstruction accuracy of the resulting latent representations.

[0007] Techniques of this disclosure may determine the accessibility of teeth for IPR based on the structure of the dentition, oral care metrics (e.g., including collisions between teeth), predicted cut surfaces, or other data. These data may inform whether IPR should be performed on specific teeth.

[0008] The processing capabilities may be deployed in a clinical context, enabling near real-time outputs for clinicians, including generating orthodontic setups and generating designs for oral care appliances based on the generated data. For example, intraoral scanning may be performed to generate a digital representation of the patient’s dentition. Subsequent automated operations can include: mesh segmentation, mesh cleanup, automated setups prediction, and / or automated prediction of data structures pertaining to IPR (e.g., predictions of which teeth should receive IPR, and / or corresponding IPR cut surfaces, etc.). IPR can then be performed while the patient waits in the treatment chair, and the intraoral scan can be repeated, yielding improved dentition data. The resulting improved dentition data can enable subsequent processing to be performed with greater accuracy. For example, subsequent automated setups prediction may generate setups which yield better tooth alignments when the patient’s teeth start out with shapes that are less pathological.

[0009] The techniques also encompass computing devices configured to carry out the described methods and systems comprising processors and storage for executing the necessary instructions. The methods enhance the efficiency and accuracy of orthodontic treatments.Brief Description of Drawings

[0010] FIG. 1 shows an example of a patient’s malocclusion dentition.

[0005] FIG. 2 shows IPR cut planes and IPR cut surfaces in the context of the patient’s malocclusion dentition.

[0006] FIG. 3 shows a method of training an ML model to generate IPR cut surface predictions for the orthodontic treatment of the patient.

[0007] FIG. 4 shows a method of training a setups prediction ML model which is configured to render one or more predictions of IPR information (e.g., configured to generate IPR cut surface predictions alongside generating the tooth transforms for an orthodontic setup).

[0008] FIG. 5 shows a method of training an ML model to render predictions of IPR accessibility for one or more teeth of the patient’s dentition.

[0009] FIG. 6 shows a method of using a fully trained ML model to generate IPR cut surface predictions for the orthodontic treatment of the patient.

[0010] FIG. 7 shows a method of using a fully trained setups prediction ML model which is configured to render one or more predictions of IPR information.

[0011] FIG. 8 shows a method of using a fully trained ML model to render predictions of IPR accessibility for one or more teeth of the patient’s dentition.

[0012] FIG. 9 shows a method of using one or more oral care metrics to determine IPR accessibility.

[0013] FIG. 10 shows a method of using a trained ML model to determine IPR accessibility.

[0014] FIG. 11 shows a fully trained reconstruction autoencoder which has been trained to reconstruct a particular tooth type.

[0015] FIG. 12 shows a collision plane and an archform which may be used to compute an oral care metric for IPR accessibility.

[0016] FIG. 13 shows IPR cut planes for two adjacent teeth.

[0017] FIG. 14 shows a method for real or near-time IPR prediction.Detailed Description

[0018] In orthodontics, interproximal reduction (IPR) may be used to remove enamel from the mesial side of a tooth, or from the distal side of a tooth. IPR plays an important role in orthodontic treatment planning, for example, as a part of automated setups prediction. IPR may be indicated for use in automated setups prediction, for example, to make space in the arch into which teeth can be moved. IPR may be applied to teeth in one or more intermediate stages of orthodontic treatment, or to teeth in the malocclusion. Techniques of this disclosure may train an ML model to predict one or more stages at which IPR should be applied (e.g., based on the distribution of past patient case data). Oral care arguments (e.g., maximum number of stages which are permitted to have IPR, among others) may be provided to such an ML model. Techniques of this disclosure may train an ML model to generate an indication of whether a tooth (or an interproximal space between teeth) is accessible to IPR. IPR may be applied to either the mesial or distal surfaces of a tooth. When neighboring teeth make contact along their respective mesial and distal surfaces, or are otherwise in alignment, then the teeth (or interproximal spaces) may be considered accessible to IPR. When neighboring teeth are misalignedor are otherwise sufficiently maloccluded that the first tooth contacts the facial surface or the lingual surface of the second tooth, then the teeth (or interproximal spaces) may be considered inaccessible for the purposes of IPR. In some implementations, techniques of this disclosure may compute oral care metrics and compare those oral care metrics values to thresholds (e.g., thresholds computed according to the distribution of the ground truth data from past patient cases) to determine IPR accessibility. In some implementations, techniques of this disclosure may train an ML model to determine IPR accessibility. Techniques of this disclosure may train an ML model to predict an IPR cut surface (e.g., an IPR cut plane) which may intersect the 3D representation of a tooth and define a volume of enamel for removal. In some instances, the intended IPR cut planes for adjacent teeth may be substantially parallel.

[0019] This disclosure pertains to digital oral care, which encompasses the fields of digital dentistry and digital orthodontics. This disclosure generally describes methods of processing three-dimensional (3D) representations of oral care data, and / or associated transforms. Data structures pertaining to IPR may be generated by methods of this disclosure and used in digital orthodontic treatment planning. In some implementations, the methods may also generate orthodontic setups transforms. A final setup, an intermediate stage or combinations or sequences thereof may be used in the design and manufacture of orthodontic appliances, such as clear tray aligners (CTAs). A fixture model may be generated from a setup (or a stage), which may be 3D printed. A clear plastic tray may be thermoformed onto such a fixture model. The thermoformed tray is cut away from the fixture model (e.g., by following a CTA trimline), thereby completing the aligner tray. In some instances, a digital fixture model may be used to create a 3D oral care representation of an aligner tray, which may then be directly 3D printed.

[0020] Techniques of this disclosure may predict one or more data structures pertaining to IPR. Such a data structure may be associated with a particular tooth of a patient’s dentition. Examples include: IPR cut surfaces (e.g., a tuple describing 3 points, or a vector and a point), a magnitude of IPR (e.g., a real value and associated designation of either mesial or distal), a stage designation for when IPR is to be performed (e.g., an integer), a flag indicating whether or not IPR is to be performed for the tooth (e.g., a Boolean value), a flag indicating whether the tooth is accessible to IPR (e.g., Boolean value), or others described herein.

[0021] Techniques of this disclosure can compute oral care metrics to generate an indication of whether a tooth (or an interproximal space between teeth) is accessible for the purposes of IPR. In some implementations, a collision plane 1200 may be fitted to the points of closest approach (e.g., as computed across a set of horizonal or vertical slices of the tooth meshes) between two (or more) adjacent teeth 1204 and 1206, as shown in FIG. 12. An example of an oral care metric can include the contact distance between the two adjacent teeth, among others. This oral care metric value (or other oral care metric values described herein) can be compared against one or more pre-computed thresholds (e.g., that were computed based on past patient data). If the oral care metric value falls outside of accepted thresholds, then the interproximal space is deemed to be inaccessible to IPR. Alternatively, one or both teeth are deemed to be inaccessible to IPR. Insome implementations, an oral care metric may be computed to measure the angle of the collision plane 1200 relative to a perpendicular plane that is tangent to an archform 1202. When this angle is not within a precomputed threshold, the interproximal space (or either tooth) is considered inaccessible to IPR.

[0022] In some implementations, a precomputed (or predicted) IPR cut plane 1300 or 1302 can be fitted to either side of the interproximal space, and an oral care metric can be defined which measures the angle defined by the collision direction relative to the archform (the angle of the collision plane), or the difference in angles between (either of) the IPR cut planes and the collision plane. If any of these angles are not within a certain threshold, the interproximal space (or either tooth) is considered inaccessible to IPR. In some implementations, a combination of a collision test and an angle test can enable the simulation of a "virtual IPR tool" being used. If there is not enough room between the teeth (e.g., between teeth 1304 and 1306 in FIG. 13) for the virtual tool to fit, or if the angle is too steep on either side, the method may indicate that a real IPR tool could not be used on the real teeth represented by our digital model. It should be appreciated by one of ordinary skill in the art that the process of performing these IPR determinations is impracticable for a human to accurately perform, let alone to perform within the time constraints described by this disclosure. There are many reasons for this, including the complexity of the 3D mesh data that describes the teeth, which may contain thousands or millions of mesh elements, as well as the sheer number of computations which must be performed to arrive at an accurate result.

[0023] In some implementations, the thresholds can be computed according to the distribution of a standard dataset which contains indications of accessibility by experts. In some implementations, these thresholds may be customized to a particular tooth type, a particular tooth position in the order of the arch, among other criteria.

[0024] Instead of using a threshold-based method for each of these metrics for determining accessibility (collision distance, collision plane angle, or IPR plane angle difference), some implementations may use a continuous measurement. For example, each oral care metric may be evaluated on a scale from “not accessible” to “most accessible.” By using this method of continuous measurement and weighting each metric, a real-valued accessibility score can be computed for each interproximal space during each intermediate stage or in the malocclusion.

[0025] Additional criteria for determining accessibility may be considered, including the probability that an interproximal space is accessible. Such a probability can be calculated based on data including tooth shape, tooth position, treatment duration (e.g., count of stages, etc.), or treatment plan. For example, an interproximal space may have a lower probability of being IPR accessible when the following are true: 1) an oral care metric is evaluated as “less accessible” on a scale from “not accessible” to “most accessible,” 2) the interproximal space which is evaluated occurs near the end of a treatment plan, and 3) the interproximal space is evaluated as "not accessible" earlier in the treatment plan. For example, an interproximal space may have a higher probability of being IPR accessible when the following are true: 1) an oral care metric is evaluated as "moreaccessible" on a scale from “not accessible” to “most accessible,” 2) the interproximal space which is evaluated occurs near the beginning of a treatment plan, and 3) the interproximal space is evaluated as “more accessible” earlier in the treatment plan.

[0026] Any of the aforementioned oral care metrics may, according to various implementations, be provided to the ML models of this disclosure (e.g., an ML model for predicting IPR accessibility, etc.), to improve the predictive accuracy of those ML models. Furthermore, one or more collision planes, one or more archforms, or one or more IPR cut planes (or surfaces) may be provided as inputs to the ML models (e.g., an ML model for predicting IPR accessibility), to provide those ML models with refined information about the shape of the patient’s dentition and / or the layout the patient’s teeth. For example, when an ML model of this disclosure is used to determine the IPR accessibility of a first interproximal space (or of either tooth associated with the interproximal space), a collision plane may be computed for the interproximal space, and an angle may be computed between that collision plane and the archform. The resulting angle may be provided to an ML model for IPR accessibility prediction, to improve the accuracy of the resulting prediction.

[0027] A machine learning (ML) model may be trained to predict a quantity of interproximal reduction (IPR) or an IPR cut surface to be applied to a 3D representation of a tooth (e.g., during the course of orthodontic setups prediction). In some implementations, IPR may be digitally applied to a 3D representation of a tooth prior to or as a part of setups prediction (e.g., during the training and / or use of a setups prediction model). IPR may be applied to the mesial side of a tooth or to the distal side of a tooth (or to both sides of the tooth). IPR may be applied to create space within an arch, for example, to remove enamel from one or more of the lower incisors on the right side of FIG. 1. IPR may be applied to other teeth, as well. Techniques of this disclosure may be trained to generate IPR cut surfaces (e.g., geometries which when combined using Boolean operations with 3D representations of teeth may be used to remove or subtract portions of those teeth), which may be used to digitally apply IPR to a tooth. An IPR cut surface may comprise a flat plane (e.g., an IPR cut plane - which may be described by 3 or more points or may comprise a normal vector in combination with a point) or a nonflat surface (e.g., which may comprise a 3D triangle mesh, 3D point cloud, voxels, one or more splines, or one or more polylines). An IPR cut surface may be applied to the 3D representation of a tooth to digitally remove material from that tooth (e.g., using a Boolean mesh processing operation).

[0028] Techniques described herein may use representation learning in the training of ML models which may generate (or modify) information pertaining to IPR for a patient case. Information pertaining to IPR may include one or more of: 1) an IPR cut surface for a particular tooth; 2) a measure of IPR magnitude to be applied to a side of a particular tooth; 3) a designation of whether a particular tooth is to undergo IPR - including an indication of whether IPR is to be applied to the mesial or distal side of the tooth; 4) a designation of one or more stages in which a particular tooth is to undergo IPR; 5) a determination regarding whether a target tooth is accessible to IPR (e.g., whether orthodontic treatment should first be applied to the arch in order to make the target tooth accessible to IPR); and 6) an orthodontic setup transform for a tooth which isaccompanied by one or more IPR cut surfaces - for application to that tooth during the course of setup generation. Table 1 summarizes selected ML models of this disclosure which pertain to IPR. A representation generation module (e.g., the first ML module) may comprise one or more ML models, such as an encoder (e.g., which may be trained as a part of an autoencoder or a transformer), a U-Net, a pyramid encoder-decoder, a transformer encoder, one or more sets of pooling and convolution layers, other encoder-decoder structures, or other neural networks which may be trained to generate latent representations of the teeth (or other aspects of a patient’s dentition) which have a lower order of dimensionality than the input data. In some implementations, such a representation-generating ML module may take as input 3D representations of the patient’s dentition (e.g., teeth or gums), and may optionally use a mesh element feature module to compute mesh element feature vectors (e.g., as described herein). A mesh element feature vector may be computed for one or more mesh elements of a 3D representation (e.g., a mesh, voxelized representation or point cloud). Such a mesh element feature vector may describe information about the structure and / or shape of the patient's teeth or of other 3D oral care representations which are provided to the models described herein. This use of mesh element features may improve the ability of a first ML module described herein to generate representations which accurately describe the teeth of the patient.

[0029] In some implementations, techniques of this disclosure may generate IPR cut surfaces (or magnitudes of IPR) to be applied to the mesial or distal sides of one or more teeth. The generated IPR cut surfaces (or IPR magnitudes) may be provided as inputs to a subsequent setups prediction model, by indicating to the setups model how much enamel should be removed from each tooth (for teeth which are designated to receive IPR) or indicating to the setups model where the enamel should be cut. Stated another way, a generated IPR cut surface may define a portion of the enamel of a tooth for removal. Examples of an IPR cut surface include a flat IPR cut plane or a non-planar geometry which defines the surface upon which IPR is to be applied. An IPR cut plane may be defined, at least in part, by a normal vector and a point which is located on that plane. Other representations of IPR cut planes are possible, such as defining three or more points (e.g., points which he on that plane). In some implementations, a non-planar cut surface may be generated by techniques of this disclosure and may comprise one or more 3D meshes (or 3D point clouds or voxels or polylines).Table 1

[0030] An ML model may be trained using Representation Learning, at least in part, to generate a prediction of an IPR cut surface (among other representations of IPR). For example, 3D representations of the patient’s teeth may be provided to a first ML module may generate latent representations of the patient’s teeth (e.g., each tooth may be described by a latent vector, latent capsule or other information-rich or reduced dimensionality form). In some implementations, optional tooth transforms (e.g., maloccluded transforms) may be provided to the first ML model. The latent representations may, in some instances, reduce the dimensionality or computing footprint of the teeth, while maintaining rich information of the shape and / orstructure of the teeth. In some implementations, the high quality of the latent representations may be demonstrated by reconstructing the latent representations (e.g., using an autoencoder decoder or a transformer decoder) into reconstructed tooth meshes. A reconstruction error may, in some instances, be computed to quantify the difference between the inputted and reconstructed 3D tooth representations. In some implementations, the first ML module may extract neural network features (e.g., using conventional techniques) from the patient’s dentition (or from other 3D representations provided to the first ML module). These neural network features may describe global aspects of a tooth (e.g., aspects of the tooth shape / structure as a whole), local aspects of a tooth (e.g., aspects of the local vicinity around a small portion of the tooth), or aspects of the tooth at intermediate scales in between global and local. Such local, intermediate or global neural network features may improve the fidelity of the representation generated by the first module, and subsequently improve the accuracy of data structure generated by the second ML module.

[0031] The 3D representations of the patient’s teeth may comprise mesh elements. Mesh element features (e.g., spatial or structural mesh element features described in Table 1, such as “Curvature” or location coordinates such as “XYZ position”) may be computed for the mesh elements, and subsequently be provided to the first ML module. The mesh element features may improve the ability of the first module to encode the shape and / or structure of the teeth (or other aspects of dentition), thereby improving the accuracy of those latent representations. The latent representations of the teeth (e.g., encoded as latent vectors or embedding vectors of lower dimensionality than the input data) may be concatenated using a variety of concatenation techniques. The concatenated latent vectors may, in some implementations, be concatenated with one or more oral care arguments. The concatenated latent vectors may subsequently be provided to a second ML module, which may be trained to generate data structures in the right-most column of Table 1. Stated another way, one or more latent representations of the patient’s dentition may be generated by the first ML module, be concatenated with vectors which contain other inputs (e.g., oral care arguments), and subsequently be provided to the second ML module for generative modeling. The second ML module may comprise one or more ML models, such one or more multilayer perceptrons (MLP) (e.g., containing optional skip connections), one or more autoencoders, one or more U-Nets, or one or more transformers (e.g., a transformer encoder, or a transformer decoder - which may have more multi-head attention blocks than a corresponding transformer encoder), among other architectures described herein. In some implementations, the second ML module may contain 4 fully connected layers (or another number of fully connected layers). In some implementations, the second ML module may contain a transformer decoder (or transformer encoder) which generates a latent representation that is then provided to a decoder, which may reconstruct that latent representation. The second ML module may be trained, at least in part, by the calculation of one or more loss values (e.g., a cross entropy loss, an LI, or and L2 loss, among others disclosed herein). In some implementations, the second ML module may contain one or more denoising diffusion probabilistic models (e.g., a U-Net which may iteratively denoisean initially noisy data structure, which was trained on a series of iteratively noisier examples of data from a training dataset).

[0032] The method 324 of FIG. 3 may train an ML model to generate one or more predicted IPR cut surfaces for one or more corresponding target teeth. For each patient case: Tooth meshes 300 may be provided to a mesh element feature module 310, which may compute mesh element feature vectors for the mesh elements of the tooth meshes 300. When the first ML module 312 contains a hierarchical neural network feature extraction module (HNNFEM) (e.g., a U-Net, a pyramid encoder-decoder, or 3D SWIN transformer, etc.), the tooth meshes and the corresponding tooth transforms (e.g., transforms for a stage at which the IPR cut surfaces are to be predicted) may be provided to the first ML module 312. When the first ML module 312 contains other latent representation generation networks (e.g., an encoder that was trained as a part of a reconstruction autoencoder, among others described herein), the tooth transforms 302 may undergo optional latent encoding (304), and then be provided to the second ML module 314. The one or more latent representations generated by the first ML module 312 may also be provided to the second ML module 314. Optional flags 308 may, in some implementations, be provided to second ML module 314 (e.g., optional flag for each tooth to indicate whether IPR is allowed, or optional flags specifying one or more target teeth - teeth for which IPR cut surface may be predicted, if applicable). Such flags may influence the output of the second ML module 314 (e.g., may determine whether a particular tooth receives IPR). The second ML module 314 may be trained to generate one or more IPR cut surfaces (or one or more magnitude of IPR) to be applied to the mesial or distal sides of each target tooth. The predicted IPR cut surfaces (or magnitudes) 318 may be compared (320) to corresponding ground truth IPR cut surfaces (or magnitudes) 306, and resulting in loss 322 (e.g., a cross entropy loss, an L 1, or an L2 loss, or among others disclosed herein). The loss 322 may be used to train (316) one or more of the first ML module 312 or the second ML module 314.

[0033] In some implementations, information pertaining to tooth type may be provided to either or both of the first 312 or second 314 ML modules. Tooth type may help the first ML module customize the generated representations. Tooth type may help the second ML module to customize the outputs described in Table 1. For example, an ML model for predicting IPR may take as input information pertaining to the type of a tooth, which may improve the generated IPR cut surface. In some implementations, the IPR cut surface may be customized to the type of tooth (e.g., based on differences in tooth shape). In particular, it should be appreciated that an IPR cut surface may, in some instances, be different for a lower central incisor than for an upper lateral incisor. In some implementations, the orientation (relative to the target tooth) of an IPR cut surface may first be generated using the ML techniques of this disclosure, and then the ML techniques of this disclosure may be trained to predict a magnitude of IPR to be applied to a tooth (e.g., compute a distance inward from either the mesial or distal side of a tooth at which the IPR cut surface is placed). As seen in the first row of Table 1, the second ML module (e.g., 4 fully connected layers with optional skip connections, among others described herein) may be trained to generate vectors (e.g., describing 3 or more points of a plane,or describing a point and a normal vector to define a plane) which describe IPR cut surfaces which are to be applied to one or more target teeth of the patient’s dentition.

[0034] The method 446 of FIG. 4 may train an ML model to generate setups transforms for the one or more teeth of a patient case in the training dataset, and also generate one or more predicted IPR cut surfaces (or IPR magnitudes) for one or more of the patient’s teeth. In some implementations, the method may also generate an indication of which teeth are to receive IPR, and (optionally) an indication of which stage of orthodontic treatment in which to perform IPR for each target tooth. A target tooth is a tooth which is designated to receive IPR. For each patient case: Tooth meshes 400 may be provided to a mesh element feature module 438, which may compute mesh element feature vectors for the mesh elements of the tooth meshes 400. When the first ML module 412 contains a hierarchical neural network feature extraction module (HNNFEM), the tooth meshes and the corresponding tooth transforms (e.g., malocclusion transforms) may be provided to the first ML module 412. When the first ML module 412 contains other latent representation generation networks (e.g., an encoder that was trained as a part of a reconstruction autoencoder, among others described herein), the tooth transforms 402 may undergo optional latent encoding (410), and then be provided to the second ML module 414. The one or more latent representations generated by the first ML module 412 may also be provided to the second ML module 414. Optional flags 440 may, in some implementations, be provided to second ML module 414 (e.g., optional flag for each tooth to indicate whether IPR is allowed). Such flags may influence the output of the second ML module 414 (e.g., may determine to which teeth the model is allowed to apply IPR, if the model determines that IPR is applicable). The second ML module 414 may be trained to generate one or more predicted setups transforms 416, one or more predicted IPR cut surfaces (or magnitudes of IPR) 418 to be applied to the mesial or distal sides of each target tooth, and / or one or more designations of which teeth are to receive IPR 436.

[0035] The predicted IPR cut surfaces (or magnitudes) 418 may be compared (424) to corresponding ground truth IPR cut surfaces (or magnitudes) 406, resulting in loss 430 (e.g., a cross entropy loss, an LI, or an L2 loss, or among others disclosed herein).

[0036] The predicted tooth transforms 416 may be compared (426) to corresponding ground truth 404, resulting in loss 432 (e.g., reconstruction loss, representation loss, or others disclosed herein).

[0037] The predicted determinations 436 (of which teeth are to receive IPR) may be compared (422) to corresponding ground truth 408, resulting in loss 428 (e.g., a cross entropy loss, an LI, or an L2 loss, or among others disclosed herein).

[0038] The losses may be used to train (434) one or more of the first ML module 412 or the second ML module 414. In some implementations, information pertaining to tooth type may be provided to either or both of the first 412 or second 414 ML modules. Tooth type may help the first ML module customize the generated representations. Tooth type may help the second ML module to customize the outputs described in Table 1, as discussed elsewhere in this disclosure.

[0039] As seen in the second row of Table 1, the second ML module may be trained to generate one or more IPR cut surfaces (e.g., which may be encoded as vectors) for one or more teeth, along with one or more setups transforms (e.g., for final setups or intermediate stages) for one or more teeth. The generated IPR cut surfaces may enable the teeth of the arch to better fit together within the predicted orthodontic setup (e.g., by minimizing collisions between teeth), since the cut surfaces and setups transforms are generated substantially concurrently. For example, a setups prediction model may be trained to generate only transforms (and has no opportunity to apply IPR to the teeth) may operate in a space where there are fewer clinically acceptable generated setups. However, a setups prediction model which has been trained to predict both of tooth transforms and IPR cut surfaces, may operate in a space where there are relatively more clinically acceptable generated setups. For example, neural network weights which have been trained to generate tooth transforms may influence the prediction of IPR cut surfaces or other aspects of IPR. The result may be a more accurate orthodontic setup, since the shapes of some teeth may be modified to produce a better occlusion. Stated another way, the latent representations generated by the first ML module may be provided to the second ML module, which may be trained to generate orthodontic setups (e.g., in the form of tooth transforms which are encoded as quaternions, transformation vectors, transformation matrices or other data structures described herein), along with generating information pertaining to IPR for one or more teeth of the patient’s dentition.

[0040] In some implementations, oral care arguments may be provided to the second ML module, including oral care parameters, oral care metrics or other oral care arguments disclosed herein. Additional values may be provided to the input of the second ML module, for example, 1) one or more values to indicate whether a particular tooth is allowed to undergo IPR, 2) one or more values to indicate whether a particular tooth is accessible to IPR - or whether orthodontic treatment is needed prior to IPR. Information pertaining to IPR may include one or more of 1) an IPR cut surface for a particular tooth, 2) a measure of IPR magnitude to be applied to a side (e.g., mesial or distal) of a particular tooth, 3) a designation of whether a particular tooth is to undergo IPR - including an indication of whether IPR is to be applied to the mesial or distal side of the tooth, or 4) a designation of one or more stages in which a particular tooth is to undergo IPR. For example, a setups prediction model for generating intermediate stages may output a determination (e.g., based on the relative positions of the teeth in the intermediate stage) that one or more teeth may benefit from IPR at the start of treatment (e.g., in the maloccluded poses), and that same setups prediction model may output a determination that one or more other teeth may benefit from IPR at one or more subsequent stages of treatment (e.g., the 5thstage or the 12thstage, to name two examples). Information pertaining to IPR may, in some implementations, be encoded as vectors which contain real values (e.g., for IPR cut surface definitions), integers (e.g., for stage IDs), Boolean values, or categorical values. The generated IPR cut surfaces may enable the shapes of one or more teeth of an arch to be changed, so that the teeth better fit into a predicted orthodontic setup. For example, IPR may be applied to the distal surface of the upper left lateral incisor in FIG. 2, to enable that tooth to fitalongside its neighbors in a predicted setup (e.g., by reducing the size of the tooth). The resulting setup may be used in the generation of an oral care appliance (e.g., an orthodontic aligner tray, or indirect bonding tray, etc.).

[0041] In some implementations, the second ML module may be trained, at least in part, by a discriminator. For example, the second ML module may be trained, at least in part, by the calculation of one or more loss values. In some implementations, a loss value may be computed, at least in part, by comparing a predicted transform to a ground truth or reference transform (e.g., using a representation loss, reconstruction loss, LI loss, L2 loss, MSE loss, smooth LI loss, or combinations, among others). In some implementations, a loss value may be computed, at least in part, by comparing a predicted IPR cut plane to a ground truth or reference IPR cut plane (e.g., using an LI, L2, or cross-entropy loss, among others). In some implementations, a loss value may be computed, at least in part, by comparing a predicted indication of whether a particular tooth will undergo IPR (e.g., on the mesial or the distal side) to a ground truth or reference indication of whether a particular tooth will undergo IPR (e.g., on the mesial or the distal side), for example, using an LI, L2, or crossentropy loss, among others. In some implementations, a loss value may be computed, at least in part, by comparing a predicted stage in which to perform IPR on a particular tooth to a ground truth or reference stage in which to perform IPR on a particular tooth (e.g., using an LI, L2, or cross-entropy loss, among others). In some implementations, a loss value may be computed, at least in part, by comparing a predicted IPR magnitude to a ground truth or reference IPR magnitude (e.g., using an LI, L2, or cross-entropy loss, among others). In some implementations, the second ML module may generate a real or an integer value (e.g., in millimeters) which indicates a quantity of IPR to apply to either of the mesial or distal sides of a tooth. Furthermore, a discriminator neural network may be trained to distinguish between a predicted IPR cut surface and a ground truth (or reference) IPR cut surface. A discriminator loss term may be computed as a result of the determination, and used to train, at least in part, the second ML module (e.g., by modifying the weights of the second ML module).

[0042] In some implementations, a Boolean input value may be provided to a setups prediction model. According to particular implementations, the Boolean input value specifies or otherwise indicates whether or not the setups prediction model is allowed to generate a plan to apply IPR on a particular tooth (or whether the setups prediction model is allowed to generate a plan to apply IPR on any of the patient’s teeth). A plan to apply IPR may include an IPR cut surface, a magnitude of IPR, a designation of which tooth (or teeth) to target for IPR, a designation of whether IPR should be applied to the mesial side, the distal side or both sides of the target tooth, or a designation of which stage (or stages) or orthodontic treatment in which to apply IPR to a particular tooth. For example, a Boolean value may be provided to the input of a setups prediction model to indicate whether IPR is allowed for a particular patient case in the training dataset. When the values is set to "true” (e.g., a value of 1), IPR is allowed, and the model has the option to generate IPR cut surfaces, where warranted. When the value is set to "false” (e.g., a value of 0), IPR is not allowed for the case, and the model is penalized (e.g., via loss function) for any IPR cut surfaces which are generated. Such a setups predictionmodel may, in deployment, be configured to enable IPR (e.g., generate a setup with the possibility that one or more IPR cut surfaces may be generated in accompaniment to the setup), or to disable IPR (e.g., generate a setup without any IPR cut surfaces). A clinician (who is treating a particular patient case) may use such a deployed model to generate one or more setups with IPR and one or more setups without IPR, and then proceed to compare the setups. The comparison, such as using the automated setups comparison techniques of co-pending provisional application 63 / 461824 (the entirety of which is incorporated by reference herein in its entirety) and may influence treatment planning for the case. IPR may be applied to a final setup, to a maloccluded setup or to an intermediate stage. In some examples, IPR may be applied at a certain interval - for example every 8 weeks of treatment with orthodontic aligners.

[0043] As seen in the third tow of Table 1, the second ML module may be trained to predict whether IPR (e.g., either mesial or distal) is required to be applied to a particular tooth in an arch. Such a prediction may comprise a vector of Boolean values, each of which corresponds to a tooth in one arch or the other arch. Such a vector of Boolean values may subsequently be provided to a setups prediction model, for example, in accompaniment with IPR cut surfaces for the teeth which are deemed to require IPR. For each tooth, there may be Boolean values to specify or otherwise indicate whether the tooth requires IPR on the mesial side, and there may be a Boolean value to specify or otherwise indicate whether the tooth requires IPR on the distal side.

[0044] As seen in the fourth row of Table 1, the second ML module may be trained to generate a designation of one or more stages during which IPR should be applied to a tooth of the patient’s dentition. The designation of stage IDs may, in some implementations, comprise a vector of stage IDs, where each dimension in the vector corresponds to a tooth in one of the arches. In some instances, each tooth may have a range of possible stages (e.g., stages 2 - 5, among other ranges) at which IPR could possibly be applied. The techniques may generate a vector of stage IDs for each tooth in each of the arches, to designate the one or more stages at which IPR is to be applied. Such a vector of stage IDs may subsequently be provided to a setups prediction model (e.g., for generating intermediate stages), to inform the model about which stages would benefit from IPR. That setups prediction model may generate the IPR cut surfaces, or may receive IPR cut surfaces as inputs, for application to the affected teeth.

[0045] As seen in the fifth row of Table 1 and in FIG. 5, the second module 512 may be trained to generate one or more indications of whether a tooth in an arch is accessible to IPR treatment (e.g., to predict whether sufficient space exists between the target tooth and nearby teeth, such that IPR may be applied). Stated another way, the second ML module 512 may generate a predicted indication (e.g., based on the shapes and poses of teeth - which may provide information about crowding) of whether a target tooth is accessible for IPR, given the crowding and / or spatial relationships of nearby teeth in the same or in the opposing arch. For example, in a situation where there is a lot of tooth crowding, the second ML module may be more likely to generate a predicted indication that one or more teeth are inaccessible to IPR (e.g., which may indicate that one or more stages of orthodontic treatment are required before IPR may be applied). The indication may, in someimplementations, be described, at least in part, as a Boolean value for each tooth under consideration (e.g., true / false value). The indication may pertain to one or more stages of orthodontic treatment (e.g., mal, intermediate stage or final setup). For example, a target tooth may be accessible during some stages of orthodontic treatment, but not during others. The second ML module 512 may generate a list of stages in which a particular tooth is accessible to IPR or may generate a list of stages in which a particular tooth is not accessible to IPR. In some implementations, a list of one or more orthodontic stage IDs may be provided to the second ML module, and the second ML module may generate a list of Boolean values to indicate whether each of the specified stages is accessible to IPR. An example of a target tooth which is inaccessible to IPR is shown in FIG. 2 (e.g., the lower left central incisor). In some instances, the patient’s dentition may first undergo orthodontic treatment to make the target tooth accessible to IPR. The second ML module may, in some implementations, generate a Boolean flag for each tooth in the arch, to indicate whether the maloccluded pose of that tooth makes the tooth accessible to IPR. If a target tooth is not accessible to IPR treatment, then one or more stages of orthodontic treatment may need to first be applied to move one or more teeth of the arch to allow for the orthodontic treatment to be applied. For example, one or more stages of clear aligners may be applied to the patient’s dentition, to make the target tooth accessible to IPR.

[0046] In some implementations, one or more oral care metrics (e.g., orthodontic, or dental restoration metrics, such as “Alignment” or “Proximal Contacts”) may be provided as input to either of the first 510 or second 512 ML modules (e.g., during training or in deployment). In some implementations, one or more oral care metrics may be computed for one or more teeth of the patient. Such oral care metrics may be compared against one or more thresholds, to make a determination regarding whether orthodontic treatment must first be applied before IPR may be applied to one or more teeth (e.g., whether one or more aligner stages must first be applied before a target tooth may be made accessible for IPR), or whether orthodontic treatment has proceeded sufficiently far such that the target tooth may undergo IPR. In some implementations, second ML module 512 may generate a determination of whether a tooth must first undergo orthodontic treatment before the tooth may undergo IPR. The second ML module 512 may be trained, at least in part, by the calculation of one or more loss values. In some implementations, a loss value may be computed, at least in part, by comparing a predicted indication of whether a particular tooth is accessible to IPR (e.g., on the mesial or the distal side) to a ground truth or reference indication of whether a particular tooth is accessible to IPR (e.g., on the mesial or the distal side), for example, using an LI, L2, or cross-entropy loss, among others disclosed herein.

[0047] The method 528 of FIG. 5 may train an ML model to generate a determination regarding whether one or more of the patient’s teeth 500 are accessible to IPR. In some implementations, oral care arguments 514 may be provided to second ML module 512. The oral care arguments may, in some implementations, provide an indication to the second ML module 512 regarding which teeth are to be considered for accessibility determination (e.g., which teeth should be analyzed by the model to determine IPR accessibility). For each patient case: Tooth meshes 500 may be provided to a mesh element feature module 506, which may computemesh element feature vectors for the mesh elements of the tooth meshes 500. When the first ML module 510 contains a hierarchical neural network feature extraction module (HNNFEM), the tooth meshes and the corresponding tooth transforms (e.g., transforms from the malocclusion, an intermediate stage or the final setup) may be provided to the first ML module 510. When the first ML module 510 contains other latent representation generation networks (e.g., an encoder 1110 that was trained as a part of a reconstruction autoencoder, among others described herein), the tooth transforms 502 may undergo optional latent encoding (524), and then be provided to the second ML module 512. The one or more latent representations generated by the first ML module 510 may also be provided to the second ML module 512. Optional flags 514 may, in some implementations, be provided to second ML module 512 (e.g., optional flag for each tooth to indicate whether accessibility should be predicted). Such flags may influence the output of the second ML module 512. The second ML module 512 may generate one or more indications 516 of whether one or more teeth of the patient’s dentition 500 are accessible to IPR. The predicted indication 516 may be compared (518) to corresponding ground truth indication 506, resulting in loss 520 (e.g., which may be computed using crossentropy, LI, L2, or other loss techniques described herein). The loss may be used to train (522), at least in part, the second ML module 512. In some implementations, one or more oral care metrics (e.g., orthodontic metrics, such as “Alignment”, or restoration design metrics such as “Arch Discrepancies”) may be computed (526) based, at least in part, on the one or more teeth of the patient case 500 (e.g., after the tooth transforms have been applied to the teeth). The oral care metrics may be provided to the second ML module 512, to provide the second ML module 512 with information about the state of occlusion of the patient’s teeth which may aide the determination of IPR accessibility.

[0048] The method 616 described in FIG. 6 may use a fully trained ML model to generate IPR cut surfaces (or magnitudes). One or more teeth 600 of a patient case (e.g., 28 teeth, among other sets of teeth) may be provided to mesh element feature module 608, which may provide its output to first ML module 610. The tooth transforms 602 may be provided to the first ML module 610 or the tooth transforms may undergo latent encoding (604) and then be provided directly to the second ML module 612. The second ML module 612 may generate predicted IPR cut surfaces (or IPR magnitudes) 614 for one or more target teeth. T arget teeth may be specified by flags 606.

[0049] The method 722 described in FIG. 7 may use a fully trained ML model to generate IPR cut surfaces (or magnitudes). One or more teeth of a patient case 700 (e.g., 28 teeth, among other sets of teeth) may be provided to mesh element feature module 716, which may provide its output to first ML module 706. The tooth transforms 702 may be provided to either the first ML module 706 or may undergo latent encoding (704), be provided directly to the second ML module 708. The second ML module 708 may generate predicted setups transforms 710, predicted IPR cut surfaces (or IPR magnitudes) 712 for one or more target teeth, and / or predicted determinations of which teeth are to receive IPR. Target teeth may be specified by flags 720.Y1

[0050] The method 818 described in FIG. 8 may use a fully trained ML model to predict one or more indications of IPR accessibility. The method 818 may determine IPR accessibility for individual teeth or groups of teeth. For example, method 818 may determine that some teeth are accessible to IPR, while some other teeth are not accessible. One or more teeth of a patient case 800 (e.g., 28 teeth, among other sets of teeth) may be provided to mesh element feature module 806, which may provide it’s output to first ML module 810. The tooth transforms 802 may be provided to either the first ML module 810 or may undergo latent encoding (808) and then be provided directly to the second ML module 812. The second ML module 812, which may generate predicted setups transforms 814, predicted indications regarding whether one or more teeth are accessible to IPR. The teeth for which accessibility is to be determined may be specified by, for example, a set of flags that are provided as a part of oral care arguments 804. In some implementations, oral care metrics may be computed (816), based at least in part on the teeth of the patient’s dentition 800, and subsequently be provided to the second ML module 812.

[0051] FIG. 9 describes a method 912 for determining whether a tooth requires orthodontic treatment prior to IPR (e.g., to make the tooth accessible to IPR). The method 912 may make determinations for individual teeth or groups of teeth. Method 912 may determine that some teeth require orthodontic treatment prior to IPR, while some other teeth do not. One or more of the oral care metrics 904 described herein may be computed for 3D patient case data 900 (e.g., 3D meshes or 3D point clouds of the patient’s teeth and / or gums, and / or transforms). For example, one or more of overbite, overjet, alignment metrics (among others described herein) may be computed. In some instances, thresholds may be determined by statistical analysis of oral care metrics values in a training dataset. Thresholds 902 and oral care metrics may be provided to the decision module 906, which may output an indication of whether orthodontic treatment should precede IPR for the patient. When an oral care metric value falls beyond or within one or more pre-determined thresholds, an indication 910 may be outputted which indicates that the patient case may require orthodontic treatment prior to IPR. These metrics may also be used to determine if orthodontic treatment has progressed to a state where IPR may be performed. When the decision module 906 determines that orthodontic treatment is not required prior to IPR, then an indication 908 that IPR may proceed may be outputted.

[0052] FIG. 10 describes a machine learning (ML) model 1004 which may be trained to identify when orthodontic pre-treatment is advised ahead of IPR. The method 1012 may make determinations for individual teeth or groups of teeth. Method 1012 may determine that some teeth require orthodontic treatment prior to IPR, while some other teeth do not. The ML model may output an indication 1008 that orthodontic treatment is advised to precede IPR, or the ML model may output an indication 1006 that IPR may proceed without intervening orthodontic treatment. Such an ML model may be trained on 3D representations of the patient’s dentition 1000 (e.g., teeth and / or gums), including positive one or more examples of patient cases requiring orthodontic pre-treatment, or one or more negative patient cases which do not require orthodontic pretreatment. In some implementations, one or more oral care metrics may be computed (1002) (e.g., Alignmentor others disclosed herein) on the dentition 1000, and subsequently provide the metrics to the ML model 1004. The metrics may aid the ML model 1004 in making the determination. In some instances, one or more mesh element features (e.g., such as those described herein) may be computed by a mesh element feature module for one or more teeth (or other elements of patient dentition). For example, a mesh element feature vector may be computed for each mesh element within each 3D representation of patient dentition data in a patient case. Patient dentition data may be provided to a representation generation module (e.g., first ML modules 312, 412, 510, 610, 706, 810, or 1010). Such mesh element features may enable the representation generation module 1010 to better understand the shape and / or structure of the patient’s dental anatomy, enabling that representation generation module to generate a better output representations (e.g., a representation which is in latent form and / or has reduced dimensionality). In some implementations, such a representation may be reconstructed using a decoder into a facsimile of the original 3D representation of the patient’s dentition (e.g., using a reconstruction autoencoder, shown in FIG. 10). The quality of the latent representation may, in some instances, be demonstrated by computing a reconstruction error between the original and reconstructed 3D representations. In some implementations, when the method 1012 generates an indication 1006 that each tooth is accessible to IPR, then the patient’s dentition may be provided to an automated setups prediction model, which may generate setup tooth transforms and / or predicted IPR cut surfaces for the patient’s teeth.

[0053] The method in FIG. 11 shows a 3D representation of a tooth 1105 (or other aspects of patient dentition, such as gums), which may be encoded by an encoder 1110 into a latent form 1115. The latent form may be reconstructed by a decoder 1120 into a reconstructed tooth 1125. The reconstructed tooth 1125 may be a close facsimile of the tooth 1105, as may be measured by the reconstruction error described herein. In some implementations, the first ML models 312, 412, 610, 706, and 810 of this disclosure may use an encoder 1110 to generate latent representations 1115 of teeth. A reconstruction autoencoder may be trained using reconstruction loss, KL-Divergence loss, or other losses described herein.

[0054] Techniques of this disclosure may be trained on cohort patient case data. A patient case may comprise 3D representations of the teeth (e.g., meshes, point clouds or voxel representations), maloccluded poses for the teeth (e.g., transforms), ground truth final setup poses for the teeth (e.g., transforms), ground truth intermediate stage poses for the teeth (e.g., transforms), ground truth IPR specifications for one or more teeth (e.g., ground truth IPR cut surfaces, ground truth designations of which teeth are to receive IPR, ground truth designations of in which state(s) IPR is to be performed), information pertaining to tooth type (or tooth identification). In some instances, a particular tooth may not be accessible to IPR during the first few stages of orthodontic treatment but may become accessible after the application of one or more aligner stages. For example, a target tooth may be inaccessible to IPR at the start of treatment due to the poses of one or more nearby teeth. The patient's teeth may need to undergo orthodontic treatment before a target tooth may be made accessible to IPR, such that, in one example, IPR may be applied after 5 aligner trays have been used in treatment (or some other number of aligner trays). In some instances, the cohort patient case data may include information pertaining toIPR accessibility (e.g., one or more ground truth flags may be associated with a tooth which indicate whether a tooth is accessible for IPR).

[0055] Oral care arguments may be provided to either of the first or the second ML modules. One or more oral care arguments (e.g., an oral care parameter, such as procedure parameters or restoration design parameters) may be defined to specify one or more aspects of an intended 3D oral care representation, which is to be generated using the machine learning models described herein, which have been trained for that purpose. In some implementations, an oral care argument may be defined which corresponds to an oral care metric, which may be received as the input to the machine learning models described herein and may be taken as an instruction to that module to generate a 3D oral care representation with the specified customization. This interplay between oral care metrics and oral care parameters may also apply to the training and deployment of other predictive models in oral care. Some oral care parameters may be defined which describe aspects of an intended IPR cut surface (e.g., a cut plane or a non-planar cut surface - such as described by a 3D triangle mesh). Some examples of IPR-related oral care parameters include (e.g., for a particular tooth) a minimum or maximum magnitude of IPR that may be placed on the mesial or distal side of that tooth. Some examples of IPR-related oral care parameters include a designation of which stages of treatment in which IPR is permitted to be performed (e.g., mal, final setup or intermediate stage - or in which specific intermediate stages IPR may be performed). Some examples of IPR-related oral care parameters include a designation of whether asymmetric IPR may be performed (e.g., asymmetric IPR is where IPR is placed on one side of an arch but not the other). A non-limiting example of asymmetric IPR is when the midline on the lower arch needs to move left or right to be aligned with the upper arch, and IPR is applied to only the LR side or the LL side, (e.g., apply to either LR1-LR3 or LL1-LL3). IPR may, in some instances, be measured in millimeters. A typical magnitude of IPR may, in some instances, be expressed as a fraction of a millimeter (0. 1 mm, or 0. 15 mm, or the like). With regard to automated IPR cut surface generation or automated setups generation, a clinician may specify clinician preferences (e.g., which may define default values for one or more oral care parameters - such as maximum allowed IPR magnitude). A clinician preference may be learned from a dataset of patient cases treated by the clinician, or the preference may be configured as an input to the automated generative model. Examples of IPR-related clinician preferences include 1) the preference to wait until the teeth are sufficiently aligned (e.g., an Alignment metric is within a threshold) to apply IPR, 2) the preference to wait until a target tooth’s proximal surface is accessible, 3) the preference to apply IPR at a particular stage (e.g., stage 1).

[0056] In some implementations, the automated techniques of this disclosure may be trained to align 3D representations of the teeth of a patient’s arch before generating predictions pertaining to IPR. Such alignment may be realized through automated setups generation (e.g., final setups or intermediate stages), such as in FIG. 4. The setups prediction model may concurrently (or independently) predict one or more IPR cut surfaces (or IPR magnitudes) for one or more of the interproximal spaces. The setups may then be used in the generationof physical aligner appliances (e.g., using a 3D printer, which may print aligners or print fixture models upon which plastic aligner trays may be thermoformed). The alignment which is produced by a setup may cause the proximal surfaces of neighboring teeth to become aligned (e.g., as seen in the alignment between the upper left central incisor and the upper left lateral incisor in FIG. 1), which may facilitate IPR (e.g., IPR which is carried out using a diamond strip, a burr, or a circular disc). Proximal surfaces refer to the mesial and distal sides of two adjacent teeth. The setups generation techniques of this disclosure may also place the teeth of the arch so that there are sufficient interproximal spaces between the teeth (e.g., 0. 1 mm, among other standards for gap measurement). Sufficient interproximal spaces may further facilitate IPR because the tools used to carry out IPR are given enough room for use. Sufficient interproximal space between two teeth may indicate that the teeth are accessible to IPR. IPR may, in some instances, be applied equally to both teeth, or differentially more to one tooth or the other tooth.

[0057] FIG. 1 illustrates the relationship between tooth alignment and the determination of whether an interproximal space is accessible to IPR. On the left of FIG. 2: The teeth are well aligned for application of IPR. IPR may be performed at any stage of the treatment of this arch. For each pair of teeth, both proximal surfaces are accessible. The filaments (e.g., 200, 202, 204, 206, 208) in on the left side of FIG. 2 describe flat IPR cut planes between some pairs of the teeth in the arch. On the right side of FIG 2: The teeth are not well aligned for IPR. IPR may be more easily performed at a later intermediate stage, after some orthodontic treatment has been performed to develop space between pairs of teeth. The filaments on the right of FIG. 2 (e.g., 210, 212, 216, or 214) describe IPR cut surfaces (e.g., nonlinear cut surfaces) between some of the teeth of the arch. These examples of IPR cut surfaces would be difficult to apply in clinical practice, due to the misalignment of the teeth and the lack of space between many pairs of teeth. An ML model which is trained according to the method in FIG. 5 may distinguish between the arch on the left (where the teeth of the arch are all accessible to IPR) or the arch on the right (where some of the teeth are not accessible to IPR). In some implementations, the method in FIG. 5 may identify one or more teeth in an arch which are accessible to IPR or may identify one or more teeth of the arch which are not accessible to IPR.

[0058] Techniques of this disclosure can generate an indication of whether a tooth is accessible to IPR (e.g., techniques based on an ML model, oral care metrics, or other techniques). In some implementations, collisions between nearby or adjacent teeth can be detected (e.g., collisions between the mesial and distal sides of adjacent teeth). Collisions at a particular stage of orthodontic treatment (e.g., in malocclusion) may, in some instances, indicate that a particular tooth is inaccessible to IPR, and can benefit from the application of one or more stages of orthodontic treatment before IPR can be applied. Collisions may be used, at least in part, to compute a real-valued accessibility score (e.g., on a scale of 0 to 1.0). Factors for scoring IPR accessibility include the magnitude of collision overlap (e.g., computing Collison depth), the angle between a collision plane and an archform (e.g., describes the overall contour or layout of an arch of teeth), or the angle between a predicted IPR cut plane and a collision plane, among others. A collision plane may be defined between two ormore teeth which are in collision. The collision plane may be defined, at least in part, by three or more overlapping mesh elements (e.g., overlapping vertices, etc.) of the two or more colliding teeth.

[0059] Information pertaining to the overlap between two or more teeth (e.g., count of overlapping mesh elements or statistics generated by other scoring techniques) may be provided to the second ML module, as described herein. For example, when two or more adjacent teeth overlap (e.g., one or more mesh elements of the first tooth are within the volume of the second tooth), a collision depth can be computed. In some implementations, collision depth may be computed according to the description of penetration depth in PCT Application No. WO2020136587A1, the entirety of which is incorporated herein by reference. Such information may be provided to the second ML module (e.g., when the second ML module predicts an orthodontic setup, am IPR cut surface, or makes a determination regarding IPR accessibility, among others described herein). For example, given the correlation between 1) count of overlapping vertices, and 2) magnitude of IPR (in some instances), a second ML module may be aided in predicting an IPR cut surface (or a magnitude of IPR) by knowing about the magnitude of overlap between the teeth.

[0060] When it is determined that a particular tooth is accessible to IPR, an indication that the tooth is accessible to IPR can be provided to a setups prediction model. Stated another way, the IPR accessibility of each tooth in an arch can be determined, and the indications of IPR accessibility for the plurality of teeth can be provided to a setups prediction model. The setups prediction model can then generate setups transforms for the patient’s teeth and may generate IPR cut surfaces (or IPR cut planes) for one or more of those teeth, to facilitate the generation of a final setup or intermediate stage. IPR cut surfaces can be predicted, if indicated by the patient’s dentition, for the teeth which have been designated as IPR accessible. In some implementations, the setups prediction model can generate an indication of one or more stages in which to perform IPR for a particular tooth. IPR can then be applied to the affected teeth at the designed intermediate stages, and then intermediate stage generation can proceed in light of the fact that IPR has been applied to those teeth.

[0061] In some implementations, the results of IPR accessibility determination can be displayed to clinicians, for the purpose of treatment planning.

[0062] Techniques of this disclosure may require a training dataset of hundreds or thousands of cohort patient cases, to ensure that the neural network is able to encode the distribution of patient cases which are likely to be encountered in clinical treatment. A cohort patient case may include a set of tooth crown meshes, a set of tooth root meshes, or a data file containing attributes of the case (e.g., a JSON file). A typical example of a cohort patient case may contain up to 32 crown meshes (e.g., which may each contain tens of thousands of vertices or tens of thousands of faces), up to 32 root meshes (e.g., which may each contain tens of thousands of vertices or tens of thousands of faces), multiple gingiva mesh (e.g., which may each contain tens of thousands of vertices or tens of thousands of faces) or one or more JSON files which may each contain tens of thousands of values (e.g., objects, arrays, strings, real values, Boolean values or Null values).

[0061] It should be understood, without loss of generality, that there are various types of 3D representations. One type of 3D representation is a 3D geometry. A 3D representation may include, be, or be part of one or more of a 3D polygon mesh, a 3D point cloud (e.g., such as derived from a 3D mesh), a 3D voxelized representation (e.g., a collection of voxels - for sparse processing), or 3D representations which are described by mathematical equations. Although the term “mesh” is used frequently throughout this disclosure, the term should be understood, in some implementations, to be interchangeable with other types of 3D representations. A 3D representation may describe elements of the 3D geometry and / or 3D structure of an object.

[0062] A 3D mesh is a data structure which may describe the geometry or shape of an object related to oral care, including but not limited to a tooth, a hardware element, or a patient’s gum tissue. A 3D mesh may include one or more mesh elements such as one or more of vertices, edges, faces and combinations thereof. In some implementations, mesh elements may include voxels, such as in the context of sparse mesh processing operations. Various spatial and structural features may be computed for these mesh elements and be provided to the predictive models of this disclosure, with the predictive models of this disclosure providing the technical advantage of improving data precision in the form of the models of this disclosure outputting more accurate predictions.

[0063] A patient’s dentition may include one or more 3D representations of the patient’s teeth (e.g., and / or associated transforms), gums and / or other oral anatomy. An orthodontic metric (OM) may, in some implementations, quantify the relative positions and / or orientations of at least one 3D representation of a tooth relative to at least one other 3D representation of a tooth. A restoration design metric (RDM) may, in some implementations, quantify at least one aspect of the structure and / or shape of a 3D representation of a tooth. An orthodontic landmark (OL) may, in some implementations, locate one or more points or other structural regions of interest on a 3D representation of a tooth. An OL may, in some implementations, be provided to the generation of an orthodontic or dental appliance, such as a clear tray aligner or a dental restoration appliance. A mesh element may, in some implementations, comprise at least one constituent element of a 3D representation of oral care data. For example, in the case of a tooth that is represented by a 3D mesh, mesh elements may include at least: vertices, edges, faces and voxels. A mesh element feature may, in some implementations, quantify some aspect of a 3D representation in proximity to or in relation with one or more mesh elements, as described elsewhere in this disclosure. Orthodontic procedure parameters (OPP) may, in some implementations, specify at least one value which defines at least one aspect of planned orthodontic treatment for the patient (e.g., specifying desired target attributes of a final setup in final setups prediction). Orthodontic Doctor preferences (ODP) may, in some implementations, specify at least one typical value for an OPP, which may, in some instances, be derived from past cases which have been treated by one or more oral care practitioners. Restoration Design Parameters (RDP) may, in some implementations, specify at least one value which defines at least one aspect of planned dental restoration treatment for the patient (e.g., specifying desired target attributes of a tooth which is to undergo treatment with a dental restoration appliance). Doctor Restoration Design Preferences(DRDP) may, in some implementations, specify at least one typical value for an RDP, which may, in some instances, be derived from past cases which have been treated by one or more oral care practitioners.

[0064] 3D oral care representations may include, but are not limited to: 1) a set of mesh element labels which may be applied to the 3D mesh elements of teeth / gums / hardware / appliance meshes (or point clouds) in the course of mesh segmentation or mesh cleanup; 2) 3D representation(s) for one or more teeth / gums / hardware / appliances for which shapes have been modified (e.g., trimmed, distorted, or filled-in) in the course of mesh segmentation or mesh cleanup; 3) one or more coordinate systems (e.g., describing one, two, three or more coordinate axes) for a single tooth or a group of teeth (such as a full arch - as with the LDE coordinate system); 4) 3D representation(s) for one or more teeth for which shapes have been modified or otherwise made suitable for use in dental restoration; 5) 3D representation(s) for one or more dental restoration appliance components; 6) one or more transforms to be applied to one or more of: dental restoration appliance library component placement relative to one or more teeth, a tooth to be placed for an orthodontic setup (either final setup or intermediate stage), a hardware element to be placed relative to one or more teeth or the like; 7) an orthodontic setup; 8) a 3D representation of a hardware element (such as facial bracket, lingual bracket, orthodontic attachment, button, hook, bite ramp, etc.) to be placed relative to one or more teeth, etc.; 8) a 3D representation of a bonding pad for a hardware element (which may be generated for a specific tooth by outlining a perimeter on the tooth, specifying a thickness to form a shell, and then subtracting-out the tooth via a Boolean operation); 9) 3D representation of a clear tray aligner (CTA); 10) the location or shape of a CTA trimline (e.g., described as either a mesh or polyline); 11) archform that describes the contours or layout of an arch of teeth (e.g., described as a 3D polyline or as a 3D mesh or surface), which may follow the incisal edges one or more teeth, which may follow the facial surfaces of one or more teeth, which may in some implementations correspond to the maloccluded arch and in other implementations correspond to the final setup arch (the effects of malocclusion on the shape of the archform may be diminished by smoothing or averaging of the shape of the archform), which may be described by one or more control points and / or a spline; 12) 3D representation of a fixture models (e.g., depictions of teeth and gums for use in thermoforming clear tray aligners, or depictions of teeth / gums / hardware for use in thermoforming indirect bonding trays); 13) one or more latent space vectors (or latent capsules) produced by the 3D encoder stage of a 3D autoencoder which has been trained on the reconstruction of oral care meshes (e.g., a variational autoencoder which has been trained for tooth reconstruction); 14) one or more oral care metrics values (e.g., such as orthodontic metrics or restoration design generation metrics) for one or more teeth; 15) one or more landmarks (e.g., 3D points) which describe the shapes and / or geometrical attributes of one or more teeth, other dentition structures or hardware structures (e.g., to be used in orthodontic setups creation or restoration appliance component generation or placement); 16) 3D representation created by scanning (e.g., optically scanning, CT scanning or MRI scanning) a 3D printed part corresponding to one or more teeth / gums / hardware / appliances (e.g., a scanned fixture model); 17) 3D printed aligners (including optionally local thickness, reinforcing rib geometry, flap positioning, or the like) 18) 3Drepresentation of the patient's dentition that was captured chairside by a clinician or medical practitioner (e.g., in a context where the 3D representation is validated chairside, before the patient leaves the clinic, so that errors can be detected and re-scans performed as necessary); 19) dental restoration tooth design (e.g., for veneers, crowns, bridges or dental restoration appliances); 20) 3D representations of one or more teeth for use in digital oral care treatment; 21) other 3D printed parts pertaining to oral care procedures or other fields; 22) IPR cut surfaces; 23) one or more orthodontic setups transforms associated with one or more IPR cut surfaces; 24) a (digital) pontic tooth design which may fill at least a portion of the space between teeth to allow room in an orthodontic setup for an erupting tooth to later emerge from the gums; or 25) a component of a fixture model (e.g., comprising fixture model components such as interproximal webbing, block-out, bite locks, bite ramps, interproximal reinforcement, gingival ridges, torque points, power ridges, pontic tooth or dimples, among others).

[0065] Some implementations of a neural network for predicting a setup may incorporate information from an oral care professional (aka doctor). This information may influence the arrangement of teeth in the final setup, bringing the positions and orientations of the teeth into conformance with a specification set by the doctor, within tolerances. A doctor’s preferences (e.g., in an orthodontic context) and / or doctor’s restoration preferences may be indicated in a treatment form, or they could be based upon characteristics in treatment plans such as final setup characteristics (e.g., amount of bite correction or midline correction in planned final setups), intermediate staging characteristics (e.g., treatment duration, tooth movement protocols, or overcorrection strategies), or outcomes (e.g., number of revisions / refinements).

[0066] Orthodontic procedure parameters may specify one or more of the following (with possible values shown in {}). Non-limiting categorical values, for some example OPP are described below. In some implementations, a real value may be specified for one or more of these OPP. For example, the Overbite OPP may specify a quantity of overbite (e.g., in millimeters) which is desired in a setup, and may be received as input of a setups prediction model to provide that setups prediction model information about the amount of overbite which is desired in the setup. Some implementations may specify a numerical value for the Overjet OPP, or other OPP. In some implementations, one or more OPP may be defined which correspond to one or more orthodontic metrics (OM). In some instances, a numerical value may be specified for such an OPP, for the purpose of controlling the output of a setups prediction model.Teeth To Move: {AnteriorsOnly, AnteriorsAndBicuspids, FullArch}Tooth Movement Restrictions: for each tooth, indicate if tooth is {DoNotMove, Missing, ToBeExtracted, Primary / Erupting, Clear}Overbite: {ShowResultingOverbiteAfterAlignment, Maintainlnitial Overbite, CorrectOpenBite,CorrectDeepBite }Overjet: {ShowResultingOverjetAfterAlignment, MaintainlnitialOveijet, ImproveResultingOverjet}Anterior / Posterior (AP) RelationshipMaintain: {Right, Left, Both}Improve canine relationship only: {Right, Left, Both}Improve canine and / or molar relationship up to 4mm: {Right, Left, Both}Correct to Class I (canine and molar): {Right, Left, Both}Crossbite (if present)Anterior: {DoNotCorrect, Correct, N / A}Posterior: {DoNotCorrect, Correct, N / A}Correction to Class I (canine and molar): {Right, Left, Both}Correct with Posterior IPR: {yes, no}Class II / III correction simulation (elastics required): {yes, no}Sequential Distalization (elastic recommended): {yes, no}Include cuts for elastics?: {yes, no}Preferred cuts for elastics: {UseButtonCutoutsOnMolarsAndHooksOnCanines,UseButtonCutoutsOnly, UseHooksOnly }Stage to start cuts for elastics: [integer]LevelingOfUpperAnteriors: [Laterals!) AmmShortcrThanCcntral. LevellncisalEdges, LevelGingivalMargins, Aslndicated}Spacing: {CloseAllSpaces, LeaveSpecificSpaces}Preferred Midline Position: {SetTheUpperMidlineToIdeal, MatchTheUpperAndLowerToEachOther}Resolve Upper Crowding by Expand: {Primarily, AsNeeded, None}Resolve Upper Crowding by Prodine: {Primarily, AsNeeded, None}Resolve Upper Crowding by IPR - Anterior: {Primarily, AsNeeded, None}Resolve Upper Crowding by IPR - Posterior Right: {Primarily, AsNeeded, None}Resolve Upper Crowding by IPR - Posterior Left: {Primarily, AsNeeded, None}Resolve Lower Crowding by Expand: {Primarily, AsNeeded, None}Resolve Lower Crowding by Procline: {Primarily, AsNeeded, None}Resolve Lower Crowding by IPR - Anterior: {Primarily, AsNeeded, None}Resolve Lower Crowding by IPR - Posterior Right: {Primarily, AsNeeded, None}Resolve Lower Crowding by IPR - Posterior Left: {Primarily, AsNeeded, None}Finishing Arch Form: {Patient’sNatural, Aslndicated}[doctor can specify an archform - selected from a set of options or custom-designed]

[0067] Other orthodontic procedure parameters may be defined, such as those which may be used to place standardized brackets at prescribed occlusal heights on the teeth. In some implementations, one or more orthodontic procedure parameters may be defined to specify at least one of the 2ndand 3rdorder rotation angles to be applied to a tooth (i.e., angulation and torque, respectively), which may enable a target setup arrangement where crown landmarks lie within a threshold distance of a common occlusal plane, for example. In some implementations, one or more orthodontic procedure parameters may be defined to specify the position in global coordinates where at least one landmark (e.g., a centroid) of a tooth crown (or root) is to be placed in a setup arrangement of teeth. Generally, an oral care parameter may be defined which corresponds to an oral care metric. For example, an orthodontic procedure parameter may be defined which corresponds to an orthodontic metric (e.g., to specify at the input of a setups prediction model an amount of a certain metric which is desired to appear in a predicted setup).

[0068] Doctor preferences may differ from orthodontic procedure parameters in that doctor preferences pertain to an oral care provider and may comprise of the means, modes, medians, minimums, or maximums (or some other statistic) of past settings associated with an oral care provider’s treatment decisions on past orthodontic cases. Procedure parameters, on the other hand, may pertain to a specific patient, and describe the needs of a particular patient’s treatment. Doctor preferences may pertain to a doctor and the doctor’s past treatment practices, whereas procedure parameters may pertain to the treatment of a particular patient. Doctor preferences (or “treatment preferences”) may specify one or more of the following (with possible values shown m {})•

[0069] Orthodontic doctor preferences may specify one or more of the following (with other possible values found elsewhere in this disclosure).Deep Bite Cases (Amount of Bite Correction) - Final Overbite: [real value in millimeters, e.g., 0.5 mm]Option - Intrude Upper Anteriors: {yes, no}Option - Include lower canines in vertical overcorrection: {yes, no}Midline Correction in Planned Final Setup: {MaintainlnitialMidline, ImproveMidlineWithlPR, Aslndicated}Deep Bite Cases - Reverse Curve of Speed: {yes, no}Anterior Open Bite Cases - Final Overbite: [real value in millimeters, e.g., 2 mm]Is Arch Expansion a Priority for Your Cases?: {Yes, No}If yes, specify acceptable expansion per quadrant in mm.When expanding upper molars, apply buccal root torque: {yes, no}Is IPR Acceptable of First Tx Design: {yes, no}Maximum IPR per contact:Upper Anterior: [specify in mm]Lower Anterior: [specify in mm]Upper and Lower Anterior: [specify in mm]Is Asymmetric IPR Acceptable?: {yes, no}Final Tooth Position (Overcorrection Strategy): {Ideal, Overcorrected}Root Movement: {MoveRootsAsNeededToAchieveTreatmentGoals, LimitPosteriorRootMovement, LimitAllRootMovement}Final Occlusal Contacts: {AllContactsBalancedWhenPossible, NoOcclusalContactOnUpperlncisors, FinishWithHeavyPosteriorContacts, Other}Is Asymmetric AP Shift Acceptable for Class Correction?: {yes, no, other}Treatment Duration: [count of stages]Tooth Movement Protocol: {protocol_A, protocol_B, protocol_C}

[0070] A setups prediction neural network of this disclosure may be trained, at least in part, by the calculation of one or more loss values (e.g., reconstruction loss or other loss values described herein). Such loss values may quantify the difference between a predicted setup and a corresponding ground truth setup. In some instances, these setups may be registered with each other (e.g., using iterative closest point (I CP) or singular value decomposition (SVD)) before the loss is computed, to reduce noise and improve the accuracy of the resulting trained setups prediction neural network. Such a registration may alternatively or additionally be performed between the maloccluded setup and the corresponding ground truth setup, with the advantage of reducing noise in the loss measurement and improving the accuracy of the trained network.

[0071] The setups prediction neural network may compute a transform for each tooth, to move that tooth into a pose which is suitable for the end of orthodontic treatment (e.g., the final setup). The pose of the tooth may include a change in position in 3D space and may also include a change in orientation (e.g., with respect to one or more coordinate axes - e.g., local coordinate axes with origin at the crown centroid). The transform may affect the change in orientation by pivoting the tooth mesh relative to a pivot point or tooth origin. This pivot point may be chosen to he within the crown centroid. Alternatives include at the apex of the root tip, origin of malocclusion transform or at a point along an archform in proximity to the tooth.

[0072] In some implementations, the setups prediction neural network may be trained conditionally on interproximal reduction (IPR) information. IPR may be applied to the teeth, to enable greater packing of teeth a in final setup. The setups model may be trained to account to IPR quantities (e.g., millimeters of offset in from either or both of the mesial and distal sides of a tooth) and / or IPR cut planes (which may be used in conjunction with mesh Boolean operations to remove material on either or both of the mesial and distal sides of a tooth). For example, IPR cut planes may be used to modify one or more tooth meshes for one or more patient cases which are used to train the setups prediction model. This step improves the accuracy of setups prediction model training by improving data precision, because material is removed from the teeth which may otherwise lead to collisions between teeth in the final setup (and result in noise in the training data). After the trained setups prediction model is deployed, IPR may be applied to a trial patient case, to modify the shapes of the teeth before the case is received as input to the setups prediction model. In some instances, IPR may be applied to one or more tooth meshes of a patient case before the computation of orthodontic metrics.

[0073] Systems of this disclosure may, in some instances, be deployed in a clinical context (such as a dental or orthodontic office) for use by clinicians (e.g., doctors, dentists, orthodontists, nurses, hygienists, oral care technicians). Such systems which are deployed in a clinical context may enable clinicians to process oral care data (such as dental scans) in the clinical environment, or in some instances, in a “chairside” context (where the patient is present in the clinical environment), an example of which is shown in method 1400 of FIG. 14. As should be appreciated by one of ordinary skill in the art, performing method 1400 while a patient is still in a clinical or “chairside” context (or environment) may generally require that method 1400 be performed in realtime or near-real-time so that results can be generated while the patient is in the clinical environment.

[0074] The method 1400 begins (1418) with the intraoral scanning (1402) of the patient, which may generate 3D meshes of the patient’s tooth crowns and / or tooth roots. Other types of patient dentition data may be collected, as well, such as volumetric cone beam computed tomography (CBCT) data, or digital photographs of the patient’s face and / or dentition. The patient’s dentition data may undergo mesh cleanup and / or mesh segmentation (1404), and then be provided (1406) to an automated predictive ML model (e.g., automated setups prediction, IPR prediction, restoration design generation, smile prediction, or setups prediction which also predicts information pertaining to IPR). In some implementations, the outputs of the automated predictive model may be provided to a smile prediction module, which may generate (1410) one or more predicted smiles. Whenthe automated predictive model does not predict the use of IPR (1408), then the method proceeds to completion (1416). When the automated predictive model does predict the use of IPR (1408), then the one or more IPR predictions 1412 are used to influence physical clinical treatment (1414) of the patient. For example, automated setups prediction may generate (1406) one or more IPR predictions 1412 (e.g., IPR cut surfaces for one or more teeth, or one or more determinations that IPR is to be performed for particular teeth, or others described herein), which may then influence physical clinical treatment ( 1414) of the patient while the patient waits in the treatment chair. After the application of IPR (1414) to one or more of the patient’s teeth, the patient’s teeth again undergo intraoral scanning (1402), and the method iterates. The application of IPR to one or more of the patient’s teeth may enable final setups generation (1406) to output tooth transforms which place the teeth in improved poses relative to the prior iteration of method 1400, because the teeth have had enamel removed, enamel which may previously have complicated tooth alignments (e.g., as measured by the Alignment oral care metric, or other oral care metrics described herein). Because all of these steps can occur while patient is in the treatment chair, the techniques described herein are described to provide near-real time feedback (e.g., instructions on which teeth require IPR, which side of applicable teeth - mesial or distal, and / or instructions regarding the amount of IPR to apply, etc.) to the clinician.

[0075] Various loss calculation techniques are generally applicable to the techniques of this disclosure. The losses include L 1 loss, L2 loss, mean squared error (MSE) loss, cross entropy loss, among others. Losses may be computed and used in the training of neural networks, such as multi-layer perceptron’s (MLP), U-Net structures, generators and discriminators (e.g., for GANs), autoencoders, variational autoencoders, regularized autoencoders, masked autoencoders, transformer structures, or the like. Some implementations may use either triplet loss or contrastive loss, for example, in the learning of sequences.

[0076] Losses may also be used to train encoder structures, or decoder structures, among others. A KL- Divergence loss may be used, at least in part, to train one or more of the neural networks of the present disclosure, such as a mesh reconstruction autoencoder in FIG. 11, with the advantage of imparting Gaussian behavior to the optimization space. This Gaussian behavior may enable a reconstruction autoencoder to produce a better reconstruction (e.g., when a latent vector representation is modified and that modified latent vector is reconstructed using a decoder, the resulting reconstruction is more likely to be a valid instance of the provided representation). There are other techniques for computing losses which may be described elsewhere in this disclosure. Such losses may be based on quantifying the difference between two or more 3D representations.

[0077] MSE loss calculation may involve the calculation of an average squared distance between two sets, vectors or datasets. MSE may be generally minimized. MSE may be applicable to a regression problem, where the prediction generated by the neural network or other machine learning model may be a real number. In some implementations, a neural network may be equipped with one or more linear activation units on theoutput to generate an MSE prediction. Mean absolute error (MAE) loss and mean absolute percentage error (MAPE) loss can also be used in accordance with the techniques of this disclosure.

[0078] Cross entropy may, in some implementations, be used to quantify the difference between two or more distributions. Cross entropy loss may, in some implementations, be used to train the neural networks of the present disclosure. Cross entropy loss may, in some implementations, involve comparing a predicted probability to a ground truth probability. Other names of cross entropy loss include “logarithmic loss,” “logistic loss,” and “log loss”. A small cross entropy loss may indicate a better (e.g., more accurate) model. Cross entropy loss may be logarithmic. Cross entropy loss may, in some implementations, be applied to binary classification problems. In some implementations, a neural network may be equipped with a sigmoid activation unit at the output to generate a probability prediction. In the case of multi-class classifications, cross entropy may also be used. In such a case, a neural network trained to make multi-class predictions may, in some implementations, be equipped with one or more softmax activation functions at the output (e.g., where there is one output node for class that is to be predicted). Other loss calculation techniques which may be applied in the training of the neural networks of this disclosure include one or more of: Huber loss, Hinge loss, Categorical hinge loss, cosine similarity, Poisson loss, Logcosh loss, or mean squared logarithmic error loss (MSLE). Other loss calculation methods are described herein and may be applied to the training of any of the neural networks described in the present disclosure.

[0079] One or more of the neural networks of the present disclosure may, in some implementations, be trained, at least in part by a loss which is based on at least one of: a Point-wise Mesh Euclidean Distance (PMD) and an Earth Mover’s Distance (EMD). Some implementations may incorporate a Hausdorff Distance (HD) calculation into the loss calculation. Computing the Hausdorff distance between two or more 3D representations (such as 3D meshes) may provide one or more technical improvements, in that the HD not only accounts for the distances between two meshes, but also accounts for the way that those meshes are oriented, and the relationship between the mesh shapes in those orientations (or positions or poses). Hausdorff distance may improve the comparison of two or more tooth meshes, such as two or more instances of a tooth mesh which are in different poses (e.g., such as the comparison of predicted setup to ground truth setup which may be performed in the course of computing a loss value for training a setups prediction neural network).

[0080] Reconstruction loss may compare a predicted output to a ground truth (or reference) output. Systems of this disclosure may compute reconstruction loss as a combination of LI loss and MSE loss, as shown in the following line of pseudocode: reconstruction loss = 0.5*Ll(all_points_target,all_points_predicted) + 0.5*MSE(all_points_target,all_points_predicted). In the above example, all_points_target is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to ground truth data (e.g., a ground truth tooth restoration design, or a ground truth example of some other 3D oral care representation). In the above example, all joints jredicted is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to generated or predicted data (e.g., a generated tooth restoration design, or agenerated example of some other kind of 3D oral care representation). Other implementations of reconstruction loss may additionally (or alternatively) involve L2 loss, mean absolute error (MAE) loss or Huber loss terms.

[0081] Reconstruction error may compare reconstructed output data (e.g., as generated by a reconstruction autoencoder, such as a tooth design which has been generated for use in generating a dental restoration appliance) to the original input data (e.g., the data which were provided to the input of the reconstruction autoencoder, such as a pre-restoration tooth). Systems of this disclosure may compute reconstruction error as a combination of LI loss and MSE loss, as shown in the following line of pseudocode: reconstruction_error = 0.5*Ll(all_points_input, all join ts rcconstruc ted) + 0.5*MSE(all_points_input, all_points_reconstructed). As above, all_points_input is a 3D representation (e.g., a 3D mesh or point cloud) corresponding to input data (e.g., the pre-restoration tooth design which was provided to a reconstruction autoencoder, or another 3D oral care representation which is provided to the input of an ML model). As above, all_points_reconstructed is a 3D representation (e.g., 3D mesh or point cloud) corresponding to reconstructed (or generated) data (e.g., a reconstructed tooth restoration design, or another example of a generated 3D oral care representation).

[0082] Some implementations may incorporate a Chamfer Distance (CD) calculation into the loss, which measures the squared distance between each point in one set of mesh elements and its nearest neighbor in another set of mesh elements. Chamfer Distance may, in some implementations, enable a differentiable method of comparing mesh element sets (such as set of vertices). Some implementations may incorporate an Earth mover’s distance (EMD) into the loss, which measures the squared distance between two sets of mesh elements. Some implementations may incorporate a Point-wise Mesh Euclidean Distance (PMD) calculation into the loss calculation. Some implementations may incorporate a Hausdorff Distance (HD) calculation into the loss calculation.

[0083] Representation loss has two components. One component is related to rotation and the other component is related to translation of the tooth. Each component is directly calculated on the coordinate system representation, for example the 3x3 rotation matrix and the 3x1 translation vector. A 4x3 matrix is formed out of the 3x3 and the 1x3 matrices. The distance is calculated as the difference between the ground truth and predicted transform.

[0084] The predicted 3x3 matrix is subtracted from the ground truth 3x3 rotation matrices, and either the LI norm or L2 norm of the result as the rotation loss is computed. The predicted 1x3 matrix is also subtracted from the 1x3 ground truth translation vector, and similarly either the LI norm or the L2 norm of the result as the translation loss is computed. The total loss is a weighted average or summation of the rotation and translation losses.

[0085] The techniques of this disclosure may include operations such as 3D convolution, 3D pooling, 3D unconvolution and 3D unpooling. 3D convolution may aid segmentation processing, for example in down sampling a 3D mesh. 3D un-convolution undoes or reverts a 3D convolution, for example, in a U-Net. 3Dpooling may aid the segmentation processing, for example in summarized neural network feature maps. 3D unpooling undoes 3D pooling, for example in a U-Net. These operations may be implemented by way of one or more layers in the predictive or generative neural networks described herein. These operations may be applied directly on mesh elements, such as mesh edges or mesh faces. These operations provide for technical improvements over other approaches because the operations are invariant to mesh rotation, scale, and translation changes. In general, these operations depend on edge (or face) connectivity, therefore these operations remain invariant to mesh changes in 3D space as long as edge (or face) connectivity is preserved. That is, the operations may be applied to an oral care mesh and produce the same output regardless of the orientation, position or scale of that oral care mesh, which may lead to data precision improvement. MeshCNN is a general-purpose deep neural network library for 3D triangular meshes, which can be used for tasks such as 3D shape classification or mesh element labelling (e.g., for segmentation or mesh cleanup). MeshCNN implements these operations on mesh edges. Other toolkits and implementations may operate on edges or faces.

[0086] A 3D representation may be produced using a 3D scanner, such as an intraoral scanner, a computerized tomography (CT) scanner, ultrasound scanner, a magnetic resonance imaging (MRI) machine or a mobile device which is enabled to perform stereophotogrammetry. A 3D representation may describe the shape and / or structure of a subject. A 3D representation may include one or more 3D mesh, 3D point cloud, and / or a 3D voxelized representation, among others. A 3D mesh includes edges, vertices, or faces. Though interrelated in some instances, these three types of data are distinct. The vertices are the points in 3D space that define the boundaries of the mesh. These points would alternatively be described as a point cloud but for the additional information about how the points are connected to each other, as described by the edges. An edge is described by two points and can also be referred to as a line segment. A face is described by a number of edges and vertices. For instance, in the case of a triangle mesh, a face comprises three vertices, where the vertices are interconnected to form three contiguous edges. Some meshes may contain degenerate elements, such as nonmanifold mesh elements, which may be removed, to the benefit of later processing. Other mesh pre-processing operations are possible in accordance with aspects of this disclosure. 3D meshes are commonly formed using triangles, but may in other implementations be formed using quadrilaterals, pentagons, or some other n-sided polygon. In some implementations, a 3D mesh may be converted to one or more voxelized geometries (i.e., comprising voxels), such as in the case that sparse processing is performed. The techniques of this disclosure which operate on 3D meshes may receive as input one or more tooth meshes (e.g., arranged in one or more dental arches). Each of these meshes may undergo pre-processing before being input to the predictive architecture (e.g., including at least one of an encoder, decoder, pyramid encoder-decoder and U-Net). This pre-processing may include the conversion of the mesh into lists of mesh elements, such as vertices, edges, faces or in the case of sparse processing - voxels. For the chosen mesh element type or types, (e.g., vertices), feature vectors may be generated. In some examples, one feature vector is generated per vertex of the mesh.Each feature vector may contain a combination of spatial and / or structural features, as specified in the following table:Table 2

[0087] Table 2 discloses non-limiting examples of mesh element features. In some implementations, color (or other visual cues / identifiers) may be considered as a mesh element feature in addition to the spatial or structural mesh element features described in Table 2. As used herein (e.g., in Table 2), a point differs from a vertex in that a point is part of a 3D point cloud, whereas a vertex is part of a 3D mesh and may have incident faces or edges. A dihedral angle (which may be expressed in either radians or degrees) may be computed as the angle (e.g., a signed angle) between two connected faces (e.g., two faces which are connected along an edge). A sign on a dihedral angle may reveal information about the convexity or concavity of a mesh surface. For example, a positively signed angle may, in some implementations, indicate a convex surface. Furthermore, a negatively signed angle may, in some implementations, indicate a concave surface. To calculate the principal curvature of a mesh vertex, directional curvatures may first be calculated to each adjacent vertex around the vertex. These directional curvatures may be sorted in circular order (e.g., 0, 49, 127, 210, 305 degrees) in proximity to the vertex normal vector and may comprise a subsampled version of the complete curvature tensor. Circular order means: sorted in by angle around an axis. The sorted directional curvatures may contribute to a linear system of equations amenable to a closed form solution which may estimate the two principal curvatures and directions, which may characterize the complete curvature tensor. Consistent with Table 2, a voxel may also have features which are computed as the aggregates of the other mesh elements (e.g., vertices, edges and faces) which either intersect the voxel or, in some implementations, are predominantly or folly contained within the voxel. Rotating the mesh may not change structural features but may change spatial features. And, as described elsewhere in this disclosure, the term “mesh” should be considered in a nonlimiting sense to be inclusive of 3D mesh, 3D point cloud and 3D voxelized representation. In some implementations, apart from mesh element features, there are alternative methods of describing the geometry of a mesh, such as 3D keypoints and 3D descriptors. Examples of such 3D keypoints and 3D descriptors are found in “TONI ONI A, et al. in ‘Learning to detect good 3D keypoints.’, Int J Comput. Vis. 2018 Vol .126, pages 1-20.”. 3D keypoints and 3D descriptors may, in some implementations, describe extrema (either minima or maxima) of the surface of a 3D representation. In some implementations, one or more mesh element features may be computed, at least in part, via deep feature synthesis (DFS), e.g. as described in: J. M. Kanter and K. Veeramachaneni, "Deep feature synthesis: Towards automating data science endeavors," 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2015, pp. 1-10, doi: 10. 1109 / DSAA.2015.7344858.

[0088] Representation generation neural networks based on autoencoders, U-Nets, transformers, other types of encoder-decoder structures, convolution and / or pooling layers, or other models may benefit from theuse of mesh element features. Mesh element features may convey aspects of a 3D representation’s surface shape and / or structure to the neural network models of this disclosure. Each mesh element feature describes distinct information about the 3D representation that may not be redundantly present in other input data that are provided to the neural network. For example, a vertex curvature may quantify aspects of the concavity or convexity of the surface of a 3D representation which would not otherwise be understood by the network. Stated differently, mesh element features may provide a processed version of the structure and / or shape of the 3D representation, data that would not otherwise be available to the neural network. This processed information is often more accessible, or more amenable for encoding by the neural network. A system implementing the techniques disclosed herein has been utilized to run a number of experiments on 3D representations of teeth. For example, mesh element features have been provided to a representation generation neural network which is based on a U-Net model, and also to a representation generation model based on a variational autoencoder with continuous normalizing flows. Based on experiments, it was found that systems using a foil complement of mesh element features (e.g., “XYZ” coordinates tuple, “Normal vector”, “Vertex Curvature”, Points-Pivoted, and Normals-Pivoted) were at least 3% more accurate than systems that did not. Points-Pivoted describes “XYZ” coordinates tuples that have local coordinate systems (e.g., at the centroid of the respective tooth). Normals-Pivoted describes “Normal Vectors” which have local coordinate systems (e.g., at the centroid of the respective tooth). Furthermore, training converges more quickly when the full complement of mesh element features are used. Stated another way, the machine learning models trained using the full complement of mesh element features tended to be more accurate more quickly (at earlier epochs) than systems which did not. For an existing system observed to have a historical accuracy rate of 91%, an improvement in accuracy of 3% reduces the actual error rate by more than 30%.

[0089] In some implementations, one or more orthodontic metrics may be used to evaluate the predicted output of a neural network, such as a setups prediction. Such a metric(s) may enable the training algorithm to determine how close the predicted output is to an acceptable output, for example, in a quantified sense. In some implementations, this use of an orthodontic metric may enable a loss value to be computed which does not depend entirely on a comparison to a ground truth. In some implementations, such a use of an orthodontic metric may enable loss calculation and network training to proceed without the need for a comparison against a ground truth example. The advantage of such an approach is that loss may be computed based on a general principle or specification for the predicted output (such as a setup) rather than tying loss calculation to a specific ground truth example (which may have been defined by a particular doctor, clinician, or technician, whose treatment philosophy may differ from that of other technicians or doctors). In some implementations, such an orthodontic metric may be defined based on a FID (Frechet Inception Distance) score.

[0090] The following is a description of some of the orthodontic metrics which are used to quantify the state of a set of teeth in an arch for the purpose of orthodontic treatment. These orthodontic metrics indicate the degree of malocclusion that the teeth are in at a given stage of clear tray aligner treatment.

[0091] An orthodontic metric that can be computed using tensors may be especially advantageous when training one of the neural networks of the present disclosure, because tensor operations may promote efficient computations. The more efficient (and faster) the computation, the faster the rate at which training can proceed.

[0092] In some examples, an error pattern may be identified in one or more predicted outputs of an ML model (e.g., a transformation matrix for a predicted tooth setup, a labelling of mesh elements for mesh cleanup, an addition of mesh elements to a mesh for the purpose of mesh in-filling, a classification label for a setup, a classification label for a tooth mesh, etc.). One or more orthodontic metrics may be selected to become an input to the next round of ML model training, to address any pattern of errors or deficiencies which may be identified in the one or more predicted outputs.

[0093] Some OM may be defined relative to an archform coordinate frame, the LDE coordinate system. In some implementations, a point may be described using an LDE coordinate frame relative to an archform, where L, D and E correspond to: 1) Length along the curve of the archform, 2) Distance away from the archform, and 3) distance in the direction perpendicular to the L and D axes (which may be termed Eminence), respectively.

[0094] Various of the OM and other techniques of the present disclosure may compute collisions between 3D representations (e.g., of oral care objects, such as teeth). Such collisions may be computed as at least one of: 1) penetration distance between 3D tooth representations, 2) count of overlapping mesh elements between 3D tooth representations, and 3) volume of overlap between 3D tooth representations. In some implementations, an OM may be defined to quantify the collision of two or more 3D representations of oral care structures, such as teeth. Some optimization algorithms, such as setups prediction techniques, may seek to minimize collisions between oral care structures (such as teeth).Between-arch orthodontic metrics are as follows.

[0095] Six (6) metrics for the comparison of two or more arches are listed below. Other suitable comparison orthodontic metrics are found elsewhere in this disclosure, such as in the section for the Setups Comparison technique.1. Rotation geodesic distance (rotation between predicted example and ground truth setup example)2. Translation distance (gap between predicted example and ground truth setup example)3. Normalized translation distance4. 3D alignment error that measures the distance between predicted mesh elements and ground truth mesh elements, in units of mm.5. Normalized 3D alignment6. Percent overlap (% overlap) by volume (alternatively % overlap by mesh elements) of predicted example and corresponding ground truth example

[0096] Within-arch orthodontic metrics are as follows.

[0097] Alignment - A 3D tooth orientation vector may be calculated using the tooth's mesial-distal axis. A 3D vector, which may be tangent vector to the archform at the position of the tooth may also be calculated. The XY components (i.e., which may be 2D vectors) may then be used to compare the orientation of the archform at the tooth's location to the tooth's orientation in XY space. Cosine similarity may be used to calculate the 2D orientation difference (angle) between the archform tangent and the tooth's mesial-distal axis.

[0098] Arch Symmetry - For each left-right pair of teeth (e.g., lower left lateral incisor and / or lower right lateral incisor) the absolute difference may be calculated between each tooth’s X-coordinate and the global coordinate reference frame’s X-axis. This delta may indicate the arch asymmetry for a given tooth pair. The result of such a calculation may be the mean X-axis delta of one or more tooth-pairs from the arch. This calculation may, in some implementations, be performed relative to the Y-axis with y-coordinates (and / or relative to the Z axis with Z-coordinates).

[0099] Archform D-axis Differences - May compute the D dimension difference (i.e., the positional difference in the facial-lingual direction) between two arch states, for one or more teeth. May, in some implementations, return a dictionary of the D-direction tooth movement for each tooth, with tooth UNS number as the key. May use the LDE coordinate system relative to an archform.

[0100] Archform (Lower) Length Ratio - May compute the ratio between the current lower arch length and the arch length as it was in the original maloccluded lower arch.

[0101] Archform (Upper) Length Ratio - May compute the ratio between the current upper arch length and the arch length as it was in the original maloccluded upper arch.

[0102] Archform Parallelism (Full arch) - For at least one local tooth coordinate system origin in the upper arch, the one or more nearest origins (e.g., tooth local coordinate system origins) in the lower arch. In some implementations, the two nearest origins may be used. May compute the straight line distance from the upper arch point to the line formed between the origins of the two teeth in the opposing (lower) arch. May return the standard deviation of the set of “point-to-line" distances mentioned above, where the set may be composed of the point-to-line distances for each tooth in the arch.

[0103] Archform Parallelism (Individual tooth) - This metric may share some computational elements with the archform _parallelism_global orthodontic metric, except that this metric may input the mean distance from a tooth origin to the line formed by the neighboring teeth in opposing arches (e.g., a tooth in the upper arch and the corresponding tooth in the lower arch). The mean distance may be computed for one or more such pairs of teeth. In some implementations, this may be computed for all pairs of teeth. Then the mean distance may be subtracted from the distance that is computed for each tooth pair. This OM may yield the deviation of a tooth from a “typical” tooth parallelism in the arch.

[0104] Buccolingual Inclination - For at least one molar or premolar, find the corresponding tooth on the opposite side of the same arch (i.e., for a tooth on the left side of the arch, find the same type of tooth on the right side and vice versa). This OM may compute an n-element list for each tooth (e.g. n may equal 2). Thislist may contain at least the tooth IDs of the teeth in each pair of teeth (e.g., LeftLowerFirstMolar and RightLowerFirstMolar in a list = [left_tooth_idx_l, right_tooth_idx_2]). Such an n-element vector may be computed for each molar and each premolar in the upper and lower arches. The buccal cusps may be identified on the molars and premolars on each of the left and right sides of the arch. Draw a line between the buccal cusps of the left tooth and the buccal cusps on the right tooth. Make a plane using this line and the z-axis of the arch. The lingual cusps may be projected onto the plane (i.e., at this point the angle of inclination may be determined). By performing an additional projection, the approximate vertical distance between the lingual cusps and the buccal cusps may be computed. This distance may be used as the buccolingual inclination OM.

[0105] Canine Overbite - The upper and lower canines may be identified. The first premolar for the given side of the mouth may be identified. On a given side of the arch, a distance may be computed between the upper canine and the lower canine, and also between the upper pre-molar and the lower pre-molar. The average (or median, or mode or some other statistic) may be computed for the measured distances. The z- component of this result indicates the degree of overbite. Overbite may be computed between any tooth in one arch and the corresponding tooth in the other arch.

[0106] Canine Overjet Contact - May calculate the collisions (e.g., collision distances) between pairs of canines on opposing arches.

[0107] Canine Overjet Contact KDE - May take an orthodontic metric score for the current patient case as input and may convert that score into to a log-likelihood using a previously trained kernel density estimation (KDE) model or distribution. This operation may yield information about where in the distribution of "typical" values this patient case lies.

[0108] Canine Overjet - This OM may share some computational steps with the canine overbite OM. In some implementations, average distances may be computed. In some implementations, the distance calculation may compute the Euclidean distance of the XY components of a tooth in the upper arch and a tooth in the lower arch, to yield oveijet (i.e., as opposed to computing the difference in Z-components, as may be performed for canine overbite). Overjet may be computed between any tooth in one arch and the corresponding tooth in the other arch.

[0109] Canine Class Relationship (also applies to first, second and third molars) - This OM may, in some implementations comprise two functions (e.g., written in Python). get_canine_landmarks(): Get landmarks for each tooth which may be used to compute the class relationship, and then, in some implementations, map those landmarks onto the global coordinate space so that measurements may be made between teeth. class_relationship_score_by_side(): May compute the average position of at least one landmark on at least one tooth in the lower arch, and may compute the same for the upper arch. Then may compute the vector from the upper arch landmark position to the lower arch landmark position, and finally projects this vector onto thelower arch to yield a quantification (e.g., as a scalar) of the amount of delta in “arch 1-axis" position there is. This OM may compute how far forward or behind the tooth is positioned on the 1-axis relative to the tooth or teeth of interest in the opposing arch.

[0110] Crossbite - Fossa in at least one upper molar may be located by finding the halfway point between distal and mesial marginal ridge saddles of the tooth. A lower molar cusp may he between the marginal ridges of the corresponding upper molar. This OM may compute a vector from the upper molar fossa midpoint to the lower molar cusp. This vector may be projected onto the d-axis of the archform, yielding a lateral measure of distance from the cusp to the fossa. This distance may define the crossbite magnitude.

[0111] Edge Alignment - This OM may identify the leftmost and rightmost edges of a tooth and may identify the same for that tooth’s neighbor.The OM may then draw a vector from the leftmost edge of the tooth to the leftmost edge of the tooth’s neighbor.The OM may then draw a vector from the rightmost edge of the tooth to the rightmost edge of the tooth’s neighbor.The OM may then calculates the linear fit error between the two vectors.Such a calculation may involve making two vectors:Vec tooth = right_tooths_leftside to left_tooths_leftsideVec_neighbor = right_tooths_rightside to left_tooths_leftsideAnd then may involve computing the dot-product of these two vectors and subtracting the result from 1. (i.e., EdgeAlignment score = 1 - abs(dot(Vec_tooth, Vec_neighbor)) ).A score of 0 may indicate perfect alignment. A score of 1 may mean perpendicular alignment.

[0112] Incisor Interarch Contact KDE - May identify the deviation of theIncisorlnterarchContact from the means of a modeled distribution of such statistics across a dataset of one or more other patient cases.

[0113] Leveling - May compute a measure of leveling between a tooth and its neighbor. This OM may calculate the difference in height between two or more neighboring teeth. For molars, this OM may use the midpoint between the mesial and distal saddle ridges as the height of the molar. For non-molar teeth, this OM may use the length of the crown from gums to tip. In some implementations, the tip may be the origin of the local coordinate space of the tooth. Other implementations may place the origin in other locations. A simple subtraction between the heights of neighboring teeth may yield the leveling delta between the teeth (e.g., by comparing Z components).

[0114] Midline - May compute the position of the midline for the upper incisors and / or the lower incisors, and then may compute the distance between them.

[0115] Molar Interarch Contact KDE - May compute a molar interarch contact score (i.e., a collision depth or other type of collision), and then may identify where that score lies in a pre-defined KDE (distribution) built from representative cases.

[0116] Occlusal Contacts - For a particular tooth from the arch, this OM may identify one or more landmarks (e.g., mesial cusp, or central cusp, etc.). Get the tooth transform for that tooth. For each cusp on the current tooth, the cusp may be scored according to how well the cusp contacts the neighboring (corresponding) tooth in the opposite arch. A vector may be found from the cusp of the tooth in question to the vertical intersection point in the corresponding tooth of the opposing arch. The distance and / or direction (i.e., up or down) to the opposing arch may be computed. A list may be returned that contains the resulting signed distances, one for each cusp on the tooth in question.

[0117] Overbite - The upper and lower central incisors may be compared along the z-axis. The difference along the z-axis may be used as the overbite score.

[0118] Overjet - The upper and lower central incisors may be compared along the y-axis. The difference along the y-axis may be used as the overjet score.

[0119] Molar Interarch Contact - May calculate the contact score between molars and may use collision measurement(s) (such as collision depth).

[0120] Root Movement d - The tooth transforms for an initial state and a next state may be recieved. The archform axes at a point L along the archform may be computed. This OM may return a distance moved along the d-axis. This may be accomplished by projecting the root pivot point onto the d-axis.

[0121] Root Movement 1 - The tooth transforms for an initial state and a next state may be received. The archform axes at a point L along the archform may be computed. This OM may return a distance moved along the 1-axis. This may be accomplished by projecting the root pivot point onto the 1-axis.

[0122] Spacing - May compute the spacing between each tooth and its neighbor. The transforms and meshes for the arch may be received. The left and right edges of each tooth mesh may be computed. One or more points of interest may be transformed from local coordinates into the global arch coordinate frame.The spacing may be computed in a plane (e.g., the XY plane) between each tooth and its neighbor to the "left". May return an array of one or more Euclidean distances (e.g., such as in the XY plane) which may represent the spacing between each tooth and its neighbor to the left.

[0123] Torque - May compute torque (i.e., rotation around and axis, such as the x-axis). For one or more teeth, one or more rotations may be converted from Euler angles into one or more rotation matrices. A component (such as a x-component) of the rotations may be extracted and converted back into Euler angles. This x-component may be interpreted as the torque for a tooth. A list may be returned which contains the torque for one or more teeth and may be indexed by the UNS number of the tooth.

[0124] In some implementations, an automated setups prediction model may be trained to generate a setup with a customized curve-of-spee (e.g., a curve-of-spee which conforms to the intended outcome of the treatment of the patient). Such a model may be trained on cohort patient case data. One or more oral care metrics may be computed on each case to quantify or measure aspects of that case's curve-of-spee. At training time, one or more of such metrics may be provided to the setups prediction model, for example, to influence the model regarding the geometry and / or structure of each case's curve-of-spee. Upon deployment of the setups prediction model, that same input pathway to the trained neural network may be configured with one or more values as instructions to the model about an intended curve-of-spee. Such values may automatically generate a setup with a curve-of-spee which meets the aesthetic and / or medical treatment needs of the particular patient case.

[0125] In some implementations, a curve-of-spee metric may measure the curvature of the occlusal or incisal surfaces of the teeth on either the left or right sides of the arch, with respect to the occlusal plane. The occlusal plane may, in some instances, be computed as a surface which averages the incisal or occlusal surfaces of the teeth (for one or both arches). In some implementations, a curvature metric may be computed along a normal vector, such as a vector which is normal to the occlusal plane. In other implementations, a curvature metric may be computed along the normal vector of another plane. In some implementations, an XY plane may be defined to correspond to the occlusal plane. An orthogonal plane may be defined as the plane that is orthogonal to the occlusal plane, which also passes through a curve-of-spee line segment, where the curve-of-spee line segment is defined by a first endpoint which is a landmarking point on a first tooth (e.g., canine) and a second endpoint which is a landmarking point on the most-posterior tooth of the same side of the arch. A landmarking point can in some implementations be located along the incisal edge of a tooth or on the cusp of a tooth. In some instances, the landmarking points for the intermediate teeth (e.g., teeth which are located between the first tooth and the most posterior tooth) on either the left or right sides of the arch may form a curved path, such as may be described by a polyline. The following is a non-limiting list of curve-of- spee oral care metrics.

[0126] 1) Measure the vertical height between a line segment and a point. Stated another way, measure a distance between a line segment and a point along the z-axis. The line segment is defined by joining the highest cusp of the most-posterior tooth (in the lower arch) and the cusp of the first tooth on that side (in the lower arch). Given the subset of teeth between the first tooth and the most-posterior tooth, the point is defined by the highest cusp of the lowest tooth of this subset. Stated another way, a curve-of-spee metric may be computed using the following 4 steps, i) Line: Form a line between the highest cusp on the most posterior tooth and the cusp of the first tooth, ii) Curve_Point_A: Given the set of teeth between the most posterior tooth and the first tooth, find the highest point of the lowest tooth, iii) Curve Point B: Project Curve Point A onto the Line to find a point (Curve Point B) along the line that is closest to Curve Point A. iv) Curve-Of-Spee: Find the height difference between Curve Point B and Curve Point A.

[0127] 2) Project one or more intermediate landmark points (e.g., points on the teeth which he between the first tooth and the most-posterior tooth on that side of the arch) and the curve-of-spee line segment onto the orthogonal plane. Compute the curve-of-spee metric by measuring the distance between the farthest of the projected intermediate points to the projected curve-of-spee line segment. This yields a measure for the curvature of the arch relative to the orthogonal plane.

[0128] 3) Project one or more intermediate landmark points and the curve-of-spee line segment onto the occlusal plane. Compute Curve of Spee in this plane by measuring the distance between the farthest of the projected intermediate points to the projected curve-of-spee line segment. This yields a measure for the curvature of the arch relative to the occlusal plane.

[0129] 4) Skip the projection and compute the distances and curvatures in the 3D space. Compute Curve of Spee by measuring the distance between the farthest of the intermediate points to the curve-of-spee line segment. This yields a measure for the curvature of the arch in 3D space.

[0130] 5) Compute the slope of the projected curve-of-spee line segment on the occlusal plane.

[0131] 6) Compute the slope of the projected curve-of-spee line segment in the orthogonal plane.

[0132] Curve-of-spee metrics 5 and 6 may help the network to reduce some more degrees of freedom in defining how the patient’s arch is curved in the posterior of the mouth.

[0133] In experiments, during training of a setups prediction model, the ground truth (or reference) setup was registered to the malocclusion (or maloccluded setup). The maloccluded teeth were provided to the setups prediction model, which generated final setup transforms for the maloccluded teeth. Loss was computed between the resulting predicted setup and the pre-registered ground truth setup, so that corresponding aspects of the two setups would line-up. The result was a more accurate loss calculation. This pre-registration operation resulted in a 6% improvement in absolute accuracy (e.g., as measured by ADD 10 score), which amounts to a reduction in error rate of nearly 50% compared with conventional techniques.

[0134] Examples of oral care metrics include Orthodontic Metrics (OM) and Restoration Design Metrics (RDM). RDM may describe the shape and / or form of one or more 3D representations of teeth (e.g., for use in IPR calculations for orthodontics, or for use in dental restoration design generation). Some RDM may quantify the shape and / or other characteristics of a tooth. Other RDM may quantify relationships (e.g., spatial relationships) between two or more teeth.

[0135] Techniques of this disclosure may, in some implementations, obtain on or more of the following examples of data for use in an oral care metrics calculation (e.g., orthodontic metrics or RDM): 1) a digital 3D model of one or more teeth in a pre-restoration state; 2) a digital 3D model one or more teeth in a postrestoration state; 3) a digital 3D model of one or more neighboring teeth in a pre-restoration state; 4) a digital 3D model of one or more neighboring teeth in a post-restoration state; 5) position information associated with one or more neighboring teeth from the 3D digital model; or 6) landmark information associated with one or more neighboring teeth from the 3D digital model.

[0136] Non-limiting examples of inter-tooth RDM are enumerated below.

[0137] 1) Bilateral Symmetry and / or Ratios: A measure of the symmetry between one or more teeth and one or more other teeth on opposite sides of the dental. For example, for a pair of corresponding teeth, a measure of the width of each tooth. In one instance, the one tooth is of normal width, and the other tooth is too narrow. In another instance, both teeth are of normal width. The following is a list of attributes that can be measured for a tooth, and compared to the corresponding measurement for one or more corresponding teeth: a) width - mesial to distal distance; b) length - gingival to incisal distance; c) diagonal - distance across the tooth, e.g., from the mesial gingival comer to the distal incisal comer (this measure is one of many that can be used to quantify the shape of teeth beyond length and width). Ratios between a and b may be computed, such as a / b or b / a. Such ratios can be indicative of whether spatial symmetry exists (e.g., by measuring the ratio a / b on the left side and measuring the ratio a / b on the right side, then compare the left and right ratios). In some implementations, where spatial symmetry is "off1, the length, width and / or ratios may not match. Such a ratio may, in some implementations, be computed relative to a standard. A number of esthetic standards are available in the dental literature. Examples include Golden Proportion and Recurring Esthetic Dental Proportion. In some implementations, spatial symmetry may be measured on a pair of teeth, where one tooth is on the right side of the arch, and the other tooth is on the left side of the arch.

[0138] 2) Proportions of Adjacent Teeth: Measure the width proportions of adjacent teeth as measured as a projection along an arch onto a plane (e.g., a plane that is situated in front of the patient's face). The ideal proportions for use in the final restoration design can be, for example, the so-called golden proportions. The golden proportions relate to adjacent teeth, such as central incisors and lateral incisors. This metric pertains to the measuring of these proportions as the proportions exist in the pre-restoration mal dentition. The ideal golden proportions are 1.6, 1, 0.6, for the central incisor, lateral incisor and cuspid, on a particular side (either left or right) for a particular arch (e.g., the upper arch). If one or more of these proportion values is off (e.g., in the case of "peg laterals"), the patient may wish for dental restoration treatment to correct the proportions.

[0139] 3) Arch Discrepancies: A measure of any size discrepancies between the upper arch and lower arch, for example, pertaining to the widths of the teeth, for the purpose of dental restoration. For example, techniques of this disclosure may make adjacent tooth width proportion measurements in the upper arch and in the lower arch. In some implementations, Bolton analysis measurements may be made by measuring upper widths, lower widths, and proportions between those quantities. Arch discrepancies may be described in absolute measurements (e.g., in mm or other suitable units) or in terms of proportions or ratios, in various implementations.

[0140] 4) Midline: A measure of the midline of the maxillary incisors, relative to the midline of the mandibular incisors. Techniques of this disclosure may measure the midline of the maxillary incisors, relative to the midline of the nose (if data about nose location is available).

[0141] 5) Proximal Contacts: A measure of the size (area, volume, circumference, etc.) of the proximal contact between adjacent teeth. In the ideal circumstance, the teeth touch along the mesial / distal surfaces and the gums fill in gingivally to where the teeth touch. Black triangles may form if the gum tissue fails to fill the space below the proximal contact. In some instances, the size of the proximal contact may get progressively shorter for teeth located farther towards the posterior of the arch. In an ideal scenario, the proximal contact would be long enough so that there is an appropriately sized incisal embrasure and the gum tissue fills in the area below or gingival to the contact.

[0142] 6) Embrasure: In some implementations, techniques of this disclosure may measure the size (area, volume, circumference, etc.) of an embrasure, the gap between teeth at either of the gingival or incisal edge. In some implementations, techniques of this disclosure may measure the symmetry between embrasures on opposite sides of the arch. An embrasure is based at least in part on the length of the length of the contact between teeth, and / or at least in part on the shape of the tooth. In some instances, the size of the embrasure may get progressively longer for teeth located farther towards the posterior of the arch.

[0143] Non-limiting examples of Intra-tooth RDM are enumerated below, continuing with the numbering of other RDM listed above.

[0144] 7) Length and / or Width: A measure of the length of a tooth relative to the width of that tooth. This metric may reveal, for example, that a patient has long central incisors. Width and length are defined as: a) width - mesial to distal distance; b) length - gingival to incisal distance; c) other dimensions of tooth body - the portions of tooth between the gingival region and the incisal edge. In some implementations, either or both of a length and a width may be measured for a tooth and compared to the length and / or width of one or more teeth.

[0145] 8) Tooth Morphology: A measure of the primary anatomy of the tooth shape, such as line angles, buccal contours, and / or incisal angles and / or embrasures. The frequency and / or dimensions may be measured. In some implementations, the observed primary tooth shape aspects may be matched to one or more known styles. Techniques of this disclosure may measure secondary anatomy of the tooth shape, such as mamelon grooves. For instance, the frequency and / or dimensions may be measured. In some implementations, the observed secondary tooth shape aspects may be matched to one or more known styles. In some examples, techniques of this disclosure may measure tertiary anatomy of the tooth shape, such as perikymata or striations. For instance, the frequency and / or dimensions may be measured. In some implementations, the observed tertiary tooth shape aspects may be matched to one or more known styles.

[0146] 9) Shade and / or Translucency: A measure of tooth shade and / or translucency. Tooth shade is often described by the Vita Classical or 3D Master shade guide. Tooth translucency is described by transmittance or a contrast ratio. Tooth shade and translucency may be evaluated (or measured) based on one or more of the following kinds of data pertaining to teeth: the incisal edge, incisal third, body and gingival third. The enamel layer translucency is general higher than the dentin or cementum layer. Shade and translucency may, in some implementations, be measured on a per-voxel (local) basis. Shade and translucency may, in someimplementations, be measured on a per-area basis, such as an incisal area, tooth body area, etc. Tooth body may pertain to the portions of the tooth between the gingival region and the incisal edge.

[0147] 10) Height of Contour: A measure of the contour of a tooth. When viewed from the proximal view, all teeth have a specific contour or shape, moving from the gingival aspect to the incisal. This is referred to as the facial contour of the tooth. In each tooth, there is a height of contour, where that shape is the most pronounced. This height of contour changes from the teeth in the anterior of the arch to the teeth in the posterior of the arch. In some implementations, this measurement may take the form of fitting against a template of known dimensions and / or known proportions. In some implementations, this measurement may quantify a degree of curvature along the facial tooth surface. In some implementations, measure the location along the contour of the tooth where the height of the curvature is most pronounced. This location may be measured as a distance away from the gingival margin or a distance away from the incisal edge, or a percentage along the length of the tooth.

[0148] ROMs may be converted to restoration design scores (RDS) that represent the RDMs’ agreement with or deviation from ideal values in a patient case dataset. These RDS can then be used to rate restoration designs and inform oral care providers and / or automated systems about which RDMs are currently in agreement with good restoration designs, suggesting that the restoration designs do not need to be improved further, and which RDMs are not in agreement with good restoration designs, suggesting that the restoration designs need to be further refined.

[0149] Because neural networks of this disclosure can be implemented as one or more neural networks, the networks may contain an activation function. When executed, an activation function outputs a determination of whether or not a neuron in a neural network will fire (e.g., send output to the next layer). Some activation functions may include binary step functions, or linear activation functions. Other activation functions impart non-linear behavior to the network, including sigmoid / logistic activation functions, Tanh (hyperbolic tangent) functions, rectified linear units (ReLU), leaky ReLU functions, parametric ReLU functions, exponential linear units (ELU), softmax function, swish function, Gaussian error linear unit (GELU), or scaled exponential linear unit (SELU). A linear activation function may be well suited to some regression applications (among other applications), in an output layer. A sigmoid / logistic activation function may be well suited to some binary classification applications (among other applications), in an output layer. A softmax activation function may be well suited to some multiclass classification applications (among other applications), in an output layer. A sigmoid activation function may be well suited to some multilabel classification applications (among other applications), in an output layer. A ReLU activation function may be well suited in some convolutional neural network (CNN) applications (among other applications), in a hidden layer. A Tanh and / or sigmoid activation function may be well suited in some recurrent neural network (RNN) applications (among other applications), for example, in a hidden layer. There are multiple optimization algorithms which can be used in the training of the neural networks of this disclosure (such as in updating theneural network weights), including gradient descent (which determines a training gradient using first-order derivatives and is commonly used in the training of neural networks), Newton's method (which may make use of second derivatives in loss calculation to find better training directions than gradient descent, but may require calculations involving Hessian matrices), and conjugate gradient methods (which may yield faster convergence than gradient descent, but do not require the Hessian matrix calculations which may be required by Newton's method). In some implementations, additional methods may be employed to update weights, in addition to or in place of the techniques described above. These additional methods include the Levenberg-Marquardt method and / or simulated annealing. The backpropagation algorithm is used to transfer the results of loss calculation back into the network so that network weights can be adjusted, and learning can progress.

[0150] The neural networks of the present disclosure may embody part or all of a variety of different neural network models. Examples include the U-Net architecture, multi-later perceptron (MLP), transformer, pyramid architecture, recurrent neural network (RNN), autoencoder, variational autoencoder, regularized autoencoder, conditional autoencoder, capsule network, capsule autoencoder, stacked capsule autoencoder, denoising autoencoder, sparse autoencoder, conditional autoencoder, long / short term memory (LSTM), gated recurrent unit (GRU), deep belief network (DBN), deep convolutional network (DCN), deep convolutional inverse graphics network (DCIGN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), neural Turing machine (NTM), or generative adversarial network (GAN). In some implementations, an encoder structure or a decoder structure may be used. Each of these models provides one or more of its own particular advantages. For example, a particular neural networks architecture may be especially well suited to a particular ML technique. For example, autoencoders are particularly suited to the classification of 3D oral care representations, due to the ability to encode the 3D oral care representation into a form which is more easily classifiable.

[0151] In some implementations, the neural networks of this disclosure can be adapted to operate on 3D point cloud data (alternatively on 3D meshes or 3D voxelized representation). Numerous neural network implementations may be applied to the processing of 3D representations and may be applied to training predictive and / or generative models for oral care applications, including: PointNet, PointNet++, SO-Net, spherical convolutions, Monte Carlo convolutions and dynamic graph networks, PointCNN, ResNet, MeshNet, DGCNN, VoxNet, 3D-ShapeNets, Kd-Net, Point GCN, Grid-GCN, KCNet, PD-Flow, PU-Flow, MeshCNN and DSG-Net.

[0152] Some implementations of the techniques of this disclosure incorporate the use of an autoencoder. Autoencoders that can be used in accordance with aspects of this disclosure include but are not limited to: AtlasNet, FoldingNet and 3D-PointCapsNet. Some autoencoders may be implemented based on PointNet.

[0153] Representation learning may be applied to setups prediction techniques of this disclosure by training a neural network to learn a representation of the teeth, and then using another neural network to generate transforms for the teeth. Some implementations may use a VAE or a Capsule Autoencoder togenerate a representation of the reconstruction characteristics of the one or more meshes related to the oral care domain (including, in some instances, information about the structures of the tooth meshes). Then that representation (either a latent vector or a latent capsule) may be used as input to a module which generates the one or more transforms for the one or more teeth. These transforms may in some implementations place the teeth into final setups poses. These transforms may in some implementations place the teeth into intermediate staging poses. In some implementations, a transform may be described by a 9x1 transformation vector (e.g., that specifies a translation vector and a quaternion). In other implementations, a transform may be described by a transformation matrix (e.g., a 4x4 affine transformation matrix).

[0154] In some implementations, systems of this disclosure may implement a principal components analysis (PCA) on an oral care mesh and use the resulting principal components as at least a portion of the representation of the oral care mesh in subsequent machine learning and / or other predictive or generative processing.

[0155] Systems of this disclosure may implement end-to-end training. Some of the end-to-end trainingbased techniques of this disclosure may involve two or more neural networks, where the two or more neural networks are trained together (i.e., the weights are updated concurrently during the processing of each batch of input oral care data). End-to-end training may, in some implementations, be applied to setups prediction by concurrently training a neural network which learns a representation of the teeth, along with a neural network which generates the tooth transforms.

[0156] According to some of the transfer learning-based implementations of this disclosure, a neural network (e.g., a U-Net) may be trained on a first task (e.g., such as coordinate system prediction). The neural network trained on the first task may be executed to provide one or more of the starting neural network weights for the training of another neural network that is trained to perform a second task (e.g., setups prediction). The first network may learn the low -level neural network features of oral care meshes and be shown to work well at the first task. The second network may exhibit faster training and / or improved performance by using the first network as a starting point in training. Certain layers may be trained to encode neural network features for the oral care meshes that were in the training dataset. These layers may thereafter be fixed (or be subjected to minor changes over the course of training) and be combined with other neural network components, such as additional layers, which are trained for one or more oral care tasks (such as setups prediction). In this manner, a portion of a neural network for one or more of the techniques of the present disclosure (e.g., setups prediction) may receive initial training on another task, which may yield important learning in the trained network layers. This encoded learning may then be built upon with further task-specific training of another network.

[0157] In accordance with this disclosure, transfer learning may be used for setups prediction, as well as for other oral care applications, such as mesh classification (e.g., tooth or setups classification), mesh element labeling, mesh element in-filling, procedure parameter imputation, mesh segmentation, coordinate systemprediction, restoration design generation, mesh validation (for any of the applications disclosed herein). In some implementations, a neural network trained to output predictions based on oral care meshes may first be partially trained on one of the following publicly available datasets, before being further trained on oral care data: Google PartNet dataset, ShapeNet dataset, ShapeNetCore dataset, Princeton Shape Benchmark dataset, ModelNet dataset, ObjectNet3D dataset, ThingilOK dataset (which is especially relevant to 3D printed parts validation), ABC: A Big CAD Model Dataset For Geometric Deep Learning, ScanObjectNN, VOCASET, 3D- FUTURE, MCB: Mechanical Components Benchmark, PoseNet dataset, PointCNN dataset, MeshNet dataset, MeshCNN dataset, PointNet++ dataset, PointNet dataset, or PointCNN dataset.

[0158] In some implementations, a neural network which was previously trained on a first dataset (either oral care data or other data) may subsequently receive further training on oral care data and be applied to oral care applications (such as setups prediction). Transfer learning may be employed to further train any of the following networks: GCN (Graph Convolutional Networks), PointNet, ResNet or any of the other neural networks from the published literature which are listed above.

[0159] Reconstruction characteristics may comprise values in of a latent representation (e.g., a latent vector) that describe aspects of the shape and / or structure of the 3D representation that was provided to the representation generation module that generated the latent representation. The weights of the encoder module of a reconstruction autoencoder, for example, may be trained to encode a 3D representation (e.g., a 3D mesh, or others described herein) into a latent vector representation (e.g., a latent vector). Stated another way, the capability to encode a large set (e.g., hundreds, thousands or millions) of mesh elements into a latent vector (e.g., of hundreds or a thousand real values - e.g., 512, 1024, etc.) may be learned by the weights of the encoder. Each dimension of that latent vector may contain a real number which describes some aspect of the shape and / or structure of the original 3D representation. The weights of the decoder module of the reconstruction autoencoder may be trained to reconstruct the latent vector into a close facsimile of the original 3D representation. Stated another way, the capability to interpret the dimensions of the latent vector, and to decode the values within those dimensions, may be learned by the decoder. In summary, the encoder and decoder neural network modules are trained to perform the mapping of a 3D representation into a latent vector, which may then be mapped back (or otherwise reconstructed) into a 3D representation that is substantially similar to an original 3D representation for which the latent vector was generated.

[0160] One or more of the neural networks models of this disclosure may have attention gates integrated within. Attention gate integration provides the enhancement of enabling the associated neural network architecture to focus resources on one or more input values. In some implementations, an attention gate may be integrated with a U-Net architecture, with the advantage of enabling the U-Net to focus on certain inputs, such as input flags which correspond to teeth which are meant to be fixed (e.g,. prevented from moving) during orthodontic treatment (or which require other special handling). An attention gate may also be integrated with an encoder or with an autoencoder (such as VAE or capsule autoencoder) to improve predictive accuracy, inaccordance with aspects of this disclosure. For example, attention gates can be used to configure a machine learning model to give higher weight to aspects of the data which are more likely to be relevant to correctly generated outputs. As such, and because a machine learning model configured with these attention gates (or mechanisms) utilizes aspects of the data that are more likely to be relevant to correctly generated outputs, the ultimate predictive accuracy of those machine learning models is improved.

[0161] The latent space vector A for a tooth mesh (which may comprise thousands of interconnected mesh elements) describes the reconstruction characteristics of that tooth mesh in a compact form, for example a vector of length N (e.g., where in one example N = 128). This latent vector may be reconstructed into a close facsimile of the input tooth mesh through the operation of a decoder that has been trained for that task. The advantage of training a setups prediction neural network to take a latent vector as an input is to provide information about the reconstruction characteristics of the tooth mesh to the network. Reconstruction characteristics may contain information about local and / or global attributes of the mesh. Reconstruction characteristics may include information about mesh structure. Information about shape may, in some instances, be included. An awareness of these reconstruction characteristics may better enable the trained setups prediction model to predict a final setup or intermediate staging, thereby providing the technical improvement of improved data precision. A further advantage of using the latent space vector is the vector’s size. A neural network may encode an understanding of the input mesh and pose data more resource-efficiently if those data are presented in a compact form (such as a vector of 128 real values), as opposed to inputting the full mesh (which may contain thousands of mesh elements). The latent representation of a mesh may provide a more favorable signal-to-noise ratio than the original form of that mesh or those meshes, thereby improving the capability of a subsequent ML model (such as a neural network or SVM) to form predictions, draw inferences, and / or otherwise generate outputs (such as transforms or meshes) based on the input mesh(es).

[0162] The first ML module may, in some implementations, include one or more hierarchical feature extraction modules (e.g., modules which extract global, intermediate or local neural network features from a 3D representation - such as a point cloud). Examples of hierarchical neural network feature extraction modules (HNNFEM) include 3D SWIN Transformer architectures, U-Nets or pyramid encoder-decoders, among others. A 3D SWIN Transformer may extract hierarchical neural network features from a 3D representation through a series of consecutive stages of decreasing resolution. The input 3D representation may first undergo (optional) voxelization, and then be encoded in a latent representation. The latent representation may be provided to one or more Swin3D blocks, which may extract hierarchical features from the latent representation. At the top-level (stage 1), the hierarchical features are local features. The latent representation may be provided to stage 2, which may downsample the latent representation, and provide the downsampled latent representation to one or more Swin3D blocks. The resulting hierarchical features are now slightly more global than the features that were extracted in stage 1. Execution flows through stages 3, 4, 5, etc., until theglobal-most hierarchical neural network features are extracted. The 3D SWIN transformer structure then outputs the accumulated hierarchical neural network features from the several stages.

[0163] A HNNFEM may be trained to generate multi-scale voxel (or point) embeddings of a 3D representation. For example, a HNNFEM of one or more layers (or levels) may be trained on 3D representations of patient dentitions to generate neural network feature embeddings which encompass global, intermediate or local aspects of the 3D representation of the patient’s dentition. In some implementations, such embeddings may then be provided to a second ML module (e.g., which may contain one or more transformer decoder blocks, or one or more transformer encoder blocks), which may be trained to generate transforms for 3D representations of teeth or 3D representations of appliance components (e.g., transforms to place teeth into setups poses, or to place appliances, appliance components, fixture model components or other geometries relative to aspects of the patient’s dentition). Stated another way, a HNNFEM may be trained (on 3D representations of patient dentitions or 3D representations of appliances, appliance components or fixture model components) to operate as a multiscale feature embedding network.

[0164] The second ML module may, in some implementations, unite (e.g., by concatenation) the multiscale features before the transforms are predicted. This consideration of multi-scale neural network features may enable small interactions between aspects of the patient’s dentition (e.g., local features) to be considered during the setups prediction, during 3D representation generation or during 3D representation modification. For example, during setups prediction, collisions between teeth may be considered by the setups prediction model, and the model may be trained to minimize such collisions (e.g., by learning the distribution of a training dataset of orthodontic setups with ground truth that contains few or no collisions). This consideration of multiscale neural network features may further enable the whole tooth shape (e.g., global features) to be considered during final setups transform prediction. A HNNFEM may, in some implementations, contain ‘skip connections’, as are found in some U-NETS. In some implementations, neural network weights for the techniques of this disclosure may be pre-trained on other datasets, such as 3D indoor room segmentation datasets. Such pre-trained weights may be used via transfer learning, to fine-tune a HNNFEM which has been trained to extract local / intermediate / global neural network features from 3D representations of patient dentitions. A HNNFEM (e.g., which has been trained on 3D representations of patient dentitions, appliance components, or fixture model components) may entail an important technical improvement over other techniques, in that the HNNFEM may enable memory-efficient self-attention operations to be computed on sparse voxels. Such an operation is very important when the 3D representations which are provided at the input contain large quantities of mesh elements (e.g., large quantities of points, voxels, vertices / face / edges).

[0165] In some implementations, a HNNFEM may be trained to generate representations of teeth for use in setups prediction. The HNNFEM (e.g., which may, in some implementations, function as a type of encoder) may be trained to generate a latent representation (or latent vector or latent embedding) of a 3D representation of the patient’s dentition. The HNNFEM may be trained to generate hierarchical neural network features (e.g.,local, intermediate or global neural network features) of the 3D representation of the patient’s dentition. In other implementations, either a U-Net or a pyramid encoder-decoder structure may be trained to extract hierarchical neural network features. In some implementations, the latent representation may contain one or more of such local, intermediate, or global neural network features. Such a point cloud generation model may, in some implementations, contain a decoder (or ‘upscaling’ block) which may reconstruct the input 3D representation from that latent representation. A HNNFEM may have a symmetrical / mirrored arrangement, as may also appear in a UNET. The transformer decoder (or transformer encoder) may be trained to encode sequential or mutually dependent aspects of the patient's dentition (e.g., set of teeth and gums). Stated another way, the pose of one tooth may be dependent on the pose of surrounding teeth. For example, when the second ML module 414 or 708 contains at least one transformer (e.g., transformer encoder or transformer decoder) the second ML module 414 or 708 may learn dependencies between teeth or may be trained to minimize collisions (e.g., through the use of training by backpropagation as guided by loss calculation, such as LI, L2, mean squared error (MSE), or cross entropy loss, among others). It may be beneficial for an ML model to account for the sequential or mutually dependent aspects of the patient's dentition during setups prediction, tooth restoration design generation, fixture model generation, appliance component generation (or placement), to name a few examples. In some implementations, the output of the transformer decoder (or transformer encoder) may be reconstructed into a 3D representation (e.g., a 3D point cloud or 3D voxelized geometry). In some implementations, the latent space output of the transformer decoder (or transformer encoder) may be sampled, to generate points (or voxels). The latent representation which is generated by the transformer decoder (or transformer encoder) may be provided to a decoder. This latter decoder may perform one or more of a deconvolution operation, an upscaling operation, a decompression operation, or a reconstruction operation, among others.

[0166] Techniques described herein may be trained to generate transforms which may place the patient’s teeth into poses suitable for use in orthodontic setups (e.g., intermediate stages or final setups), according to the specification of the oral care arguments which may, in some implementations, be provided to the second ML module 414 or 708. Oral care arguments may include oral care parameters as disclosed herein, or other real- valued, text-based or categorical inputs which specify intended aspects of the one or more 3D oral care representations which are to be generated. In some instances, oral care arguments may include oral care metrics, which may describe intended aspects of the one or more 3D oral care representations which are to be generated. Oral care arguments are specifically adapted to the implementations described herein. For example, the oral care arguments may specify the intended designs (e.g., including shape and / or structure) of 3D oral care representations which may be generated (or modified) according to techniques described herein. In short, implementations using the specific oral care arguments disclosed herein generate more accurate 3D oral care representations than implementations that do not use the specific oral care arguments. Oral care arguments can provide the second ML module 414 or 708 with information about the patient’s arches whichmay improve the accuracy of IPR cut surface prediction (or the prediction of other information pertaining to IPR). For example, the “Canine Class Relationship” metric can provide the model with information about the relative fit of the lower arch with the upper arch. When the arches are not fit together properly, then the model can predict one or more IPR cut surfaces to modify tooth shapes and improve the fit of the arches. Oral care arguments may, in some instances, include information from a Bolton analysis of the patient’s dentition (e.g., ratio of the mesiodistal widths of the upper arch teeth to the lower arch teeth).

[0167] In some instances, a text encoder may encode a set of natural language instructions from the clinician (e.g., generate a text embedding). A text string may comprise tokens. An encoder for generating text embeddings may, in some implementations, apply either mean-pooling or max-pooling between the token vectors. In some instances, a transformer (e.g., BERT or Siamese BERT) may be trained to extract embeddings of text for use in digital oral care (e.g., by training the transformer on examples of clinical text, such as those given below). In some instances, such a model for generating text embeddings may be trained using transfer learning (e.g., initially trained on another corpus of text, and then receive further training on text related to digital oral care). Some text embeddings may encode text at the word level. Some text embeddings may encode text at the token level. A transformer for generating a text embedding may, in some implementations, be trained, at least in part, with a loss calculation which compares predicted outputs to ground truth outputs (e.g., softmax loss, multiple negatives ranking loss, MSE margin loss, cross-entropy loss or the like). In some instances, the non-text arguments, such as real values or categorical values, may be converted to text, and subsequently embedded using the techniques described herein. The following are examples of natural language instructions that may be issued by a clinician to the generative models described herein: “Generate a setup to set to Class I molar and canine, 2 mm overbite and add 2mm of expansion 5-575-5”, “Generate a setup to align with proclination and expansion and finish with .5 mm spaces U2-2 for future restorative”, or “Adjust the setup with no second molar movement, rotate upper first molars mesial out for Class I, level lower to a reverse curve of Spee 2 mm, advance the mandible with elastics to Class I canine.”

[0168] Aspects of the present disclosure can provide one or more technical solutions to various technical problems disclosed herein, including: predicting IPR cut surfaces, predicting orthodontic setups, determinations regarding whether teeth are accessible to IPR, determinations regarding whether to perform orthodontic treatment before IPR, and / or other data structures or determinations pertaining to IPR. In particular, by practicing techniques disclosed herein computing systems specifically adapted to perform IPR prediction for oral care appliance generation are improved. For example, aspects of the present disclosure improve the performance of a computing system having a 3D representation of the patient’s dentition by reducing the consumption of computing resources. In particular, aspects of the present invention reduce computing resource consumption by decimating 3D representations of the patient’s dentition (e.g., reducing the counts of mesh elements used to describe aspects of the patient’s dentition) so that computing resources are not unnecessarily wasted by processing excess quantities of mesh elements. Additionally, decimating the meshesdoes not reduce the overall predictive accuracy of the computing system (and indeed may actually improve predictions because the input provided to the ML model after decimation is a more accurate (or better) representation of the patient’s dentition). For example, noise or other artifacts which are unimportant (and which may reduce the accuracy of the predictive models) are removed. That is, aspects of the present invention provide for more efficient allocation of computing resources while simultaneously increasing predictive accuracy of the underlying ML model.

[0169] Furthermore, aspects of the present disclosure may need to be executed in a time -constrained manner, such as when an oral care appliance must be generated for a patient immediately after intraoral scanning (e.g., while the patient waits in the clinician’s office). As such, aspects of the present disclosure are necessarily rooted in the underlying computer technology of IPR prediction for oral care appliance generation and cannot be performed by a human, even with the aid of pen and paper. For instance, implementations of the present disclosure must be capable of: 1) storing thousands or millions of mesh elements of the patient’s dentition in a manner that can be processed by a computer processor; 2) performing calculations on thousands or millions of mesh elements, e.g., to quantify aspects of the shape and or / structure of an individual tooth in the 3D representation of the patient’s dentition; and 3) predicting IPR cut surfaces, orthodontic setups, determinations regarding whether teeth are accessible to IPR, determinations regarding whether to perform orthodontic treatment before IPR, and / or other data structures pertaining to IPR, and do so during the course of a short office visit.

[0170] Techniques of this disclosure may, in some implementations, use PointNet, PointNet++, or derivative neural networks (e.g., networks trained via transfer learning using either PointNet or PointNet++ as a basis for training) to extract local or global neural network features from a 3D point cloud or other 3D representation (e.g., a 3D point cloud describing aspects of the patient’s dentition - such as teeth or gums). Techniques of this disclosure may, in some implementations, use U-Nets to extract local or global neural network features from a 3D point cloud or other 3D representation.

[0171] 3D oral care representations are described herein as such because 3-dimensional representations are currently state of the art. Nevertheless, 3D oral care representations are intended to be used in a nonlimiting fashion to encompass any representations of 3-dimensions or higher orders of dimensionality (e.g., 4D, 5D, etc.), and it should be appreciated that machine learning models can be trained using the techniques disclosed herein to operate on representations of higher orders of dimensionality.

[0172] In some instances, input data may comprise 3D mesh data, 3D point cloud data, 3D surface data, 3D polyline data, 3D voxel data, or data pertaining to a spline (e.g., control points). An encoder-decoder structure may comprise one or more encoders, or one or more decoders. In some implementations, the encoder may take as input mesh element feature vectors for one or more of the inputted mesh elements. By processing mesh element feature vectors, the encoder is trained in a manner to generate more accurate representations of the input data. For example, the mesh element feature vectors may provide the encoder with more informationabout the shape and / or structure of the mesh, and therefore the additional information provided allows the encoder to make better-informed decisions and / or generate more-accurate latent representations of the mesh. Examples of encoder-decoder structures include U-Nets, autoencoders or transformers (among others). A representation generation module may comprise one or more encoder-decoder structures (or portions of encoders-decoder structures - such as individual encoders or individual decoders). A representation generation module may generate an information-rich and / or reduced-dimensionality representation of the input data, which may be more easily consumed by other generative or discriminative machine learning models.

[0173] A U-Net may comprise an encoder, followed by a decoder. The architecture of a U-Net may resemble a U shape. The encoder may extract one or more global neural network features from the input 3D representation, zero or more intermediate -level neural network features, or one or more local neural network features (at the most local level as contrasted with the most global level). The output from each level of the encoder may be passed along to the input of corresponding levels of a decoder (e.g., by way of skip connections). Like the encoder, the decoder may operate on multiple levels of global-to-local neural network features. For instance, the decoder may output a representation of the input data which may contain global, intermediate or local information about the input data. The U-Net may, in some implementations, generate an information-rich (optionally reduced-dimensionality) representation of the input data, which may be more easily consumed by other generative or discriminative machine learning models.

[0174] An autoencoder may be configured to encode the input data into a latent form. An autoencoder may train an encoder to reformat the input data into a reduced-dimensionality latent form in between the encoder and the decoder, and then train a decoder to reconstruct the input data from that latent form of the data. A reconstruction error may be computed to quantify the extent to which the reconstructed form of the data differs from the input data. The latent form may, in some implementations, be used as an information-rich reduced-dimensionality representation of the input data which may be more easily consumed by other generative or discriminative machine learning models. In most scenarios, an autoencoder may be trained to input a 3D representation, encode that 3D representation into a latent form (e.g., a latent embedding), and then reconstruct a close facsimile of that input 3D representation at the output.

[0175] A transformer may be trained to use self-attention to generate, at least in part, representations of its input. A transformer may encode long-range dependencies (e.g., encode relationships between a large number of inputs). A transformer may comprise an encoder or a decoder. Such an encoder may, in some implementations, operate in a bi-directional fashion or may operate a self-attention mechanism. Such a decoder may, in some implementations, may operate a masked self-attention mechanism, may operate a cross-attention mechanism, or may operate in an auto-regressive manner. The self-attention operations of the transformers described herein may, in some implementations, relate different positions or aspects of an individual 3D oral care representation in order to compute a reduced-dimensionality representation of that 3D oral care representation. The cross-attention operations of the transformers described herein may, in someimplementations, mix or combine aspects of two (or more) different 3D oral care representations. The autoregressive operations of the transformers described herein may, in some implementations, consume previously generated aspects of 3D oral care representations (e.g., previously generated points, point clouds, transforms, etc.) as additional input when generating a new or modified 3D oral care representation. The transformer may, in some implementations, generate a latent form of the input data, which may be used as an information-rich reduced-dimensionality representation of the input data, which may be more easily consumed by other generative or discriminative machine learning models.

[0176] In some implementations, an encoder-decoder structure may first be trained as an autoencoder. In deployment, one or more modifications may be made to the latent form of the input data. This modified latent form may then proceed to be reconstructed by the decoder, yielding a reconstructed form of the input data which differs from the input data in one or more intended aspects. Oral care arguments, such as oral care parameters or oral care metrics may be provided to the encoder, the decoder, or may be used in the modification of the latent form, to influence the encoder-decoder structure in generating a reconstructed form that has desired characteristics (e.g., characteristics which may differ from that of the input data).

[0177] Techniques of this disclosure may, in some instances, be trained using federated learning. Federated learning may enable multiple remote clinicians to iteratively improve a machine learning model (e.g., validation of 3D oral care representations, mesh segmentation, mesh cleanup, other techniques which involve labeling mesh elements, coordinate system prediction, non-organic object placement on teeth, appliance component generation, tooth restoration design generation, techniques for placing 3D oral care representations, setups prediction, generation or modification of 3D oral care representations using autoencoders, generation or modification of 3D oral care representations using transformers, generation or modification of 3D oral care representations using diffusion models, 3D oral care representation classification, imputation of missing values), while protecting data privacy (e.g., the clinical data may not need to be sent “over the wire” to a third party). Data privacy is particularly important to clinical data, which is protected by applicable laws. A clinician may receive a copy of a machine learning model, use a local machine learning program to further train that ML model using locally available data from the local clinic, and then send the updated ML model back to the central hub or third party. The central hub or third party may integrate the updated ML models from multiple clinicians into a single updated ML model which benefits from the learnings of recently collected patient data at the various clinical sites. In this way, a new ML model may be trained which benefits from additional and updated patient data (possibly from multiple clinical sites), while those patient data are never actually sent to the 3rd party. Training on a local in-clinic device may, in some instances, be performed when the device is idle or otherwise be performed during off-hours (e.g., when patients are not being treated in the clinic). Devices in the clinical environment for the collection of data and / or the training of ML models for techniques described herein may include intra-oral scanners, CT scanners, X-ray machines, laptop computers, servers, desktop computers or handheld devices (such as smart phones with imagecollection capability). In addition to federated learning techniques, in some implementations, contrastive learning may be used to train, at least in part, the ML models described herein. Contrastive learning may, in some instances, augment samples in a training dataset to accentuate the differences in samples from different classes and / or increase the similarity of samples of the same class.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A method of generating information pertaining to an interproximal reduction (IPR) in orthodontic treatment, the method comprising: receiving, by processing circuitry of a computing device, a digital representation of a patient’s dentition; providing, by the processing circuitry, the digital representation of the patient’s dentition to a partially trained machine learning (ML) model that comprises at least one of a first ML module and a second ML module, wherein the second ML module has been initially trained to generate one or more interproximal reduction (IPR) cut surfaces; generating, by the processing circuitry and using the ML model, a prediction for one or more IPR cut surfaces based on the digital representation of the patient’s dentition; and further training, by the processing circuitry, the ML model based on the generated prediction, to modify the ML model by performing operations comprising: predicting, using the ML model, the one or more IPR cut surfaces based on the digital representation of the patient’s dentition; quantifying, using the ML model, a difference between a representation of the one or more IPR cut surfaces predicted by the ML model and a representation of one or more reference IPR cut surface; generating a loss value based on the quantified difference; and modifying the ML model based at least in part on the loss value to form a modified ML model.

2. The method of claim 1, wherein the digital representation includes a plurality of mesh elements and a respective mesh element feature vector associated with at least one mesh element in the plurality of mesh elements.

3. The method of claim 2, further comprising computing, by the processing circuitry, at least one of a respective spatial feature or a respective structural feature corresponding to at least one of the respective mesh element feature vectors.

4. The method of claim 1, wherein the first ML module encodes the digital representation of the patient’s dentition into one or more latent representations which have a lower order of dimensionality than the original digital representation.

5. The method of claim 4, wherein the first ML module comprises at least one of an encoder, a U-Net , a pyramid encoder-decoder, a 3D SWIN transformer, a transformer encoder, or a transformer decoder.

6. The method of claim 1, wherein the second ML module comprises at least one of a transformer encoder, a transformer decoder, a decoder, a multi-layer perceptron (MLP), or an encoder.

7. The method of claim 1, further comprising providing, by the processing circuitry, at least one oral care argument as an input to the second ML module.

8. The method of claim 6, wherein the second ML module generates one or more predicted IPR cut surfaces for a first tooth of the digital representation of the patient’s dentition.

9. The method of claim 8, wherein each of the one or more predicted IPR cut surfaces is a mesial IPR cut surface or a distal IPR cut surface.

10. The method of claim 8, wherein the one or more predicted IPR cut surfaces further comprises one or more values pertaining to a magnitude of IPR to be applied to the tooth.

11. The method of claim 8, further comprising: receiving, by the processing circuitry, one or more stages pertaining to orthodontic treatment performed on the patient’s dentition; and generating, by the processing circuitry using the ML model, one or more determinations of which ones of the one or more stages of orthodontic treatment in which to perform IPR based on the one or more predicted IPR cut surfaces.

12. The method of claim 8, further comprising: receiving, by the processing circuitry, one or more stages pertaining to orthodontic treatment performed on the patient’s dentition; and generating, by the processing circuitry using the ML model, a determination of whether a tooth of the patient’s dentition is accessible to IPR.

13. The method of claim 12, further comprising specifying, based on the determination of whether a tooth of the patient’s dentition is accessible to IPR, whether to perform IPR on the tooth.

14. The method of claim 1, wherein the digital representation comprises at least one of a 3D mesh, a 3D point cloud, or a voxelized representation.

15. The method of claim 8, further comprising: generating, by the second ML module, one or more predicted tooth transforms, each of the one or more predicted transforms specifying the pose of a respective tooth in an orthodontic setup; and generating, using the one or more predicted tooth transforms, an orthodontic setup.

16. The method of claim 15, wherein the predicted one or more IPR cut surfaces are used to modify the shapes of respective one or more teeth of the patient’s dentition.

17. The method of claim 8, further comprising: accessing, by the processing circuitry, a fully trained ML model that has been trained to generate one or more predicted IPR cut surfaces; generating, by the processing circuitry and using the ML model, at least one of a prediction for one or more IPR cut surfaces, and a determination for one or more teeth to receive IPR, based on the digital representation of the patient’s dentition; outputting, by the one or more processors, the at least of a prediction and a determination to a clinician; performing IPR on the patient; scanning the patient’s teeth to generate post-IPR scans of the patient’s dentition; and performing, by the one or more processors, automated setups prediction using the generated post-IPR scans of the patient’s dentition.

18. The method of claim 12, wherein determining accessibility is based, at least in part, upon at least one measurement of the collisions between at least two teeth of the patient’s dentition.

19. The method of claim 12, further comprising: receiving, by the processing circuitry, one or more collision planes that specify, at least in part, one or more collisions between one or more aspects of the digital representation of the patient’s dentition; determining, by the processing circuitry using an ML model, accessibility of one or more teeth of the patient’s dentition based, at least in part, on an angle between the collision plane and the predicted IPR cut surface.

20. The method of claim 19, further comprising:receiving, by the processing circuitry, one or more archforms which describe one or more aspects of the patient’s dentition; determining, by the processing circuitry using an ML model, accessibility of one or more teeth of the patient’s dentition based, at least in part, on an angle between the collision plane and the one or more archforms.

Images