Deep Learning for Generating Intermediate Orthodontic Aligner Stages

Abstract

A method for generating intermediate stages of an orthodontic aligner using machine learning or deep learning techniques is disclosed. The method receives a dental malocclusion and a planned setup position for the teeth. The malocclusion can be represented by translation and rotation or by a digital 3D model. The method uses one or more deep learning methods to generate intermediate stages of the aligner between the malocclusion and the planned setup position. The intermediate stages can be used to generate a setup that is output in a format, such as a digital 3D model, suitable for use in manufacturing the corresponding aligner.

Description

[Background technology]

[0001] Intermediate staging of teeth from the malocclusion stage to the final stage requires determining precise individual tooth movements so that the teeth move toward their final state without colliding with each other and follow an optimal, preferably short, trajectory. Because each tooth has six degrees of freedom and the average dental arch has about 14 teeth, finding the optimal trajectory of teeth from the initial stage to the final stage has a large and complex search space. There is a need to simplify this optimization problem. Summary of the Invention

[0002] A method for generating an intermediate stage of an orthodontic aligner includes receiving a dental malocclusion and a planned setup position for the teeth. The method uses one or more deep learning methods to generate an intermediate stage of the aligner between the malocclusion and the planned setup position. The intermediate stage can be used to generate a setup that is output in a format, such as a digital 3D model, suitable for use in manufacturing the corresponding aligner. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a diagram of a system for generating intermediate stages of orthodontic appliances. [Figure 2] 1 is a flowchart of a method for creating an intermediate stage of an orthodontic appliance. [Figure 3] FIG. 10 illustrates generating intermediate targets for orthodontic appliances. [Figure 4] FIG. 1 shows a malocclusion and corresponding intermediate stages. [Figure 5] 10A-10C are diagrams of user interfaces for side-by-side display of staging options generated by different staging techniques. DETAILED DESCRIPTION OF THE INVENTION

[0004] Embodiments include a possibly partially or fully automated system that uses deep learning to generate a set of orthodontic intermediate stages that allow a set of teeth to move from a malocclusion state to a final setup state or allow partial treatment from one state to another (e.g., from an initial state to a specific intermediate state). These stages include the arrangement of the teeth at a specific point in treatment. Each arrangement ("state" or "setup") of the teeth can be represented by a digital three-dimensional (3D) model. The digital setup can be used to fabricate orthodontic appliances, such as clear tray aligners, for moving the teeth along the treatment path. Clear tray aligners can be fabricated, for example, by converting the digital setup into a corresponding physical model and thermoforming a sheet of material over the physical model, or by 3D printing the aligners from the digital setup. Other orthodontic appliances, such as brackets and archwires, can also be configured based on the digital setup.

[0005] The system uses machine learning, particularly deep learning, techniques to train a model using historical data from intermediate stages. Using a portion of one known sequence or sequence of sequences, the system predicts the next sequence or sequence of sequences. For example, the system uses a neural network to take two different states, predict intermediate states between the different states, and recursively call the neural network for the desired resolution. In the example of a time series, a recurrent neural network predicts the next state or sequence of states instead of using interpolation to find the next state. As another example, a generative model takes a starting state, an ending state, and a fragment of the path between the starting and ending states as inputs to predict intermediate states.

[0006] Near real-time results, the ability to easily adapt to different treatment protocols, and the ability for the network to learn physician or practitioner preferences over time to efficiently generate treatment plans preferred by the physician or practitioner, also improving customer satisfaction, are advantages of deep learning or machine learning approaches for interim staging.

[0007] FIG. 1 is a diagram of a system 10 for generating intermediate stages of orthodontic appliances (21). The system 10 includes a processor 20 that receives a malocclusion and planned setup positions of teeth (12). The malocclusion can be represented using translation and rotation (along with a transformation). The transformation can be derived, for example, from a digital 3D model (mesh) of the malocclusion. Systems for generating digital 3D images or models based on image sets from multiple views are disclosed in U.S. Patent Nos. 7,956,862 and 7,605,817. These systems can use an intraoral scanner to obtain digital images from multiple views of teeth or other intraoral structures, and process these digital images to generate a digital 3D model representing the scanned teeth and gums. The system 10 can be implemented, for example, using a desktop, notebook, or tablet computer.

[0008] Deep learning for intermediate stage generation As the system acquires more data, machine learning methods, and especially deep learning methods, begin to rival or even surpass the performance of explicitly programmed methods. Deep learning methods have the advantage of being able to infer useful features directly from the data through a training process using nonlinear functional combinations of higher-dimensional latent or hidden features, eliminating the need to manually create features. While attempting to solve the staging problem, it may be desirable to operate directly on the 3D mesh of the malocclusion. Methods such as PointNet, PointCNN, and MeshCNN are well suited to this problem. Alternatively, deep learning can be applied to processed mesh data. For example, it can be applied after the whole-mouth mesh has been segmented into individual teeth and a standard tooth coordinate system has been defined. At this stage, useful information such as tooth position, tooth orientation, tooth dimensions, and interdental gaps is available. Tooth position is the Cartesian coordinate of the tooth's reference origin position, defined in a semantic context. Tooth orientation can be expressed as a rotation matrix, a unit quaternion, or another 3D rotation representation, such as Euler angles relative to a global reference system. The dimensions are real-valued 3D spatial extents, and the gaps can be binary presence indicators or real-valued gap sizes between teeth, especially if a particular tooth is missing. Deep learning methods can be implemented to use a variety of heterogeneous feature types.

[0009] As identified in the flowchart of FIG. 2, there are several candidate models that may be useful. The method of FIG. 2 may be implemented in a software or firmware module for execution by a processor, such as processor 20. The method receives inputs such as a tooth malocclusion and a planned setup position (step 22). The malocclusion can be represented by tooth position, translation, and orientation, or by a digital 3D model or mesh. The method uses deep learning algorithms or techniques to generate intermediate stages of orthodontic appliances based on the malocclusion to correct the malocclusion (step 24). The intermediate stages can be used to generate a setup output as a digital 3D model that can later be used to manufacture corresponding aligners. These deep learning methods may include multilayer perceptrons (26), time series prediction techniques (28), generative adversarial networks (30), video interpolation models (32), Seq2Seq models (34), and occlusion (36), as further described below. After generating the intermediate stages, the method may perform post-stage processing (step 38).

[0010] Multilayer Perceptron (26) The goal is to predict tooth positions and orientations at intermediate stages using the malocclusion and setup positions. A multilayer perceptron (MLP) architecture takes a set of features as input, then passes these features through a series of linear transformations followed by nonlinear functions to output a set of numerical values. The input features are the translation and rotation differences between the malocclusion and setup positions, and the output features are the translation and rotation differences between the malocclusion and intermediate positions. By recursively calling the trained MLP model, the system can create a set of target states representing tooth movement from malocclusion to Position 1, from Position 1 to Position 2, ..., from Position N to Setup. The system then performs linear interpolation between these target states to achieve tooth movement according to the tooth movement limits for each stage.

[0011] The model was trained on tooth movements from previous clear tray aligner cases. Some results on an independent test set not used during training are displayed in Figure 3, which shows intermediate targets generated by the MLP predicting tooth movements at intermediate positions. Target A was generated using malocclusion → setup movement as the input feature vector. Target B was generated using malocclusion → target A, and target C was generated using target A → setup.

[0012] Time Series Forecasting Methods (28) The staging problem can be posed as a prediction problem, which may be formulated in several different ways. 1. Given the current stage, predict the next stage. 2. Given stages up to n-1, predict the nth stage. 3. Given stages up to n-1, predict the next k stages (sequence generation).

[0013] All of these techniques can be implemented using recurrent neural network-based architectures such as RNNs, gated recurrent units, and long short-term memory neural networks. For sequence generation, encoder-decoder type architectures with any of the aforementioned algorithms can also be used.

[0014] Generative Adversarial Networks (GAN) (30) GANs can be used to create computer-generated examples that are essentially indistinguishable from human-generated examples. The model includes two parts: a generator that generates new examples, and a discriminator that attempts to distinguish between examples generated by the generator and human-generated examples. The performance of each part is optimized through model training on example data.

[0015] For this application, we trained a GAN to generate tooth movements. The generator takes as input 1) the tooth positions in the malocclusion and final position, and 2) a fragment of the path through the staging along which we wish to generate new tooth positions. Once the GAN is trained, the system can call the trained generator multiple times to generate tooth positions at multiple points throughout treatment.

[0016] Video Interbetweening Model (32) A video interpolation model is used to generate frames that occur between two frames of a video. This technique is used in techniques such as slow-motion video generation and frame recovery in video streaming. For the purposes of this embodiment, a video interpolation model was used to generate the two final stages, i.e., the intermediate stage that occurs between the malocclusion and the final setup. Specifically, a model was trained that is a modification of a bidirectional predictive network architecture. This network uses two encoder models to encode the tooth positions and orientations of the malocclusion and final stages into a latent feature space. These features are then passed to a decoder model that predicts the tooth positions and orientations that occur between the malocclusion and the final tooth positions. Figure 4 shows the malocclusion (left image) and intermediate stage (right image) generated using a bidirectional neural network.

[0017] Seq2Seq model (34) Seq2Seq models are used to generate sequences of data given an input sequence of data. They are often used in language processing applications for language translation, image captioning, and text summarization. In this embodiment, a seq2seq model was trained to generate sequences of intermediate stage tooth positions between the malocclusion and the final tooth position.

[0018] The constructed model is an encoder-decoder model. The encoder portion of the model encodes the input sequence of malocclusions and final tooth positions into a hidden vector of features using an MLP network. The decoder portion of the model then generates the tooth positions for the next stage from the encoded input sequence features and the sequences of all previous tooth position stages using a long-short term memory (LSTM) network. The complete output sequence of an intermediate stage is generated by recursively predicting the next stage positions using the decoder network until the model generates a flag that signals the network to stop.

[0019] Occlusion(36) To further improve staging results, both the upper and lower dental arches can be considered when searching for a collision-free path. By analyzing the occlusion map for the target stage, cross-arch interferences can be avoided, leading to better tracking, greater patient comfort, and ultimately treatment success. This occlusion method can use any of the deep learning methods described herein when generating intermediate stages for both the upper and lower dental arches.

[0020] Post-processing (38) The stages produced by the deep learning model can be displayed directly to the user or can undergo post-processing steps to make them easier to use. Examples of post-processing steps that may be desirable include:

[0021] 1. Fixed tooth reset - Teeth that the doctor or practitioner has designated should not move during treatment can be returned to their initial position.

[0022] 2. Collision Removal - As a post-processing step, collisions can be removed from stages generated by machine or deep learning algorithms if the algorithms introduce collisions. Below is an example method for post-processing collision removal:

[0023] 2a. Move teeth along the dental arch to remove collisions. First, calculate the total amount of space that exists in the dental arch and the total amount of collisions. If there is more collision than there is space, start with the most mesial tooth in each quadrant and move all of the teeth distally from their current position until they no longer collide with their mesial neighbors.

[0024] If there is more space than collisions, we attempt to preserve the space proportionally in the resulting packing. To do this, we first calculate the excess space that exists at the starting position (total space - total collisions = T). Starting with the most mesial tooth in each quadrant, we do one of the following:

[0025] If starting with an impact with a mesial neighbor, the tooth is moved distally away from the impact with its neighbor.

[0026] If we start with an initial space S with its mesial neighbor, we move the tooth to keep the space [S*(S / T)] with its mesial neighbor in its final position.

[0027] 2b. Iterative Collision Elimination. The general problem statement is that the teeth should be moved as little as possible from their initial positions to reduce or eliminate collisions between the teeth. An iterative search and optimization algorithm can be used to identify a set of tooth positions that minimizes collisions between the teeth while penalizing perturbations of the teeth from their starting positions. One implementation of this approach uses Levenberg-Marquardt optimization with the following cost function: Sum of collisions between all teeth + sum of squares of tooth movements from starting position

[0028] The search can also be biased to move teeth only in certain directions. For example, one implementation restricts tooth movement to the xy plane, preventing teeth from moving in a direction opposite to the direction in which the teeth move between the malocclusion and the setup position.

[0029] Customization Customizing these models to implement different types of treatment plans can be achieved by training the model with data belonging to that category, such as cases from a particular doctor or practitioner, cases where a certain treatment protocol has been applied, or cases with some refinements. This approach can eliminate the need to encode new protocols, since it only requires training the model on the correct subset of data. Alternatively, deep learning models have the potential to learn which protocols to apply to specific cases instead of needing to be instructed (i.e., the network automatically scales to identify congestion), making this a more adaptable approach that does not require explicit protocol development to learn the correct treatment strategy to apply.

[0030] comparison Deep learning techniques enable the rapid generation of multiple staging options, which can then be displayed to the physician (or practitioner) and patient, allowing them to compare treatments and select the option that best suits their preferences. FIG. 5 illustrates a user interface displaying different staging options for a particular stage side-by-side using the staging techniques described herein. The user interface of FIG. 5 can be displayed, for example, on display device 16. As shown in FIG. 5, the user interface includes a command function at the bottom for comparing staging options for a particular stage of a planned treatment, a zoom function in the center, command icons for rotating the image, and a command icon at the top right for selecting a view of the staging options.

Claims

1. 1. A method for generating an intermediate stage of an orthodontic aligner, the steps being performed by a processor: receiving dental malocclusions and planned setup positions of said teeth; generating intermediate stages of aligners between the malocclusion and the planned setup position using one or more deep learning methods; and outputting the intermediate stage.

2. The method of claim 1 , wherein the receiving step includes receiving tooth translations and rotations for the malocclusion.

3. The method of claim 1 , wherein the receiving step includes receiving a digital 3D model for the malocclusion.

4. The method of claim 1 , wherein the receiving step includes receiving a final stage for the planned setup location.

5. The method of claim 1 , wherein the outputting step includes outputting the intermediate stage as a digital 3D model.

6. The method of claim 1 , wherein the generating step includes using a multi-layer perceptron to generate the intermediate stage.

7. The method of claim 1 , wherein the generating step includes using a time series forecasting technique to generate the intermediate stage.

8. The method of claim 1 , wherein the generating step includes using a generative adversarial network to generate the intermediate stage.

9. The method of claim 1 , wherein the generating step includes using a video interpolation model to generate the intermediate stage.

10. The method of claim 1 , wherein the generating step includes using a seq2seq model to generate the intermediate stage.

11. The method of claim 1 , wherein the generating step includes using an articulation technique to generate the intermediate stage.

12. The method of claim 1 , further comprising performing post-processing on one or more of the intermediate stages.

13. The method of claim 12 , wherein the post-processing step includes resetting a fixed tooth for the intermediate stage.

14. The method of claim 12 , wherein the post-processing step includes removing inter-tooth collisions for the intermediate stage.

15. the generating step includes generating intermediate stages for a particular point in treatment by at least two different deep learning methods; the outputting step includes displaying the intermediate stage for the particular point in treatment. The method of claim 1.

16. The method of claim 15 , wherein the displaying step includes displaying the intermediate stages of the particular point in treatment side-by-side within a user interface.

17. A system for generating intermediate stages of orthodontic aligners, comprising a processor configured to perform any of the methods of claims 1 to 16.

Images