Deep Learning for Automatic Smile Design

Abstract

A method for displaying post-treatment teeth to show a person what their smile will look like after planned orthodontic treatment is disclosed. The method includes receiving a digital 3D model of the teeth or a rendered image of the teeth and an image of the person, such as a digital photograph. The method uses a generator network to generate a generated image of the person showing the person's teeth, i.e., the person's smile, after the planned orthodontic treatment. The method uses a discriminator network that processes the input image, the generated image, and the true image to train the generator network through a deep learning model to generate a photorealistic image of the person after the planned treatment.

Description

[Background technology]

[0001] Orthodontic clear tray aligners allow patients to receive high-quality, customizable treatment options. Potential patients may view past clinical cases when they are considering treatment. A high-level overview of one pipeline is as follows: A potential patient arrives at the doctor's office, the doctor takes a scan of the patient's teeth and extracts a three-dimensional (3D) mesh, which is processed by an algorithm to generate a mesh of the patient's teeth in their final alignment.

[0002] A common question from patients is, "What will my new smile look like?" Although patients have the ability to view previous clinical studies and even a 3D mesh of their newly aligned teeth, neither option provides patients with a true feel for how their teeth and smile will look after orthodontic treatment. This may dissuade potential patients from undergoing treatment altogether. Summary of the Invention

[0003] A method for displaying teeth after planned orthodontic treatment includes receiving a digital 3D model of the teeth or a rendered image of the teeth and an image of a person. The method uses a generator network to generate a generated image of the person showing the person's teeth after the planned orthodontic treatment. The method uses a discriminator network that processes the input image, the generated image, and the true image to train the generator network. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a diagram of a system for generating images of human smiles showing post-processing results. [Figure 2] FIG. 1 is a diagram of a generator network for the system. [Figure 3] FIG. 1 is a diagram of a discriminator network for the system. [Figure 4] We present some results after training the system. DETAILED DESCRIPTION OF THE INVENTION

[0005] An embodiment includes an automated system for generating images of a person's smile that show potential post-treatment aligner results before treatment begins. The system utilizes data including images of a person and corresponding 3D scans to learn how to generate photorealistic images. While the system is trained to generate a person's smile from pre-treatment scans, the scans can be swapped out for post-treatment scans to give the person the ability to see potential post-treatment results. Alternatively or additionally, the system can be used to show the person's appearance after each stage, treatment, or selected stage of treatment.

[0006] The ability for a person to view post-treatment photographs of their smile before any treatment begins can give the person confidence to proceed with the treatment process and help convince those who may be unsure. Additionally, if any aesthetic changes are needed, the person can provide feedback to the doctor or practitioner, who can modify the alignment of the mesh to meet the person's wishes.

[0007] FIG. 1 is a diagram of a system 10 for generating an image of a person's smile, showing post-processing results (21). The system 10 includes a processor 20 that receives a digital 3D model (mesh) or rendered image of teeth and a corresponding image (e.g., a digital photograph) of a person (12). The digital 3D model can be generated, for example, from an intraoral 3D scan or a dental impression scan. The system 10 can also include an electronic display device 16, such as a liquid crystal display (LCD) device, and an input device 18 for receiving user commands or other information. Systems for generating digital 3D images or models based on image sets from multiple views are disclosed in U.S. Patent Nos. 7,956,862 and 7,605,817, both of which are incorporated by reference herein as if fully set forth. These systems can use an intraoral scanner to obtain digital images from multiple views of teeth or other intraoral structures, and process these digital images to generate a digital 3D model representing the scanned teeth and gums. The system 10 can be implemented, for example, using a desktop, notebook, or tablet computer.

[0008] The system is built on a deep learning model known as a generative machine learning or generative adversarial network (GAN). This class of algorithms involves a pair of differentiable functions, often deep neural networks, whose goal is to learn an unknown data distribution. The first function, known as the generator, produces data samples given some input (e.g., random noise, conditional class labels, etc.). The generator and feature extraction network also receive the pixel-by-pixel difference between the generated image and the ground truth image in the form of a loss function. The second function, known as the discriminator, attempts to distinguish the "fake" data generated by the generator from the "true" data coming from the true data distribution. As the generator attempts to trick the discriminator into classifying the data as "true," the generated data becomes more realistic.

[0009] The system uses a Conditional GAN ​​(cGAN), where the generator is conditioned on either two-dimensional (2D) rendered images of a person's scanned teeth or a 3D mesh model of those teeth, along with an image of the person smiling with the teeth removed. The generator (see Figure 2), represented as a Convolutional Neural Network (CNN), aims to use the conditional information provided by those scans to inpaint missing data of the person's smile.

[0010] As shown in FIG. 2 , generator network 30 receives a 3D mesh or rendered image of teeth (22) via feature extraction network 26, which provides features 28. Feature extraction network 26 can be implemented, for example, using an inference engine to provide features in a low-noise output mesh or rendered image. Generator network 30 also receives an image (e.g., a digital photograph) of a person's face (24). Generator network 30 then generates an image (32) of the person smiling using each of the provided tooth models, allowing the person to view smiles with any number of different tooth styles. Specifically, the system can target a use case of viewing post-treatment smiles.

[0011] The discriminator, also referred to as a CNN, has two training steps. As shown in FIG. 3, the discriminator network 44 receives a 3D mesh or rendered image of teeth (34) via a feature extraction network 40, which provides features 42. The feature extraction network 40 can be implemented, for example, using an inference engine that provides features in a noisy output mesh or rendered image. In the first step, the discriminator network 44 is provided with an image (e.g., a digital photograph) (36) of a person with the mouth removed, a true image (46) of the person without the mouth removed, and features (42) of the person's scan extracted from another neural network. In this case, the discriminator network 44 should classify the data triplet as "true" (47) because the person's photograph comes from a real dataset. In a second step, the discriminator network 44 is again fed with an image of a person (e.g., a digital photograph) 36 with the mouth removed, and features of the person scan 42 extracted from another neural network, but this time with an image 38 generated by the generator. In this case, the discriminator network 44 should classify triplets of data as "false" 45. The generator and discriminator are trained simultaneously, improving each other.

[0012] An image of a smiling person with teeth excluded can be generated by finding the features of the smile (e.g., corners of the mouth) in the image, extracting the extent of those features, and painting those features white.

[0013] The generator network, discriminator network, and feature extraction network may be implemented in software or firmware modules for execution by a processor, such as processor 20. The generated images may be displayed, for example, on display device 16.

[0014] The dataset for the following experiments consists of approximately 5,000 patients. Each patient has a frontal photograph of their smile as well as a scan of their teeth. For this experiment, we used 2D renderings of the scans as context information.

[0015] Figure 4 shows the results after training the system. Each column contains a different patient, and each row contains scans of a matching patient (e.g., the patient in column A matches the scan of the patient in row A). In practice, the system works as follows: A potential patient arrives at the doctor's office and receives a scan of their teeth. This scan is processed by a staging algorithm to position the teeth in their final alignment or intermediate stages of treatment. Staging algorithms for generating intermediate and final stages or setups are described, for example, in U.S. Patent Application Publication No. 2020 / 0229900 and PCT Application Publication No. 2020 / 202009. Using a generator network, this scan is swapped with the patient's pre-treatment scan to generate a photograph of the patient with post-treatment results. Swapping the scans can include filling in the excluded mouth of the person in the input image.

[0016] To test the feasibility of this, we swapped scans from different patients and then generated their corresponding photographs. For example, column A of Figure 4 shows a photograph of patient A generated using the patient's own scan (row A) and scans from patients B and C (rows B and C).

[0017] 2-4, line drawings are used to represent people and their smiles. An embodiment typically uses real and generated photographs, as described above.

Claims

1. 1. A method for displaying teeth after planned orthodontic treatment, comprising the steps performed by a processor: receiving a digital 3D model of the teeth or a rendered image of the teeth and an image of a person; using a generator network to generate a generated image of the person showing the person's teeth after the planned orthodontic treatment of the teeth; and using a discriminator network that processes input images, generated images, and true images to train the generator network.

2. The method of claim 1 , further comprising receiving a final alignment or stage of the teeth after the planned orthodontic treatment.

3. The method of claim 2 , further comprising excluding the person's teeth in the received image.

4. The method of claim 3 , wherein using the generator network comprises filling in the excluded teeth in the received images with the final alignment or stage of the teeth.

5. 10. The method of claim 1, further comprising using a feature extraction network to extract features from the digital 3D model of teeth or rendered image of teeth for the generator network.

6. 10. The method of claim 1, further comprising using a feature extraction network to extract features from the digital 3D model of teeth or rendered image of teeth for the discriminator network.

7. The method of claim 1 , wherein the image comprises a digital photograph.

8. The method of claim 1 , wherein when the discriminator network is provided with a true image, the discriminator network classifies the input image as a true image for training the generator network.

9. The method of claim 1 , wherein the discriminator network classifies the input images as false when provided with generated images to train the generator network.

10. The method of claim 1 , wherein the planned orthodontic treatment includes a final stage or setup.

11. The method of claim 1 , wherein the planned orthodontic treatment includes an intermediate stage or setup.

12. A system for displaying teeth after planned orthodontic treatment, comprising a processor configured to perform any of the methods according to claims 1 to 11.

Images