Visual representation of the gingival line generated based on a 3D tooth model

Abstract

Systems and methods for simulating dental treatment are disclosed. A method can include: capturing a first 2D image of a patient's face including the patient's teeth; constructing a parametric 3D model of the patient's teeth and gums based on the first 2D image; displaying a simulation result of dental treatment of the patient's teeth by rendering the parametric 3D model with the patient's teeth in one or more positions and / or orientations corresponding to treatment goals of a dental treatment plan; and rendering a second 2D image of the patient's face with gums and teeth according to the simulation result of the dental treatment plan. As described herein, the dental treatment plan can include orthodontic elements and / or prosthetic elements. The simulation result can correspond to an estimated result and / or an expected result of the dental treatment plan.

Description

[0001] Cross-reference

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 847,780, filed May 14, 2019, which is incorporated herein by reference.

[0003] The technical field relates to digital dental technology and, in particular, to providing simulation results of dental (e.g., orthodontic, restorative, etc.) treatment by evaluating two-dimensional (2D) depictions of a patient's untreated teeth with reference to parameters related to a model dental arch. Background Art

[0004] Orthodontic treatment typically involves addressing problems of tooth misalignment and / or jaw misalignment and can include the diagnosis, prevention, and / or correction of malocclusions. Persons seeking orthodontic treatment may seek a treatment plan from an orthodontist (e.g., a professional who has received special training after graduating from a dental school). Many orthodontic treatment plans involve treatment using braces, brackets, wires, and / or polymeric devices. Orthodontic professionals who design and / or implement orthodontic devices may adjust the orthodontic devices for persons seeking orthodontic treatment at different times.

[0005] Many people are referred for orthodontic treatment by a dentist, other treating professional, or other person. For example, many adolescent patients or persons with severe malocclusions may be referred for orthodontic treatment by their dentist or parents. However, many others may not know whether they should receive orthodontic treatment. For example, many people with mild malocclusions may not know whether orthodontic treatment is appropriate or desirable for them.

[0006] In addition, many people may imagine what they would look like smiling without tooth misalignment and / or jaw misalignment, e.g., after (one or more) estimated dental treatments and / or anticipated dental treatments, after implants or other devices having a configuration adapted to their face, age, traditions, and / or lifestyle, etc. are inserted. While it may be desirable to enable people to imagine what their smile and / or appearance would look like after completing viable treatment options, the computational burden and / or computational cost of existing tools makes this difficult to do. With existing tools, it is also difficult for people to imagine how dental treatment would meaningfully impact a patient's life. Summary of the Invention

[0007] The present disclosure generally relates to systems, methods, and / or computer-readable media related to dental treatment of simulating patient teeth, and in particular to providing a photo-realistic rendering of a two-dimensional (2D) image of a patient that represents one or more simulated (e.g., estimated and / or anticipated) results of a dental treatment plan. Embodiments herein produce near-exact and realistic renderings of simulated results of dental treatment and / or animations of three-dimensional (3D) models, which may not have been generated before or may have been generated only in a most basic manner using manual photo-editing tools. As described herein, the described embodiments use automated agents and / or rules to provide exact and realistic renderings of simulated results of dental (e.g., orthodontic, restorative, etc.) treatment and / or animations of 3D models, which were not possible before. Embodiments herein enable people considering orthodontic treatment and / or undergoing orthodontic treatment to visualize on a computer simulations of automatically generated simulated orthodontic treatment results and can inform a person's choice of whether to pursue orthodontic treatment during the general course and / or a particular course of orthodontic treatment. As described herein, the present disclosure also relates to systems and methods for accurately and realistically simulating a 3D model of teeth in a final orthodontic position in a 2D image of a person.

[0008] A computer-implemented method for simulating one or more simulated results of a dental treatment is disclosed. In some embodiments, a computer-implemented method for simulating orthodontic treatment may include capturing a first 2D image. In some embodiments, the first 2D image may include a representation of a patient's face and the patient's teeth. The method may include identifying one or more shapes associated with at least one tooth of the patient. The method may further include constructing a parametric 3D model of the patient's teeth based on the first 2D image using one or more case-specific parameters of the one or more shapes associated with at least one tooth of the patient. The method may further include: simulating the results of a dental treatment plan for the patient's teeth to produce a simulated result of the dental treatment plan; and modifying the parametric 3D model to provide a modified 3D model representing the simulated result of the dental treatment plan. The method may further include using the modified 3D model to render a second 2D image representing the patient's face, wherein the second 2D image represents the patient's teeth according to the simulated result of the dental treatment plan.

[0009] In some embodiments, constructing the parametric 3D model includes: finding the edges of the teeth and lips in the first 2D image; aligning the parametric tooth model with the edges of the teeth and lips in the first 2D image to determine case-specific parameters; and storing the case-specific parameters of the parametric tooth model that align the parametric tooth model with the edges of the teeth, gums, and lips in the first 2D image.

[0010] In some embodiments, rendering the second 2D image includes: accessing a parametric 3D model of a patient's teeth; projecting one or more tooth positions from the first 2D image onto the parametric 3D model; and mapping color data from the 2D image to corresponding positions on the parametric 3D model to generate a texture for the parametric 3D model; and using the texture as part of the second 2D image of the patient's face.

[0011] In some embodiments, the predetermined position is based on the average position of multiple teeth of a previous patient after dental treatment.

[0012] In some embodiments, the predetermined position is based on the average position of multiple teeth of a previous patient before dental treatment.

[0013] In some embodiments, the computer-implemented method may include finding the edges of teeth and lips in the first 2D image; and aligning the parametric 3D tooth model with the edges of the teeth, gums, and lips in the first 2D image.

[0014] In some embodiments, the first 2D image may include a side image representing the profile of the patient's face.

[0015] In some embodiments, the simulation result of the dental treatment plan may include an estimated result of the dental treatment plan.

[0016] In some embodiments, the simulation result of the dental treatment plan may include an expected result of the dental treatment plan.

[0017] In some embodiments, the dental treatment plan may include an orthodontic treatment plan, a restorative treatment plan, or some combination thereof.

[0018] In some embodiments, capturing the first 2D image may include: instructing a mobile phone or camera to image the patient's face, or collecting the first 2D image from a storage device or network system.

[0019] In some embodiments, constructing a parametric 3D model of a patient's teeth based on a 2D image using one or more case-specific parameters of one or more shapes associated with at least one tooth of the patient may include: roughly aligning the teeth represented in the 3D parametric model with the patient's teeth represented in the 2D image; and performing an expectation step to first determine the probability that the projection of the silhouette of the 3D parametric model matches one or more edges of the 2D image.

[0020] In some embodiments, constructing a parametric 3D model of a patient's teeth based on 2D images using one or more case-specific parameters of one or more shapes associated with at least one tooth of the patient may include: performing a maximization step using a small angle approximation to linearize a rigid transformation of the teeth in the 3D model; and performing an expectation step to secondarily determine a probability that a projection of a silhouette of the 3D parametric model matches an edge of the 2D image.

[0021] In some embodiments, a computer-implemented method may include iterating through a first plurality of cycles of the expectation step and the maximization step using a first subset of the parameters; and after iterating through a first plurality of cycles of the expectation step and the maximization step using the first subset of the parameters of the 3D parametric model, iterating through a second plurality of cycles of the expectation step and the maximization step using the first subset and a second subset of the parameters.

[0022] In some embodiments, a computer-implemented method may include capturing a first 2D image of a patient's face (including the patient's teeth). The method may include constructing a parametric 3D model of the patient's teeth based on the 2D image, the parametric 3D model including case-specific parameters of the shape of at least one tooth of the patient. The method may further include: simulating an estimated (e.g., estimated final) orthodontic position and / or an expected (e.g., expected final) orthodontic position of the patient's teeth by gathering information about one or more model dental arches (which represent a smile in the absence of tooth misalignment and / or jaw misalignment), and by rendering a 3D model of the patient's teeth in a predetermined position (e.g., a position corresponding to the position of the teeth in the model dental arch), and rendering a second 2D image of the patient's face with the teeth in an estimated orthodontic position.

[0023] In some embodiments, constructing the parametric 3D model includes: finding the edges of the teeth and lips in the first 2D image; aligning a parametric tooth model with the edges of the teeth and lips in the first 2D image to determine case-specific parameters; and storing the case-specific parameters of the parametric tooth model that align the parametric tooth model with the edges of the teeth, gums, and lips in the first 2D image.

[0024] In some embodiments, rendering the second 2D image includes: rendering the parametric model of the patient according to the position of the teeth in the first 2D image; projecting the 2D image onto the rendered parametric model of the patient according to the position of the teeth in the first 2D image; and mapping color data from the 2D image to corresponding positions on the 3D model to generate a texture of the 3D model; and using the generated texture to render a second 2D image of the patient's face with the teeth in an estimated orthodontic position.

[0025] In some embodiments, rendering the second 2D image further includes applying a simulated treatment to the second 2D image or viewing customization options for the second 2D image.

[0026] In some embodiments, the simulated treatment or viewing customization options can include changing one or more of the gum margins, replacing teeth, adjusting jaw position, or adjusting color data.

[0027] In some embodiments, the predetermined position is based on a combination of (e.g., average) positions of multiple teeth of previous patients after orthodontic treatment and / or when there is no tooth misalignment or jaw misalignment.

[0028] In some embodiments, the predetermined position is based on a combination of (e.g., average) positions of multiple teeth of previous patients before orthodontic treatment.

[0029] In some embodiments, the method can include: finding the edges of teeth and lips in the first 2D image; and aligning a parametric tooth model with the edges of the teeth, gums, and lips in the first 2D image.

[0030] In some embodiments, the first 2D image includes a profile image.

[0031] A computer-implemented method for constructing a 3D model of teeth from 2D images is disclosed. The method can include: capturing a 2D image of a patient's face (including the patient's teeth); determining the edges of teeth and gums within the first 2D image; fitting the teeth in a 3D parametric model of teeth to the edges of the teeth and gums within the first 2D image, the 3D parametric model including case-specific parameters of the shape of the patient's teeth; determining values of the case-specific parameters of the 3D parametric model based on this fitting.

[0032] In some embodiments, fitting the teeth in the 3D parametric model of teeth to the edges of the teeth and gums within the first 2D image includes: roughly aligning the teeth in the 3D parametric model with the teeth in the 2D image; and performing desired steps to determine the probability that the projection of the silhouette of the 3D parametric model matches the edges of the 2D image.

[0033] In some embodiments, fitting the teeth in the 3D parametric model of teeth to the edges of the teeth and gums within the first 2D image further includes: performing a maximization step using a small angle approximation to linearize the rigid transformation of the teeth in the model; and performing desired steps to again determine the probability that the projection of the silhouette of the 3D parametric model matches the edges of the 2D image.

[0034] In some embodiments, the computer-implemented method further includes: performing a first round of multiple loop traversals on the expectation step and the maximization step using a first subset of the parameters; and after performing the first round of multiple loop traversals on the expectation step and the maximization step using the first subset of the parameters of the 3D parameter model, performing a second round of multiple loop traversals on the expectation step and the maximization step using the first subset and a second subset of the parameters.

[0035] In some embodiments, the number of times in the first round is the same as the number of times in the second round.

[0036] In some embodiments, the first subset of the case-specific parameters of the 3D parameter model is one or more of a scale factor, tooth position, and tooth orientation.

[0037] In some embodiments, the second subset of the parameters of the 3D parameter model is tooth shape and one or more of tooth position and tooth orientation.

[0038] A computer-implemented method for providing simulation results of orthodontic treatment is disclosed. The method may include: constructing a 3D parameter model of the dental arch, the 3D parameter model including general parameters of tooth shape, tooth position, and tooth orientation; capturing a 2D image of a patient; constructing a case-specific 3D parameter model of the patient's teeth from the 2D image; determining the case-specific parameters of the constructed parameter model; rendering the 3D parameter model of the patient's teeth in an estimated and / or expected final position (e.g., no tooth misalignment and / or jaw misalignment); and inserting the rendered 3D model into the 2D image of the patient.

[0039] In some embodiments, constructing a 3D parameter model of the patient's teeth from the 2D image includes: finding the edges of the teeth, gums, and lips in the first 2D image; and aligning the 3D parameter model with the edges of the teeth, gums, and lips in the first 2D image.

[0040] In some embodiments, the method further includes applying a texture to the rendered 3D parameter model of the patient's teeth in an estimated and / or expected final position (e.g., no tooth misalignment and / or jaw misalignment), where the texture is derived from the 2D image of the patient.

[0041] In some embodiments, the texture is derived from the 2D image of the patient by: projecting the 2D image onto the rendered parameter model of the patient according to the position of the teeth in the first 2D image; and mapping the color data from the 2D image to the corresponding positions on the 3D model to obtain the texture of the 3D model.

[0042] In some embodiments, rendering a 3D parametric model of a patient's teeth in an estimated and / or expected final position includes: generating an average shape of the teeth based on the 3D parametric model; adjusting the shape of the teeth in the 3D parametric model based on case-specific tooth shape parameters; positioning the teeth with an average tooth position and orientation based on average position and orientation parameters such that the teeth have a case-specific shape and average position and orientation; and scaling the dental arch based on case-specific dental arch scaling parameters.

[0043] A non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform any of the methods described herein.

[0044] A system is disclosed. The system can include: a photo-parametrization engine configured to generate a 3D parametric dental arch model from 2D images of a patient's face and teeth, the parametric 3D model including case-specific parameters of the shape of at least one tooth of the patient; and a parametric treatment prediction engine configured to identify an estimated and / or expected outcome of orthodontic treatment for the patient based on the 3D parametric dental arch model and historical models of multiple patients and / or an ideal dental arch model.

[0045] In some embodiments, the system includes a treatment projection rendering engine configured to render the 3D parametric dental arch model.

[0046] In some embodiments, the photo-parametrization engine, the parametric treatment prediction engine, and the treatment projection rendering engine are together configured to perform the methods described herein.

[0047] Incorporation by reference

[0048] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The novel features of the invention are particularly set forth in the appended claims. The features and advantages of the invention will be better understood from the following detailed description of illustrative embodiments and the accompanying drawings, in which illustrative embodiments of the principles of the invention are utilized:

[0050] Figure 1 A method of providing an estimated outcome of orthodontic treatment in accordance with one or more embodiments herein is shown;

[0051] Figure 2 A parametric tooth model in accordance with one or more embodiments herein is shown;

[0052] Figure 3A An example showing the degree of matching between a parametric tooth model and an original 3D model according to one or more embodiments herein;

[0053] Figure 3B A method for determining general parameters from historical cases and / or ideal cases according to one or more embodiments herein;

[0054] Figure 4 An alignment of past cases used in determining the parameters of a parametric model according to one or more embodiments herein;

[0055] Figure 5 A method for generating a parametric model of a patient's teeth and converting the parametric model into a 3D model of an arch according to one or more embodiments herein;

[0056] Figure 6 A method for constructing a 3D model from 2D images according to one or more embodiments herein;

[0057] Figure 7A A method for constructing a patient-specific parametric model of a patient's teeth according to one or more embodiments herein;

[0058] Figure 7B A tooth model having a gingival margin and a lip margin according to one or more embodiments herein;

[0059] Figure 8 A method for rendering a patient's teeth in an initial position using a parametric model of a patient's arch according to one or more embodiments herein;

[0060] Figure 9 A method for constructing a 3D model and applying a texture to the 3D model according to one or more embodiments herein;

[0061] Figure 10A A method for simulating an estimated result of orthodontic treatment of a patient's teeth according to one or more embodiments herein;

[0062] Figure 10B A method for simulating a patient's orthodontic treatment based on matching tooth shape parameters according to one or more embodiments herein;

[0063] Figure 11 An example of a method for rendering teeth according to an estimated result of a dental treatment plan according to one or more embodiments herein;

[0064] Figure 12 A system for simulating an estimated result of orthodontic treatment according to one or more embodiments herein;

[0065] Figure 13 Illustrates an example of one or more elements of an estimated orthodontic treatment simulation system in accordance with one or more embodiments herein;

[0066] Figure 14 Shows a tooth repositioning instrument in accordance with one or more embodiments herein;

[0067] Figure 15 Shows a tooth repositioning system in accordance with one or more embodiments herein;

[0068] Figure 16 Shows a method of orthodontic treatment using multiple instruments in accordance with one or more embodiments herein;

[0069] Figure 17 Shows a method of designing an orthodontic instrument in accordance with one or more embodiments herein;

[0070] Figure 18 Shows a method of planning orthodontic treatment in accordance with one or more embodiments herein;

[0071] Figure 19 Is a simplified block diagram of a system for designing an orthodontic instrument and planning orthodontic treatment in accordance with one or more embodiments herein.

[0072] Figure 20 Shows a method of simulating a gingival line in accordance with one or more embodiments herein;

[0073] Figure 21A Shows a gingival line input determined from a 2D image of a patient's teeth in accordance with one or more embodiments herein;

[0074] Figure 21B Depicts a process of modeling a gingival line based on an input determined from a 2D image of a patient's teeth in accordance with one or more embodiments herein;

[0075] Figure 22 Shows gingival line parameters in accordance with one or more embodiments herein;

[0076] Figure 23 Shows a method of leveling a gingival line in accordance with one or more embodiments herein;

[0077] Figure 24 Depicts a process of rendering a realistic composite image of a patient's face and a patient's tooth model in accordance with one or more embodiments herein. Detailed Description

[0078] The embodiments discussed herein provide tools such as automated agents to visualize the orthodontic effects of malocclusions, crossbites, etc., without the computational burden and / or cost of scanning the patient's dentition or dental impressions, and also calculate the final positions of treatment plans for the patient's dentition. As discussed in detail herein, these techniques can involve obtaining a two-dimensional (2D) representation (e.g., an image) of the patient's dentition, obtaining one or more parameters representing the properties of the patient's dentition in the 2D representation, and using the one or more parameters to compare the properties of the patient's dentition with the properties of a model dental arch (e.g., the properties of historical cases and / or the properties representing the ideal dental arch form). The techniques herein can provide a basis for simulating the results of a dental treatment plan.

[0079] As used herein, the "simulation results of a dental treatment plan" can include, for example, the estimated and / or expected results of a dental treatment plan after implementing one or more dental procedures such as orthodontic procedures, restorative procedures, etc. The "estimated results of a dental treatment plan" as used herein can include an estimate of the state of the patient's dentition after a dental procedure. In some cases, the estimated results of a dental treatment plan as used herein may differ from the "actual results of a dental treatment plan", which can represent the state of the patient's dentition after implementing the dental treatment plan. In various cases, the estimated and actual results of a dental treatment plan as used herein may differ from the "expected results of a dental treatment plan", which can represent the expected state of the patient's dentition after implementing the dental treatment plan. It should also be noted that the "estimated results of an orthodontic treatment plan" can include an estimate of the state of the patient's dentition after correcting any malocclusions, crossbites, etc. suffered by the patient. In some embodiments, the estimated results of an orthodontic treatment plan include the estimated state of the patient's dentition in the case where the patient's dentition has been changed to have a model and / or ideal dental arch form reflected by one or more databases based on historical cases and / or ideal dental arch forms. The "actual results of an orthodontic treatment plan" can represent the state of the patient's dentition after implementing the orthodontic treatment plan; the "expected results of an orthodontic treatment plan" can represent the expected state of the patient's dentition after implementing the orthodontic treatment plan.

[0080] The features and advantages of the present disclosure can be better understood with reference to the following detailed description of illustrative embodiments and the accompanying drawings, in which the principles of the embodiments of the present disclosure are utilized.

[0081] Figure 1 An example of a method 100 for providing estimated and / or expected results of a dental treatment plan in accordance with one or more embodiments herein is shown. Method 100 can be performed by any system disclosed herein. It should be noted that various examples can include more than Figure 1more or fewer boxes than shown.

[0082] In block 110, one or more two-dimensional (2D or 2-D) images of a patient are captured. In some embodiments, the (one or more) 2D images depict the patient's mouth and include one or more images of the face, head, neck, shoulders, torso, or the patient as a whole. The (one or more) 2D images of the patient may include images of the patient with the patient's mouth in one or more positions; for example, the patient's mouth may be in a smiling position (e.g., a social smile position), a resting position with relaxed muscles and slightly open lips, or a retruded anterior open bite or anterior closed bite position.

[0083] In some embodiments, an image of the patient is obtained using an image capture device. As used herein, an "image capture device" (i.e., "image capture system") may include any system capable of capturing an image. Examples of image capture devices include cameras, smartphones, digital imaging devices, components of a computer system configured to capture images, etc. Images may be captured using a lens with a predetermined focal length at a distance from the patient. Images may also be captured remotely and then received for processing. In some embodiments, an image of the patient is obtained from a computer storage device, a network location, a social media account, etc. The images may be a series of images or videos captured from one or more perspectives. For example, the images may include one or more of a frontal face image and a side image, and the side image includes one or more three-quarter side images and full side images.

[0084] In block 120, a three-dimensional (3D or 3-D) model of the patient's teeth is generated based on the 2D images of the patient. As referenced Figure 6 and discussed in more detail elsewhere herein, generating the 3D model may include identifying the patient's teeth. Generating the 3D model may also include identifying the patient's gums, lips, and / or mouth opening, forming parametric models of each identified tooth, and combining the parametric tooth models into one or more (e.g., two, three, etc.) parametric dental arch models. As further discussed herein, the parametric models of the patient's teeth and dental arches may be based on or referenced to average parametric tooth and dental arch models.

[0085] The "parametric model of a patient's teeth" (e.g., "parametric model of a patient's dentition") used herein can include a model of a patient's dentition (e.g., a statistical model) characterized by a probability distribution with a finite number of parameters. The parametric model of a patient's dentition can include a parametric model of the patient's teeth and / or dental arches. The parametric model can include models of objects representing various dimensions and can include parametric 2D models, parametric 3D models, etc. Modeling a patient's teeth using a parametric model of the patient's dentition can reduce the memory and computational requirements when manipulating, comparing, or otherwise using digital models (as described herein) and simplify the comparison between different models. In some embodiments, the parametric model of a tooth can be expressed as:

[0086]

[0087] where is the average tooth shape and is a general parameter. As described herein, each tooth (e.g., the upper right canine tooth numbered 6 in the universal tooth numbering system) has its own average tooth shape, which is calculated from thousands of teeth with the same tooth number. The symbol τ can represent the index of each tooth, which can be based on the universal tooth numbering system or another numbering system. can be the principal component of the shape of each tooth, which is also a general parameter, and can be the coefficient of the principal component of the tooth shape, which is a case-specific parameter. Thus, equation (1) can be used to represent the parametric model of each tooth specific to a patient (e.g., the lower left incisor, the upper right canine, etc.) based on the specific shape of each tooth relative to the average tooth shape of that tooth.

[0088] The parametric model of a patient's dentition can be expressed as:

[0089]

[0090] where is as described above with reference to equation (1), is the average tooth position and is a general parameter; T τ is the deviation of the position of the patient's tooth from the corresponding average tooth position and is a case-specific parameter; and Φ is the dental arch scaling factor that scales the dimensionless parameter value to a real-world value and is also a case-specific parameter. T is the global perspective of the dental arch from a certain perspective and is a case-specific parameter, and in some embodiments, is only used when matching with a 2D image because dental arch scans usually do not have a perspective while 2D images (e.g., cameras) do.

[0091] To generate a parametric 3D model of a patient's teeth from a 3D dental model (which is obtained from an image of the patient's teeth or from another method known in the art), the teeth can be modeled based on the displacement of a fixed shape (e.g., a fixed sphere) from the scanned tooth surface. To illustrate this, reference is made to Figure 2 , which shows a parametric tooth model according to one or more embodiments herein. In the example of Figure 2 , a sphere 210 with a plurality of vertices in a fixed or known position and orientation is shown. A tooth 220 can be placed at the center of the sphere 210 or otherwise modeled. In some embodiments, the volume center of the tooth 220, the scanned portion of the tooth 220, or the crown of the tooth 220 can be aligned with the center of the sphere 210. Then, each vertex 230a, 230b of the sphere 210 can be mapped to a position on the surface of the tooth model. In some embodiments, the mapping can be represented by an n*3 matrix, where n represents the number of points on the sphere (e.g., 2500), and then for each point, the x, y, and z positions are recorded. In some embodiments, the matrix stores the difference between the positions on the average tooth and the corresponding positions on the actual model. In this way, each tooth is represented by the same 2500 points, and it is easy to compare the differences between teeth. This difference can be represented by PCA components as where each specific case ultimately has a unique set, because is common to all cases. To illustrate this, reference is made to Figure 3A , which shows an example of the degree of match between a parametric model 320 of a tooth and an original 3D model 310. The parameters of a specific tooth can be stored in a data warehouse (e.g., a database) and can be represented by .

[0092] A parametric model of a patient's dental arch can involve parameterizing the position and orientation of each tooth in the dental arch as well as the scale of the dental arch. The case-specific parameters of a specific tooth can be stored in a matrix or other data warehouse, as shown in the following equation (3):

[0093]

[0094] where ξ ij is the rotational component that defines the orientation of the tooth relative to the dental arch and can be a function of the three rotational angles α τ , β τ , γ τ of the tooth. Δ τ,x , Δ τ,y and Δ τ,zis the translation of the center point of the tooth relative to the origin of the dental arch. Rotation can be based on the orientation of the orthogonal axes (e.g., the long axis, the buccolingual axis, and the mesiodistal axis) of each tooth relative to a fixed reference or relative to an average rotation. In some embodiments, one or more of these components can be represented as a deviation or variation relative to the average dental arch discussed herein.

[0095] Similarly, a scaling factor Φ is applied to the tooth and the dental arch position of the tooth to scale the dental arch from a generic representation or unitless representation to a real-world scale that represents the actual size and position of the tooth in the dental arch. In some embodiments, the scaling can be between the projected 3D units (e.g., millimeters) and the image size (e.g., pixels).

[0096] As discussed above and elsewhere herein, a parametric model of a dental arch including teeth can be represented based on an average dental arch model including teeth. The average dental arch model can be determined based on the average of a large dataset of patient dental arch scans. In some embodiments, the average dental arch model can be based on a large dataset of parameterized dental arches.

[0097] For example, an average dental arch can be constructed from a set of previously scanned and / or segmented dental arches. Figure 3B An example of a method 350 for determining an average dental arch model is depicted and discussed in further detail herein.

[0098] In some embodiments, a parametric model of a dental arch can be converted to a 3D model of the dental arch. Figure 5 A method 500 for converting a parametric model of one or more teeth to a 3D model of a dental arch according to some embodiments is depicted and discussed in further detail herein.

[0099] Returning to Figure 1 , at block 130, identify the estimated and / or expected results of a dental treatment plan on the 3D model to obtain the estimated and / or expected results of an orthodontic treatment plan. As referenced Figures 9 - 11As discussed, the estimated and / or expected results of a dental treatment plan can be based on a parametric model of the patient's teeth at positions indicated by a predetermined dental arch model. In some embodiments, the predetermined dental arch model can be based on an average dental arch model, which can be based on a historical average of dental arch scans collected from the patient or other dental arch models from the patient. In some embodiments, for example, the predetermined dental arch model can be based on an idealized model, such as a clinically ideal dental arch or dental arch model, where the tooth positions and / or orientations are predetermined based on, for example, one or more of the aesthetic and clinical characteristics of the teeth (such as correct occlusion). In some embodiments, the estimated and / or expected results of the dental treatment plan can correspond to the expected final positions (or estimates thereof) of the patient's teeth after implementing the treatment plan. In some embodiments, the estimated and / or expected results of the dental treatment can correspond to estimates of the final positions and intermediate positions during the treatment planning process. As described herein, a dental treatment plan can include an orthodontic treatment plan, a restorative treatment plan, some combination thereof, and the like.

[0100] At block 140, generate one or more second 2D images showing the estimated and / or expected results of the orthodontic treatment. As discussed herein, the one or more second 2D images can be 2D facial images of the patient, where the teeth are aligned according to the estimated and / or expected results of the dental treatment plan. In some embodiments, for example, as discussed below with reference to Figure 9 and elsewhere herein, the image can include an estimated texture and / or projected texture of the patient's teeth. The "projected texture" or "estimated texture" of the patient's teeth as used herein can include a projection / estimate of the tooth texture and can include the tactile, appearance, consistency, and / or other properties of the tooth surface. At block 150, provide the one or more second 2D images generated at block 140 to the user. For example, the image can be rendered for viewing by the patient or a dental professional. In some embodiments, the second 2D image can be loaded into memory or retrieved from memory.

[0101] Turning to Figure 3B , Figure 3BA method for determining general parameters from historical cases and / or ideal cases according to one or more embodiments herein is shown. At block 360, historical cases and / or ideal cases are obtained. Historical cases and / or ideal cases can be retrieved from a data warehouse. Historical cases and / or ideal cases can include cases representing previously scanned and segmented dental arch models. In some embodiments, historical cases and / or ideal cases can represent dental arch models of treated patients (e.g., patients who have been treated in the past) and / or ideal dental arch models representing the expected outcomes of various forms of orthodontic treatment. In various embodiments, historical cases and / or ideal cases can represent dental arch models having an ideal dental arch form. In some embodiments, historical cases and / or ideal cases can include multiple regions where the teeth of the modeled dental arch correspond to the positions of implants to be implanted in the patient's dental arch.

[0102] At block 370, the historical cases and / or ideal cases are aligned. In some embodiments, each dental arch of the historical cases and / or ideal cases is aligned at multiple positions. As an example, each dental arch of the historical cases and / or ideal cases can be aligned at the following three positions: between the central incisors, and at each distal end on the left and right sides of each dental arch. For example, Figure 4 A set of dental arches 400 is shown, which is aligned at the position between the central incisors 410, at the left distal end 430 of the dental arch, and at the right distal end 420 of the dental arch. It should be noted that without departing from the scope and gist of the inventive concept described herein, the historical cases and / or ideal cases can be aligned at a variety of positions and different numbers of positions.

[0103] Return Figure 3B , determining the average dental arch model and determining the distribution of dental arch models can include performing sub-operations at block 370. For example, taking the average of each dental arch to determine Then determining the local deformations of each tooth, and comparing with to determine T τ , then after aligning each tooth, β can be determined τ .

[0104] At block 380, an estimation of the distribution of case-specific parameters is determined. For example, determining the relative shape, position, and rotation of each tooth to construct the distribution of each case-specific parameter. For example, determining the coefficients of the principal components of the surface model of each corresponding tooth in all retrieved models of the distribution.

[0105] The position and orientation of each tooth can be averaged to determine the average position of each tooth, and the orientation of each tooth can be averaged to determine the average orientation of each tooth. The average position and average orientation are used to determine

[0106] In block 390, the mean of the estimated distribution of case-specific parameters is used as a general parameter for the mean tooth position and the mean tooth shape.

[0107] Figure 5 Method 500 for generating a parametric model of one or more teeth of a patient and converting the parametric model of the one or more teeth into a 3D model of an arch is depicted. Method 500 can be used to generate a 3D model of an arch of a patient based on a parametric model of the patient's teeth.

[0108] In block 510, the mean shape of each tooth is determined. As an example, the mean shape of each tooth is determined In some embodiments, as discussed herein, the mean shape can be based on the mean shape of a set of dental arches from historical cases and / or ideal cases. In some embodiments, for example, the mean shape can be rendered on a screen for viewing by the patient or a dental professional. For example, the mean tooth shape 512 is rendered for viewing. In some embodiments, the mean shape can be loaded into memory or retrieved from memory. The mean shape can also be initialized as a set of matrices, each matrix corresponding to a tooth.

[0109] In block 520, a principal component analysis shape adjustment is performed on the mean shape of the teeth. As discussed herein, this adjustment shapes the teeth based on the patient's specific teeth, e.g., based on a scan of the patient's teeth, 2D images, or other imaging techniques. As an example, a principal component analysis shape adjustment is performed on the mean shape of the teeth. For each tooth in the model, the case-specific coefficients of the principal components are applied to the principal components After the shape adjustment is completed, in some embodiments, for example, the adjusted shape can be rendered on a screen for viewing by the patient or a dental professional. For example, the adjusted tooth shape 522 is rendered for viewing. In some embodiments, the adjusted shape can be stored in memory. The adjusted shape can also be stored as a set of matrices, each matrix corresponding to a tooth.

[0110] At block 530, an average tooth pose is determined. In some embodiments, as discussed herein, the average tooth pose may be based on the average tooth poses of a set of dental arches from historical cases and / or ideal cases. In some embodiments, at block 530, each adjusted tooth 522 is placed at its corresponding average position and orientation determined by the average dental arch. In some embodiments, for example, the average tooth pose may be rendered on a screen for viewing by the patient or a dental professional. For example, the average tooth pose 532 is rendered for viewing. In some embodiments, the average tooth pose may be loaded into memory or retrieved from memory. The average tooth pose may also be initialized as a set of matrices, each matrix corresponding to a tooth in the tooth pose. In some embodiments, prior to adjusting the shape of the teeth at block 520, the average tooth shape from block 510 may be placed at its corresponding average tooth pose at block 530. In other words, the order of blocks 520 and 530 may be swapped.

[0111] At block 540, a tooth pose adjustment is performed on the average tooth pose. As discussed herein, such adjustment is based on the patient's specific teeth, e.g., to adjust the shape of the teeth based on a scan of the patient's teeth, a 2D image, or other imaging techniques. In some embodiments, as discussed above, the pose adjustment T τ is based on the specific tooth poses of the patient's dental arch. In some embodiments, at block 540, as discussed herein, the position and orientation of each tooth 522 are adjusted such that the tooth is placed at the position and orientation determined by or otherwise based on the position and orientation of the teeth in the imaged patient's dental arch. In some embodiments, for example, the adjusted tooth pose may be rendered on a screen for viewing by the patient or a dental professional. For example, the adjusted tooth pose 542 is rendered for viewing. In some embodiments, the adjusted tooth pose may be stored in memory. The adjusted tooth pose may also be stored as a set of matrices and / or data structures, e.g., each matrix and / or data structure corresponding to a tooth in the tooth pose. In some embodiments, prior to adjusting the shape of the teeth at block 520, the average tooth shape from block 510 may be placed at its corresponding adjusted tooth pose at block 540. In other words, the order of blocks 520 and the combination of blocks 530 and 540 may be swapped such that block 520 occurs after blocks 530 and 540.

[0112] At block 550, the dental arch is scaled such that the dental arch is generated according to the patient's teeth and dental arch dimensions. In some embodiments, as discussed above, the dental arch scaling factor Φ is based on the particular teeth and dental arch specific to the patient. In various embodiments, for example, when scaling a 3D model for integration into a 2D image, the dental arch scaling factor can also be based on one or more of the image dimensions of the patient's 2D image. In some embodiments, at block 550, as discussed herein, the dimensions of each tooth 522 and the dental arch are adjusted such that the scaled dental arch matches the dimensions of the patient's dental arch, where the dimensions of the patient's dental arch are determined, for example, by the dimensions of the teeth and dental arch in the imaged patient's dental arch or otherwise. In some embodiments, for example, the scaled dental arch can be rendered on a screen for viewing by the patient or a dental professional. For example, the scaled dental arch 552 is rendered for viewing. In some embodiments, the scaled dental arch can be stored in memory. The scaled dental arch can also be stored as a set of matrices and / or data structures, each matrix and / or data structure corresponding to a tooth in a tooth pose. In some embodiments, the average tooth shape from block 510 can be placed in its corresponding scaled position and dimensions at block 550 before adjusting the shape of the teeth at block 520. In other words, the order of blocks 520 and blocks 530, 540, and 550 can be swapped such that block 520 occurs after blocks 530, 540, and 550.

[0113] The blocks of method 500 can be executed in an order other than Figure 5 the order shown. For example, blocks 510 and 530 can be executed before blocks 520, 540, and 550. In some embodiments, blocks 510, 520, 530, and 550 can be executed before block 540. These and other modifications to the order of the blocks in method 500 can be made without departing from the spirit of the present disclosure.

[0114] Note Figure 6 , Figure 6 that method 600 for constructing a 3D model from 2D images is shown in accordance with one or more embodiments disclosed herein.

[0115] At block 610, a 2D image of the patient is captured. In some embodiments, the 2D image includes one or more images of the patient's mouth and face, head, neck, shoulders, torso, or the patient as a whole. The 2D image of the patient can include images of the patient's mouth in one or more positions. For example, the patient's mouth can be in a smiling position (e.g., a social smile position), a resting position with relaxed muscles and slightly open lips, or a retruded anterior open bite or anterior closed bite position.

[0116] In some embodiments, an image of a patient is captured using an image capture system. The image can be captured using a lens with a predetermined focal length at a certain distance from the patient. The image can also be captured remotely and then received for processing. The image can be collected from a storage system, a network location, a social media website, etc. The image can be a series of images of a video captured from one or more perspectives. For example, the images can include one or more of a front face image and a side image, and the side image includes one or more three-quarter side images and full side images.

[0117] At block 620, the edges of the patient's oral features are determined. For example, the edges of one or more of the patient's teeth, lips, and gums can be determined. Machine learning algorithms such as convolutional neural networks can be used to identify the patient's lips (e.g., the inner edge of the lips defining an open mouth), teeth, and a preliminary determination of the gum contour. The machine learning algorithm can be trained based on pre-identified markers of the lips, teeth, and gums visible within the patient's 2D image. The initial contour can be a weighted contour such that the machine learning algorithm is confident that for a given position in the image (e.g., at each pixel) it is the edge or contour of the patient's lips, teeth, or gums.

[0118] The initial contours can be extracted from the image. The initial contours can have a brightness or other scale applied to them. For example, in a grayscale image of the contour, a value between 0 and 255 can be assigned to each pixel, which can indicate the confidence of the pixel being part of the contour or can indicate the magnitude of the contour at that location.

[0119] Then, the pixels representing the contour can be subjected to binarization to change the pixels from a scale such as 0 to 255 to a binary scale such as 0 or 1, thereby creating a binarized tooth contour. During the binarization process, the value of each pixel is compared to a threshold. If the value of the pixel is greater than the threshold, then a new first value can be assigned to it, for example, the new first value is 1; and for example, if the pixel is less than the threshold, then a new second value can be assigned to it, for example, the new second value is 0.

[0120] The binarized tooth contour can be thinned, thereby reducing the thickness of the contour to, for example, a single pixel width, thereby forming a thinned contour. The width of the contour being thinned can be measured as the shortest distance from a contour pixel adjacent to a non-contour pixel on the first side of the contour to a contour pixel adjacent to a non-contour pixel on the second side of the contour. The single pixel representing the thinned contour at a particular location can be located at the midpoint of the width between the pixel on the first side and the pixel on the second side. After thinning the binarized tooth contour, the thinned contour can be a single-width contour at a position corresponding to the midpoint of the binarized contour.

[0121] At block 630, a parameterized 3D tooth and dental arch model is matched to each tooth of the patient depicted in the patient's 2D image. The matching can be based on the edges determined at block 630, also referred to as contours. Figure 7A An example of the process of matching the tooth model and the dental arch model to the edges is shown. Returning to Figure 6 , after identifying the teeth and dental arch and modeling the teeth and dental arch, or as part of such a process, missing or broken teeth can be inserted into the parameterized 3D tooth and dental arch model. For example, when a tooth is missing or severely damaged, the parameterized model of that tooth can be replaced with an average tooth model, thereby simulating a dental prosthesis, such as a dental veneer, crown, or implant prosthesis. In some embodiments, the missing tooth can remain as a space in the dental arch. In some embodiments, a broken tooth can be left as is without replacing it with an average-shaped tooth.

[0122] During the matching process, case-specific parameters of the parametric dental arch model are varied and cycled through until there is a match between the parametric dental arch model and the teeth depicted in the 2D image. This match can be determined based on the projection of the edges of the silhouette of the parametric model matching the edges of the lips, teeth, and gums identified in the 2D image.

[0123] At block 640, the parametric model of the patient's teeth is rendered. Referring to Figure 8 and examples of the process of rendering the parametric model of the teeth are described elsewhere in this document. During the rendering process, a 3D model of the patient's teeth is formed based on the data describing the patient's teeth and dental arch. For example, a 3D model of the patient's teeth is formed based on the parametric 3D model formed at block 630 or described elsewhere in this document. In some embodiments, the 3D model is directly inserted into the patient's 2D image. In these embodiments, the action in block 640 can be omitted or combined with the action in block 650 such that, for example, the 3D tooth model is rendered in 2D form for insertion into the 2D image.

[0124] Optionally, at block 640, before rendering in 2D form, a simulated treatment or viewing customization option can be applied to the 3D model, such as gum line adjustment, jaw position adjustment, missing tooth insertion, or broken tooth repair. In some embodiments, the edges of the parametric model (e.g., lips, gum line, missing teeth, or broken teeth) can be altered to show the simulation results of cosmetic or other treatments and procedures, or the 2D patient image can be adjusted for customized viewing. For example, a user such as a dental professional or a patient can adjust the gum line to simulate gum treatment. As another example, the user can choose to display, hide, or repair missing or broken teeth to simulate tooth repair or replacement procedures. In another example, the user can choose to adjust the jaw position in the simulated image before or after treatment to simulate the appearance of an open bite, closed bite, or partial open bite. The jaw position parameter can be defined by the distance between the tooth surfaces in the upper jaw and the tooth surfaces in the lower jaw, e.g., the distance between the incisal surfaces of the upper central incisors relative to the incisal surfaces of the lower central incisors. The jaw position parameter can be defined and changed by the user. For example, the distance between the incisal surfaces of the upper central incisors and the incisal surfaces of the lower central incisors can vary between -5 mm (which indicates that the lower incisors overlap the upper incisors by 5 mm) and 10 mm (which indicates a gap of 10 mm between the upper incisors and the lower incisors). The gum line can be adjusted by replacing the patient's gum line mask shape with the average gum line mask shape from the historical shape data repository. The display of the gum line adjustment can be optionally selected or adjusted by the user. Missing or broken teeth can be replaced with the average tooth shape from the historical shape data repository. The display of the missing tooth replacement or broken tooth repair can be optionally selected by the user. For example, the simulated treatment or viewing customization option can be rendered on the screen for viewing by the user, patient, or dental professional.

[0125] At block 650, the 3D tooth model is inserted into the patient's 2D image. The inner lip edge determined at block 620 can be used to define the contour of the patient's mouth in the 2D image. At block 650, the region of the 2D image defined by the open mouth can be removed, and the 3D model can be placed behind the 2D image and within the open mouth. In some embodiments, inserting the 3D tooth model into the 2D image further includes rendering an image of the parametric gum line in the 2D image. For example, the parametric gum line can be determined as shown and described in Figure 20 Method 2001.

[0126] At block 660, a texture is applied to the teeth. Refer to Figure 9Examples of applying textures are discussed in more detail. In some embodiments, when applying a texture to a tooth, a 2D image of a patient is projected onto a 3D model of the tooth, such as the parametric 3D model of the patient's tooth determined, for example, at block 630. When the 2D image is projected into the 3D model, pixels at each location in the 2D image are assigned to locations on the 3D model. In some embodiments, the pixel values or texture information from the 2D image are processed before being applied to the 3D model. Other techniques such as image inpainting, blurring, image processing filters, pix2pix transformation techniques, etc. can also be used to generate textures applied to the surface of the 3D model. The projected pixels at each location on the surface of the 3D model form the texture for that model. For example, the pixels projected onto a particular tooth of a patient form the texture for that tooth model. This texture can be applied to the 3D tooth model.

[0127] Figure 7A Method 700 for constructing a patient-specific parametric model of a patient's tooth according to some embodiments is shown.

[0128] At block 710, the rough alignment of each corresponding average parametric tooth is aligned with the respective center of the tooth in the 2D image. The center of the parametric tooth can be determined based on the center of the area of the projection of the silhouette of the parametric tooth. The center of the tooth identified in the 2D image of the patient can be determined based on the center of the area defined by the tooth edge and the corresponding lip edge and / or gum edge. Before the desired step and the maximization step of blocks 720 and 730, the center of the 2D image tooth and the center of the parametric tooth can be aligned, respectively.

[0129] Method 700 can dynamically generate a parametric tooth model with a lip edge and a gum edge that match the tooth in the 2D image at block 710. Additionally or alternatively, the lip edge and / or gum edge can be applied and / or dynamically adjusted at any one of blocks 720, 730, and 740. The 3D tooth model can be computed on the fly for changing the placement of the lip and gum lines. The parametric models provided using these models are much more accurate than the parametric models provided using only the gum line alone.

[0130] When applying lip and gum information at block 710, in some embodiments, only the portion of the parametric model that extends beyond the lip edge and / or gum edge is used to determine the center of the area of the tooth. Turning Figure 7B , this figure depicts an example of a tooth model with a gum edge 760 and a tooth model with a lip edge 770. At any one of blocks 710, 720, 730, 740, the position of the gum edge and / or lip edge can be modified to adjust the fit of the silhouette of the tooth to the visible portion of the corresponding tooth in the 2D image.

[0131] Adding the lip margin or gingival margin of the teeth can significantly improve efficiency and the operation process. As a result, the process matches the tooth model to the 2D image much more realistically compared to the process of not applying the lip margin and gingival margin on the tooth model, and the process uses a greater variety of photos.

[0132] Return to Figure 7A , at block 720, perform the desired steps. In some embodiments, the Expectation Management (EM) engine and / or an engine configured to create a 3D model perform block 720. At the desired steps, project the silhouette of the teeth onto the 2D image, and evaluate the edge of the silhouette against the edge of the teeth in the 2D image determined based on the lip margin, gingival margin, and tooth edge. This evaluation can be determining the normal at the position at the edge of the silhouette and the nearest position in the 2D image with a similar normal. Then determine the probability that the two edges are the same. This process can be repeated for each position at the edge of the silhouette.

[0133] At block 730, perform the maximization step of the EM engine. In the maximization step, use a small angle approximation to provide a maximized analytical solution. Compared to other methods such as the Gauss - Newton iteration method, the small angle approximation provides an improved solution with the analytical solution. The small angle approximation greatly reduces the computation time and solves for the exact solution faster.

[0134] Blocks 720 and 730 can be iteratively performed on a single parameter or a subset of parameters, thus performing the desired steps, then the maximization step, then returning to the desired steps, and so on until a threshold for convergence of the single parameter or subset of parameters is reached. Then, the process can proceed to block 740.

[0135] At block 740, add the optimized parameter or subset of parameters to the parameter model. For example, the optimization of the parameter model can start with Φ and T, and then after cycling through EM blocks 720 and 730, additional parameters are added. For example, T τ , and then it is optimized through EM blocks 720 and 730 for additional iterations. The number of iterations before adding additional parameters can vary. In some embodiments, EM blocks 720 and 730 can be cycled through 3 times, 5 times, 7 times, 10 times, 15 times, 20 times, or any number of times (e.g., any integer). Finally, Add it to the parametric model and process the parametric model through EM blocks 720 and 730 until convergence is achieved. During this process, outliers can be identified and filtered out. In some embodiments, after block 740, process 700 can loop back to block 710 instead of looping back to block 720 and proceeding directly to the desired step, where, in block 710, a rough alignment process is performed based on the updated parametric model.

[0136] At block 750, the parameters of the parametric model are output to another engine or even to a data warehouse for later retrieval. For example, as described with reference to Figure 8 and elsewhere in this document, a rendering engine can retrieve the parameters of the parametric model for rendering.

[0137] Figure 8 Depicts a method 800 for rendering a patient's teeth in an initial position using a parametric model of a patient's dental arch according to one or more embodiments of the present document.

[0138] At block 810, the average shape of each tooth is determined In some embodiments, as discussed herein, the average shape can be based on, for example, the average shape of a set of dental arches obtained from historical cases and / or cases representing an ideal dental arch form. In some embodiments, for example, the average shape can be rendered on a screen for viewing by a patient or a dental professional. In some embodiments, the average shape can be loaded into memory or retrieved from memory. The average shape can also be initialized as a set of matrices, each matrix corresponding to a tooth.

[0139] At block 820, perform a principal component analysis shape adjustment on the average shape of the teeth For each tooth in the model, apply the case-specific coefficients of the principal components to the principal components After completing the shape adjustment, in some embodiments, for example, the adjusted shape can be rendered on a screen for viewing by a patient or a dental professional. For example, render the adjusted tooth shape for viewing. In some embodiments, the adjusted shape can be stored in memory. The adjusted shape can also be stored as a set of matrices, each matrix corresponding to a tooth.

[0140] At block 830, an average tooth pose is determined. In some embodiments, as discussed herein, the average tooth pose can be based on the average tooth pose of a set of scanned dental arches. In some embodiments, at block 830, each adjusted tooth is placed at its corresponding average position and orientation determined by the average dental arch. In some embodiments, for example, the average tooth pose can be rendered on a screen for viewing by a patient or a dental professional. In some embodiments, the average tooth pose can be loaded into a memory or retrieved from a memory. The average tooth pose can also be initialized as a set of matrices and / or other data structures, for example, each matrix and / or other data structure corresponding to a tooth in a tooth pose. In some embodiments, before adjusting the shape of the teeth at block 820, the average tooth shape from block 810 can be placed at its corresponding average tooth pose at block 830. In other words, the order of blocks 820 and 830 can be swapped.

[0141] At block 840, a tooth pose adjustment is performed based on the average tooth pose. In some embodiments, as discussed above, the pose adjustment T τ is based on the specific tooth poses of the patient's dental arches. In some embodiments, as discussed herein, at block 840, the position and orientation of each tooth are adjusted such that it is placed at a position and orientation determined by or otherwise based on the positions and orientations of the teeth in the patient's dental arches. In some embodiments, for example, the adjusted tooth pose can be rendered on a screen for viewing by a patient or a dental professional. In some embodiments, the adjusted tooth pose can be stored in a memory. The adjusted tooth pose can also be stored as a set of matrices and / or other data structures, for example, each matrix and / or data structure corresponding to a tooth in a tooth pose. In some embodiments, before adjusting the shape of the teeth at block 820, the average tooth shape from block 810 can be placed at its corresponding adjusted tooth pose at block 840. In other words, the order of blocks 820 and the combination of blocks 830 and 840 can be swapped such that block 820 occurs after blocks 830 and 840.

[0142] At block 850, the dental arch is scaled such that the resulting dental arch has tooth dimensions according to the patient's teeth and dental arch dimensions. In some embodiments, as discussed above, the dental arch scaling factor Φ is based on the particular teeth and dental arch specific to the patient. In some embodiments, at block 850, as discussed herein, the dimensions of the dental arch are adjusted such that the scaled dental arch matches the dimensions of the patient's dental arch, which may be determined, for example, by the dimensions of the teeth and dental arch in the patient's dental arch or otherwise. In some embodiments, for example, the scaled dental arch may be rendered on a screen for viewing by the patient or a dental professional. In some embodiments, the scaled dental arch may be stored in a memory. The scaled dental arch may also be stored as a set of matrices and / or data structures, for example, each matrix and / or data structure corresponding to a tooth in a tooth pose. In some embodiments, the average tooth shape from block 810 may be placed in its corresponding scaled position and dimensions at block 850 before adjusting the shape of the teeth at block 820. In other words, the order of block 820 and blocks 830, 840, and 850 may be swapped such that block 820 occurs after blocks 830, 840, and 850.

[0143] The blocks of method 800 may be executed in an order other than the Figure 8 order shown. For example, blocks 810 and 830 may be executed before blocks 820, 840, and 850. In some embodiments, blocks 810, 820, 830, and 850 may be executed before block 840. These and other modifications to the order of the blocks in method 800 may be made without departing from the spirit of the present disclosure.

[0144] Figure 9 Method 900 for constructing a 3D model and applying a texture to the 3D model in accordance with one or more embodiments herein is depicted. The texture may help provide details such as color information to the 3D model of the patient's teeth. As described herein, process 900 uses an image of the patient's teeth to provide a realistic texture to the patient's teeth.

[0145] Thus, as described elsewhere herein, for example, in the discussion Figure 5 relating to, at block 910, a 3D model of the patient's teeth and a 2D image of the patient are obtained. The 2D image and the 3D model should depict the teeth in the same position, e.g., the 3D model may be a parametric model derived from the patient's 2D image.

[0146] At block 920, a 2D image of a patient's teeth is projected onto a 3D model and the image is aligned with the model. This alignment can be performed by matching the contours in the 3D model with the contours in the 2D image. As described elsewhere herein, the contours can include the determined lip edges, gum edges, and tooth edges.

[0147] At block 930, color information from the 2D image is texture mapped onto the 3D model. The color information can include lighting conditions such as specular highlights, accurate tooth coloring and texture, and other tooth features such as dental ornaments, gold teeth, etc. Once the texture is mapped onto the 3D model, the teeth in the 3D model can be repositioned, for example, to depict the estimated and / or expected results of a dental treatment plan (e.g., an orthodontic treatment plan, a restorative treatment plan, some combination thereof, etc.). As referenced Figure 10A 、 Figure 10B and / or described elsewhere herein, this final position includes both accurate color and tooth feature information and an accurate 3D positioning of the patient's teeth, and can be used, for example, to simulate the estimated and / or expected results of a dental treatment plan (e.g., a final orthodontic position) for the patient's teeth.

[0148] In some embodiments, the texture model is adjusted to simulate clinical or cosmetic treatments. For example, the color can be adjusted to simulate a teeth whitening procedure. Simulating these treatments can include generating a mask from a 2D projection of the 3D tooth model, where the model is in an arrangement inserted into the 2D image, e.g., in an initial or final position. The mask can be applied to the 2D image of the patient's teeth in the initial or final position. Color adjustment or whitening can also be applied to the masked region. The color adjustment and whitening parameters can be optionally selected and adjusted by the user. For example, the color adjustment and whitening can be rendered on a screen for viewing by the user, the patient, or a dental professional.

[0149] Figure 10A Method 1000 for simulating the estimated and / or expected results of a dental treatment plan for a patient's teeth in accordance with one or more embodiments herein is shown.

[0150] At block 1010, a model of a set of average dental arches is constructed. In some embodiments, the model is a parametric 3D model of a set of scanned historical dental arches and / or an average dental arch representative of an ideal dental arch form. In some embodiments, the scans are from the initial positions of a patient's teeth. In some embodiments, the scans are from a patient after completion of orthodontic treatment. In other embodiments, the scans are taken regardless of whether the patient has received orthodontic treatment. In some implementations, the historical cases and / or ideal cases may represent dental arch models of treated patients (e.g., patients who have been treated in the past) and / or ideal dental arch models representing the expected outcomes of various forms of orthodontic treatment. Block 1010 may include reference Figure 3B to process 350 described. At block 1010, cases are obtained. These cases may be retrieved from a data warehouse and may include previously scanned and segmented dental arch models.

[0151] These dental arches may also be aligned. Each dental arch in the data warehouse may be aligned at multiple locations. For example, each dental arch may be aligned at three locations: between the central incisors, and at each distal end on the left and right sides of each dental arch. For example, Figure 4 shows a set of dental arches 400 that are aligned at a location between central incisors 410, at the left distal end 430 of the dental arch, and at the right distal end 420 of the dental arch.

[0152] In some embodiments, determining the average dental arch model and determining the distribution of dental arch models includes performing sub-steps. For example, each historical dental arch may be rescaled to determine Φ, and then averaged for each dental arch to determine Then the local deformations of each tooth are determined and compared with to determine T τ , and then after aligning each tooth, β τ is determined.

[0153] A distribution estimate of case-specific parameters is also determined. For example, the relative shape, position, and rotation of each tooth are determined to construct a distribution of each case-specific parameter. For example, for the surface model of each corresponding tooth in all retrieved models, the coefficients of the principal components are determined of the distribution.

[0154] The position and orientation of each tooth may be averaged to determine the average position of each tooth, and the orientation of each tooth may be averaged to determine the average orientation of each tooth. The average position and average orientation are used to determine

[0155] Finally, the average of the distribution of the estimates of the case-specific parameters may be used as a general parameter for the average tooth position and average tooth shape.

[0156] At block 1020, a 2D image of the patient is captured. In some embodiments, the 2D image includes an image of the patient's mouth and one or more of the face, head, neck, shoulders, torso, or the patient as a whole. The 2D image of the patient may include an image of the patient with the patient's mouth in one or more positions. For example, the patient's mouth may be in a smiling position (e.g., a social smiling position), a resting position with relaxed muscles and slightly parted lips, or a retruded anterior open bite or anterior closed bite position.

[0157] In some embodiments, an image of the patient is obtained, e.g., by taking an image of the patient using an image capture system. The image may be captured using a lens with a predetermined focal length at a certain distance from the patient. In some implementations, the image of the patient is obtained from a computer storage device, a network location, a social media account, etc. The image may also be captured remotely and then received for processing. The image may be a series of images from a video taken from one or more perspectives. For example, the images may include one or more of a frontal face image and a side image, and the side image includes one or more three-quarter side images and full side images.

[0158] At block 1030, a 3D model of the patient's teeth is constructed from the 2D image of the patient's teeth. The 3D model may be based on a parametric model constructed according to the process described in the reference Figure 6 where the edges of the patient's oral features are determined. For example, the edges of one or more of the patient's teeth, lips, and gums may be determined. The initial determination of the patient's lips (e.g., the inner edges of the lips defining an open mouth), teeth, and gum contours may be identified by a machine learning algorithm such as a convolutional neural network.

[0159] Then, an initial contour may be extracted from the image. Then, the pixels representing the contour may be binarized. The binarized tooth contour is thinned to reduce the thickness of the contour to, for example, a single pixel width to form a thinned contour.

[0160] Using the thinned or otherwise processed contour, a parameterized 3D tooth and dental arch model is matched to each tooth of the patient depicted in the 2D image of the patient. This matching is based on the contour determined above. After identifying and modeling the teeth and dental arch, or as part of such a process, missing or broken teeth may be inserted into the parameterized 3D tooth and dental arch model.

[0161] At block 1040, case-specific parameters are estimated. In some embodiments, the case-specific parameters may be determined at block 1030 as being based on, for example, in Figure 6Part of the model construction of the process described in. During the matching process or at block 1040, the case-specific parameters of the parametric dental arch model are changed and looped through the case-specific parameters until a match is found between the parametric dental arch model and the teeth depicted in the 2D image. As described herein, this match can be determined based on the projection of the edges of the silhouette of the parametric model matching the edges of the lips, teeth, and gums identified in the 2D image.

[0162] At block 1050, the patient's teeth are rendered according to the simulation results of a dental treatment plan that uses the case-specific parameters for the tooth shape or dental arch scale of the patient and also uses the average values of the tooth positions. Figure 11 An example of a method for rendering teeth according to a dental treatment plan and / or the expected results of the dental treatment plan using appropriate case-specific parameter values and average parameter values is shown.

[0163] Optionally, at block 1050, other changes such as gum line adjustment, jaw position adjustment, missing tooth replacement, or broken tooth repair can be applied to the 3D model before rendering it in 2D form. In some embodiments, the edge features and other features of the model, such as the lips, gum line, and missing or broken teeth, can be changed to show the simulation results of a treatment procedure, or the 2D image of the patient can be adjusted for customized viewing. For example, the user can adjust the gum line to simulate gum treatment. In another example, the user can choose to display, hide, or repair missing or broken teeth to simulate a tooth repair or replacement procedure. In this case, the missing or broken teeth can be replaced with ideal teeth based on the ideal parameters discussed above. The missing or broken teeth can be replaced based on a tooth in the historical data warehouse, or with the patient's own teeth. For example, a corresponding tooth from the opposite side of the patient's dental arch can be used. For example, if the upper left canine tooth is missing or damaged, then a mirror model of the patient's upper right canine tooth can be used in the position of the missing upper left canine tooth. In another example, the user can choose to adjust the jaw position in the simulated image before or after treatment to simulate the appearance of an open bite, closed bite, or partial open bite. The jaw position parameter can be defined by the distance between the tooth surfaces in the upper jaw and the tooth surfaces in the lower jaw. For example, the distance between the incisal surfaces of the upper central incisors relative to the incisal surfaces of the lower central incisors. The jaw position parameter can be defined and changed by the user. The gum line can be adjusted by replacing the patient's gum line mask shape with the average gum line mask shape from the historical average shape database. The display of the gum line adjustment can be optionally selected or adjusted by the user. The missing or broken teeth can be replaced with the average tooth shape from the historical average shape database. The display of the missing tooth replacement or broken tooth repair can be optionally selected by the user. For example, the simulated treatment or viewing customization options can be rendered on the screen for viewing by the user, patient, or dental professional.

[0164] At block 1060, the 3D model is inserted into the 2D image of the patient. For example, as part of the model construction process, the 3D rendering of the teeth can be placed in the patient's mouth defined by the lip contour determined at block 1030. In some embodiments, inserting the 3D tooth model into the 2D image also includes rendering an image of a parameterized gum line in the 2D image. For example, it can be as described with respect to Figure 20The method 2001 as shown and described is used to determine the parameterized gingival line. In some embodiments, in block 1060, the texture model is adjusted to simulate clinical aesthetic treatment. For example, the color can be adjusted to simulate a teeth whitening procedure. A mask can be generated from a 2D projection of the 3D tooth model. The mask can be applied to the 2D image of the patient's teeth at the initial position or the final position. Color adjustment or whitening can also be applied to the masked area. The color adjustment and whitening parameters can be optionally selected and adjusted by the user.

[0165] The user can select, deselect, or adjust the simulated treatment or view customization options performed at block 1050 or block 1060 on the projected 2D image. Parameters can be adjusted to simulate treatments such as whitening, gingival line treatment, tooth replacement, or tooth restoration, or for viewing customization options such as showing an open bite, a closed bite, or a partial open bite. In one example, the user can optionally adjust the color parameters to simulate teeth whitening. The user can adjust the color parameters to increase or decrease the degree of teeth whitening. In another example, the user can optionally adjust the gingival line parameters for oral hygiene to simulate gingival line treatment or change. The user can adjust the gingival line parameters to raise the gingival line (reduce the amount of tooth surface exposure), thus simulating, for example, gingival recovery due to improved oral hygiene or gingival line treatment, or lower the gingival line (increase the amount of tooth surface exposure), thus simulating, for example, gingival recession due to poor oral hygiene. In another example, the user can optionally select the tooth replacement parameters. The user can select the tooth replacement parameters to replace one or more missing, broken, or damaged teeth with ideal teeth (such as teeth from a historical data warehouse or the patient's own teeth). In another example, the user can optionally adjust the jaw line parameters for a custom view of the jaw position. The user can increase the distance between the teeth in the upper jaw relative to the teeth in the lower jaw to simulate an open bite, or decrease the distance between the teeth in the upper jaw relative to the teeth in the lower jaw to simulate a closed bite. In this case, the user can simulate the appearance of orthodontic treatment, aesthetic treatment, or clinical treatment at various jaw positions.

[0166] Figure 10B Process 1005 is depicted for simulating the final treatment position of a patient's teeth using a matching dental arch identified from a parameterized search of a treatment plan data warehouse.

[0167] As described at Figure 10A in block 1020, at block 1015, a 2D image of the patient is captured, and as in Figure 10A block 1030, at block 1025, a 3D model of the patient's teeth is constructed from the 2D image of the patient's teeth. As in Figure 10A block 1040, at block 1035, case-specific parameters are estimated.

[0168] At block 1045, the tooth shape parameters of the patient's teeth from the 3D model are used to perform a parametric search of the treatment plan data warehouse. When matching the shape of the patient's teeth to the shape of the teeth in the historical treatments in the data warehouse, the shape of the teeth in the patient's parametric 3D model is compared to the parametric shape of the teeth in each of the historical records in the treatment data warehouse. When a match between the tooth shapes is found, the final tooth position and orientation or the final dental arch model of the matching record can be used to simulate the treatment plan. A parametric template dental arch can be identified based on the comparison of the tooth shape parameters, thereby identifying the matching template dental arch. In some embodiments, for example, the match can be based on the closest matching template dental arch. In some embodiments, the template shape can be loaded into memory or retrieved from memory. The template shape can also be initialized as a set of matrices, each matrix corresponding to a tooth. Once a match is found in a record within the treatment data warehouse, the final dental arch model is retrieved from the matching record and can be used as a basis for the patient's target final tooth position.

[0169] In some embodiments, the tooth position and orientation in the matching record are used as a basis for the tooth position and orientation of the patient's target final tooth position. When using the tooth position and orientation, the tooth position and orientation in the matching record can be used to modify the patient's parametric dental arch model, or the shape of the patient's teeth can be used to update the matching record. In some embodiments, the model of each tooth of the patient with the tooth shape unchanged is placed in the final tooth pose determined from the matching record. In some embodiments, the model of each tooth of the patient with the shape adjusted (e.g., the tooth shape from the matching record) is placed in the final tooth pose determined from the matching record. Optionally, a tooth pose adjustment can be performed to adjust the position of the teeth in the final tooth pose.

[0170] In some embodiments, the final dental arch model from the matching record is used as the final position for the simulation of the patient's treatment in 2D rendering. In some embodiments, the teeth of the final dental arch model from the matching record are placed at positions based on the positions of the teeth in the patient's 2D photograph. Additionally, the teeth of the final dental arch model are placed at positions based on the positions of the teeth in the patient's parametric 3D model. Optionally, the final dental arch model with the position adjusted according to the patient's 2D photograph or 3D model can be used as the simulation position in 2D rendering.

[0171] The shape of the template teeth can be adjusted based on the patient's parametric 3D model to more closely match the shape of the patient's teeth. Optionally, a principal component analysis shape adjustment can be performed on the template tooth shape. After completing the shape adjustment, in some embodiments, for example, the adjusted shape can be rendered on a screen for viewing by a user, patient, or dental professional. In some embodiments, the adjusted shape can be stored in a memory. The adjusted shape can also be stored as a set of matrices, each matrix corresponding to a tooth. In some embodiments, the patient's teeth are used to replace the teeth in the matching template from the treatment data repository.

[0172] The template dental arch can be scaled so that the resulting dental arch is based on the patient's tooth and dental arch dimensions. In some embodiments, as discussed above, the template dental arch scaling factor Φ is based on specific teeth and dental arches specific to the patient. In some embodiments, for example, when scaling a 3D model for integration into a 2D image, the template dental arch scaling factor can also be based on one or more of the image dimensions of the patient's 2D image. In some embodiments, as discussed herein, the dimensions of each template tooth and the template dental arch are adjusted so that the scaled template dental arch matches the dimensions of the patient's dental arch, the dimensions of which are determined, for example, by the teeth and dental arch in the scanned patient dental arch or otherwise. In some embodiments, for example, the scaled template dental arch can be rendered on a screen for viewing by a user, patient, or dental professional. In some embodiments, the scaled template dental arch can be stored in a memory. The scaled template dental arch can also be stored as a set of matrices, each matrix corresponding to a tooth in a tooth pose. In some embodiments, the ideal template tooth shape can be placed in its corresponding scaled position and dimensions before adjusting the shape of the teeth.

[0173] At block 1045, simulation treatment or viewing customization options such as gum line adjustment, jaw position, missing tooth implantation, or broken tooth repair can be applied to the 3D model before rendering it in 2D form. As described in block 640, simulation treatment or viewing customization option changes can be applied and adjusted.

[0174] At block 1055, the 3D model is inserted into the patient's 2D image. For example, a 3D rendering of the teeth can be placed in the patient's mouth defined by the lip contour, which was determined as part of the model building process at block 1025. The rendering can be performed at block 1045 with or without implementing simulation treatment or viewing customization options. In some embodiments, inserting the 3D tooth model into the 2D image also includes rendering an image of a parametric gum line in the 2D image. For example, it can be as described with respect to Figure 20The method shown and described in 2001 is used to determine the parameterized gingival line. In some embodiments, at block 1055, the texture model is adjusted to simulate clinical treatment. For example, the color can be adjusted to simulate a teeth whitening procedure. A mask can be generated from a 2D projection of the 3D tooth model. The mask can be applied to a 2D image of the patient's teeth at an initial position or a final position. Color adjustment or whitening can also be applied to the masked area. The color adjustment and whitening parameters can be optionally selected and adjusted by the user.

[0175] The user can select, deselect, or adjust the simulated treatments and viewing customization options performed at block 1045 or block 1055 on the projected 2D image. Parameters can be adjusted to simulate treatments such as whitening, gingival line treatment, tooth replacement, or tooth restoration, or for viewing customization options such as showing an open bite, a closed bite, or a partial open bite.

[0176] Figure 11 An example of a method 1100 for rendering teeth based on estimated and / or expected results of a dental treatment plan is shown. The dental treatment plan is based on patient-derived values using case-specific parameters of a scale and tooth shape, and on a parametric model of the patient's dental arch using an average value of tooth positions (e.g., an average value derived from a historical dental arch and / or an ideal dental arch).

[0177] At block 1110, the average shape of each tooth is determined In some embodiments, as discussed herein, the average shape can be based on the average shape of a set of scanned dental arches. In some embodiments, for example, the average shape can be rendered on a screen for viewing by the patient or a dental professional. In some embodiments, the average shape can be loaded into memory or retrieved from memory. The average shape can also be initialized as a set of matrices, with each matrix corresponding to a tooth.

[0178] At block 1120, the average shape of the teeth is subjected to principal component analysis shape adjustment. For each tooth in the model, the case-specific coefficients of the principal components are applied to the principal components After the shape adjustment is completed, in some embodiments, for example, the adjusted shape can be rendered on a screen for viewing by the patient or a dental professional. In some embodiments, the adjusted shape can be stored in memory. The adjusted shape can also be stored as a set of matrices and / or data structures, e.g., with each matrix and / or data structure corresponding to a tooth.

[0179] At block 1130, an average tooth pose is determined. In some embodiments, as discussed herein, the average tooth pose may be based on the average tooth pose of a set of scanned dental arches. In some embodiments, at block 1130, each adjusted tooth is placed at its corresponding average position and orientation determined by the average dental arch. In some embodiments, for example, the average tooth pose may be rendered on a screen for viewing by a patient or a dental professional. In some embodiments, the average tooth pose may be loaded into a memory or retrieved from a memory. The average tooth pose may also be initialized as a set of matrices, each matrix corresponding to a tooth in a tooth pose. In some embodiments, prior to adjusting the shape of the teeth at block 1120, at block 1130, the average tooth shape from block 1110 may be placed at its corresponding average tooth pose. In other words, the order of blocks 1120 and 1130 may be swapped.

[0180] At block 1140, the dental arch is scaled such that the resulting dental arch has tooth dimensions according to the patient's tooth and dental arch dimensions. In some embodiments, as discussed above, the dental arch scaling factor Φ is based on the specific teeth and dental arch of a particular patient. In some embodiments, as discussed herein, at block 1140, the dental arch is adjusted such that the scaled dental arch matches the dimensions of the patient's dental arch, which dimensions are determined, for example, from the teeth and dental arch in the scanned patient dental arch or otherwise. In some embodiments, for example, the scaled dental arch may be rendered on a screen for viewing by a patient or a dental professional. In some embodiments, the scaled dental arch may be stored in a memory. The scaled dental arch may also be stored as a set of matrices, each matrix corresponding to a tooth in a tooth pose. In some embodiments, prior to adjusting the shape of the teeth at block 1120, at block 1140, the average tooth shape from block 1110 may be placed at its corresponding scaled position and dimensions. In other words, the order of block 1120 and blocks 1130 and 1140 may be swapped such that block 1120 occurs after blocks 1130 and 1140.

[0181] Prior to rendering at block 1150, simulation treatments or viewing customization options such as gum line adjustment, jaw position, missing tooth insertion, or broken tooth repair may be applied to the 3D model. As described in block 640, simulation treatments or viewing customization options may be applied and adjusted. The simulation treatments or viewing customization options may be applied before or after dental arch scaling (e.g., before or after block 1140).

[0182] At block 1150, the 3D model of the patient's teeth determined at block 1140 is rendered for viewing, or the final 3D model is stored for later rendering on a screen. The 3D model can be rendered for viewing with or without simulating treatment or viewing customization options, or the 3D model can be stored for later rendering on a screen.

[0183] At block 1160, a texture is applied to the 3D model of the patient's teeth at the estimated final position and / or the expected final position. For example, a texture determined based on a 2D image of the patient's teeth according to, for example, Figure 9 Process 900 in can be applied to the 3D model. In some embodiments, the texture application process of block 1160 can occur during or as part of the rendering process of block 1150. The texture can be applied to a mask region, where the mask region can define an area corresponding to an oral feature (e.g., lips, gums, or teeth). For example, the mask region or a portion thereof can be determined by simulating gums as shown and described in Method 2001 in Figure 20 . During texture application, whether at block 1160 or during the rendering process of block 1150, the texture model can be adjusted to simulate treatment or for customizable viewing. Simulation treatments or viewing customization options including color adjustment to simulate, for example, teeth whitening can be applied to the 3D model or can occur as part of the rendering process. Alternatively, color adjustment can be performed as described at block 1060. Simulation treatments and viewing customization options, color adjustment can be adjusted by the user.

[0184] Optionally, step 1110 can be replaced with a parametric search of a treatment planning data warehouse to identify a matching template model based on the patient's tooth shape. As described using a template dental arch and tooth shape instead of an average dental arch or tooth shape, steps 1120 through 1160 can then be implemented to determine the ideal tooth pose instead of the average tooth pose.

[0185] Note Figure 20 , which depicts Method 2001 for simulating a gum line using a combination of patient-specific input parameters and simulated input parameters. At block 2011, one or more 2D images of the patient are captured or retrieved. The 2D images can be captured as described at block 110 in Figure 1 . One or more 2D images of the patient are captured. The 2D images can depict, for example, the patient's mouth in a smiling position. The 2D images can also be as in Figure 6 Block 610 of, Figure 10A Block 1020 of, or Figure 10BObtained as part of the process described at block 1015. For example, a 2D image captured at block 1020 can be retrieved at block 2011 for use in generating the gums for use in the 2D image generated at block 1050 or block 1060.

[0186] At block 2021, the edges of the oral features (e.g., teeth, gums, and lips) of the patient are determined from the 2D image of the patient's mouth. The edges of the oral features (e.g., teeth, lips, and gums) can be identified as described at block 620 of Figure 6 . Facial landmarks from the 2D image of the patient can be used to identify the open mouth within which the oral features (including teeth and gums) are located.

[0187] The initial determination of the edges of the oral features can be identified by a machine learning algorithm. An initial tooth contour is extracted from the image, and a brightness or other scale can be applied to it. The initial contour can then be subjected to binarization to change the pixels of the image to a binary scale. The binarized tooth contour is thinned to, for example, a single pixel width, thereby forming a thinned tooth contour. The initial determination of the patient's lips can be based on facial landmarks, such as lip landmarks determined according to a machine learning algorithm (e.g., a convolutional neural network). As with the tooth contour, a brightness or other scale is also applied to the lip contour. Like the initial tooth contour, the lip contour is binarized, thinned, and contoured. The binarized and thinned tooth and lip contours can define the edges of the oral features. For example, the edges can be determined as described above with respect to Figure 6 .

[0188] At block 2031, patient-specific input parameters for the gums are determined. The input parameters can include Figure 21A the patient-specific input parameters shown in. In some embodiments, the patient-specific input parameters can be determined from the shape and scale of one or more oral features of the captured 2D image of the patient. The patient-specific input parameters can be determined based on the edges of the oral features, e.g., as shown at 2100 of Figure 21B . The edges of the oral features can include, for example, the lip edge 2110, the tooth edge 2120, and the gum edge 2130 determined at block 2021. The patient-specific input parameters can include the patient tooth height, which can be the distance between the incisal edge 2120 of the tooth and the edge 2130 of the gum or the gum line, and can be determined for each tooth of the patient's dental arch. For example, as shown in Figure 21A , the patient tooth height can be represented by h gRepresentation. In some embodiments, the patient tooth height can be the maximum distance between the incisal edge 2120 of the tooth and the edge 2130 of the gingiva. In some embodiments, the maximum distance can be measured as the distance between the incisal edge 2120 of the tooth and the edge 2130 of the gingiva at the highest point of the gingiva.

[0189] In some embodiments, patient-specific input parameters can include the gingival tip distance, which can be the distance between the edge 2110 of the lip and the gingival line 2140. This distance can be determined at the location between each pair of adjacent teeth in the patient's dental arch. For example, as Figure 21A shown, the gingival tip distance can be represented by h b Representation. In some embodiments, the patient tooth height can be the maximum distance between the edge 2110 of the lip and the edge 2140 of the gingiva. The maximum value can be measured as the distance between the lip and the edge 2140 of the gingiva at the lowest point of the gingiva. Patient-specific input parameters can include the gingival height, which can be the distance between the edge 2110 of the lip and the gingival line 2130 determined for each tooth in the patient's dental arch. For example, as Figure 21A shown, the gingival height can be represented by h t Representation. In some embodiments, the gingival height can be the minimum distance between the edge 2110 of the lip and the highest point 2130 of the gingiva. Patient-specific input parameters can include the tooth width determined for each tooth in the patient's dental arch. For example, as Figure 21A shown, the tooth width can be represented by w. In some embodiments, the tooth width can be the width of the tooth at the incisal edge 2120. In some embodiments, the tooth width can be the width of the tooth at the maximum tooth width.

[0190] At block 2041, the simulated parameters of the gingiva are determined. The simulated parameters can be derived based on patient-specific input parameters and can be used when generating the simulated gingiva. In some embodiments, the gingival tip parameter represented by h, the visible tooth height represented by g, and the parameter of the tooth curvature represented by R can be a convolution of patient-specific input parameters, as Figure 22 shown. For example, they can be a convolution of patient-specific parameters and simulated input parameters or ideal aesthetic parameters. "Ideal teeth" can include the representation of teeth or parameters taken from a model dental arch (e.g., the model dental arch of a historical case and / or the model dental arch representing the ideal dental arch form). Ideal teeth can be characterized by various parameters, including but not limited to: gingival tip parameters, ideal tooth height, tooth thickness, the distance from the gingival line to the lip, or the visible tooth height. Patient-specific parameters can be, for example, the parameters of the following as discussed above: the patient tooth height represented by h g represented, the gingival tip distance represented by h b represented, the gingival height represented by ht The gingival height represented by g and the tooth width represented by w. The simulated input parameters or ideal aesthetic parameters can be, for example, parameters of the following: the ideal tooth height represented by H g the customizable tooth height scaling parameters represented by c1 and c2, the customizable gingival tip scaling parameter represented by s, and the tooth shape scaling parameter represented by m. The visible tooth height represented by g can be the visible tooth height of the patient's teeth determined based on a 2D image (represented by h g ) and the ideal tooth height (represented by H g ). The visible tooth height represented by g can be determined based on a weighted average. For example, g = c1H g + c2h g , where c1 and c2 are customizable tooth height scaling parameters, and where c1 + c2 = 1. For example, c1 can be 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, or 0.9. For example, c1 can be from 0.1 to 0.5, 0.2 to 0.6, 0.3 to 0.7, 0.4 to 0.8, or 0.5 to 0.9. In some embodiments, c1 is adjusted by a user such as a dental professional or a patient.

[0191] In some embodiments, the visible tooth height represented by g represents the visible tooth height of the patient's teeth determined based on a 2D image (represented by h g ). The gingival tip parameter h for each tooth can be determined based on patient-specific input parameters. For example, where h = h b – h t . The gingival tip parameter describes the shape of the gingiva around the tooth. A gingival tip parameter of 0 describes a gingival line having a straight line across the tooth, and a positive gingival tip parameter describes a gingival line having a curved shape, where the distance between the lip and the gingival line is greater between two teeth than at the midline of the tooth. In some embodiments, the gingival tip parameter h can be determined based on the difference between the patient-specific tooth height represented by h g , the ideal tooth height represented by H g , and the customizable gingival tip scaling parameter represented by s. For example, where h = s(H g – h g )(h b – h t ), where s is determined by the difference between H g and h g . For example, when the difference is large, it is likely a rectangular tooth, and s takes a larger value; when the difference is small, s takes a relatively small value. In some embodiments, it can be based on the ideal tooth length H g and the patient tooth length h gAdjust s by the difference. For example, s can be 0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, or 2.0. For example, s can vary from 0.8 to 1.0, from 0.9 to 1.1, from 1.0 to 1.2, from 1.1 to 1.3, from 1.2 to 1.4, from 1.3 to 1.5, from 1.4 to 1.6, from 1.5 to 1.7, from 1.6 to 1.8, from 1.7 to 1.9, from 1.8 to 2.0, or from 0.8 to 2.0. In some embodiments, s is adjusted by a user such as a dental professional or a patient.

[0192] For each tooth, the maximum threshold of the gingival tip parameter represented by h can be set to be a portion of the patient tooth height represented by h g For example, the gingival tip parameter h can be set to not exceed 0.2h g 、0.3h g 、0.4h g 、0.5h g 、0.6h g 、0.7h g 、0.75h g 、0.8h g 、0.85h g or 0.9h g 。The maximum threshold of the gingival tip parameter h can be set to be a portion of the visible tooth height represented by g. For example, h may not exceed 0.2g, 0.3g, 0.4g, 0.5g, 0.6g, 0.7g, 0.75g, 0.8g, 0.85g, or 0.9g.

[0193] The tooth curvature parameter represented by R can be a function of the patient-specific gingival width parameter represented by h t 、the tooth thickness represented by T, and the tooth shape scaling parameter represented by m. In some embodiments, the tooth curvature parameter R can be determined by R = mh t T. In some embodiments, the tooth thickness represented by T can be determined based on a parametric 3D model of the patient's tooth. For example, where T is the distance between the lingual and buccal surfaces of the tooth at the midpoint of the gingival line. In some embodiments, m is adjusted based on the patient tooth length represented by h g 。For example, m can be 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0. For example, m can vary from 0.1 to 0.3, from 0.2 to 0.4, from 0.3 to 0.5, from 0.4 to 0.6, from 0.5 to 0.7, from 0.6 to 0.8, from 0.7 to 0.9, or from 0.8 to 1.0. In some embodiments, m is adjusted by a user such as a dental professional or a patient.

[0194] In some embodiments, input parameters (e.g., the parameters represented by h, g, and R) are used to determine a parameterized gingival line. Patient-specific input parameters, convolutions of patient-specific input parameters, and simulated input parameters, or any combination thereof, can be used to determine the gingival line. In some embodiments, determining the parameterized gingival line includes approximating a parametric 3D model of a tooth among multiple teeth of a patient as a cylinder. The cylinder can be cross-sectioned by an inclined plane, wherein the orientation and position of the inclined plane are determined based on the input parameters such that the intersection line between the inclined plane and the cylinder passes through coordinates defined by h, g, and R. The intersection line between the inclined plane and the cylinder can be used to define the edge of the gingiva. In some embodiments, determining the parameterized gingival line includes cross-sectioning a 3D model of a patient's tooth by an inclined plane, wherein the orientation and position of the inclined plane are determined based on the input parameters such that the intersection line between the inclined plane and the cylinder passes through coordinates defined by h, g, and R. The intersection line between the inclined plane and the 3D model of the patient's tooth can be used to define the edge of the gingiva. Points on the gingival side of the intersection line between the inclined plane and the cylinder or the 3D model of the patient's tooth are rendered as the gingiva, while points on the incisal side of the intersection line between the inclined plane and the cylinder are rendered as the tooth. In some embodiments, the points are 3D point clouds of points. In some embodiments, the points are pixels of a 2D image.

[0195] At block 2051, the gingival line is optionally leveled. The gingival line intersecting the corresponding coordinates of the gingival line of each tooth can be adjusted by fitting to a quadratic polynomial, as Figure 23 shown. The gingival line can be adjusted such that the corresponding coordinates of the gingival line are positioned along a line defined by the quadratic polynomial fit. In some embodiments, the corresponding coordinates of each tooth are the endpoints of the gingival curve or Figure 21B the highest gingival point 2130 in Figure 21B (i.e., the point on the gingival line of the tooth where the distance between the gingival line and the lip is the shortest). In some embodiments, the corresponding coordinates of each tooth are the tip points of the gingival curve or

[0196] the lowest gingival point 2140 in Figure 23 (i.e., the point on the gingival line between two teeth where the distance between the gingival line and the lip is the longest). Figure 1 block 140 of Figure 6 block 650 of Figure 10A block 1060 of Figure 10B block 1055 of

[0197] As Figure 20 shown, the parametric gingival line simulation using a combination of patient-specific parameters, simulated parameters, and ideal aesthetic parameters, and optionally the result of gingival line leveling as Figure 23 shown is that the gingival line in the rendered 2D image will match the initially captured 2D image more closely and produce a more aesthetically corrected simulated 2D image compared to the parametric gingival line defined solely by patient-specific input parameters. This improvement may be more pronounced when the lips cover a portion of one or more teeth, one or more teeth are chipped or damaged, one or more teeth are crooked, or the patient's tooth aspect ratio significantly deviates from the ideal aspect ratio. Additionally, the parametric gingival line simulation has lower computational intensity compared to generating a 3D gingival model using a 3D mesh template.

[0198] At block 2061, the simulated positions of the parametric gingival line and the 3D tooth model are rendered and inserted into the patient's 2D image at the initial or final matching position. The rendering of the 2D image is described in more detail at block 1060 of Figure 10A . The 3D rendering of the teeth can be placed in the patient's mouth defined by the lip contour. It is also possible to render the image of the parametric gingival line identified in method 2001 of Figure 20 .

[0199] In some embodiments, a neural network such as a generative adversarial network or a conditional generative adversarial network can be used to integrate the 3D model of the teeth in the final position with the patient's facial image and match the color, hue, shading, and other aspects of the 3D model with the facial photograph. The neural network is trained using the facial image. In some embodiments, the facial image can include an image of a person's face with a social smile. In some embodiments, the facial image can include a facial image of the patient before orthodontic treatment. During training, the patient's teeth and their contours can be identified. For example, each tooth can be identified by type (e.g., upper left central incisor, lower right canine). Other aspects and features of the image can also be identified during training, such as the position and color of the gingiva, the color of the teeth, the relative brightness of the inner surface of the mouth, etc.

[0200] Refer to Figure 24, after training, the neural network receives an input at block 4804 for generating a realistic rendering of a patient's tooth in a clinical final position. In some embodiments, the input can include one or more of a rendered image of a 3D model of a patient's tooth in a clinical final position or a 3D-rendered model 4808 of a patient's tooth in a clinical final position, where the clinical final position is determined, for example, based on an orthodontic treatment plan, a parametric gum line, a blurred initial image 4810 of the patient's tooth, and a color-coded image 4812 of a 3D model of the patient's tooth in a clinical final position.

[0201] A rendered image of a 3D model of a patient's tooth in a clinical final position or a 3D-rendered model 4808 of a patient's tooth in a clinical final position can be determined based on the clinical orthodontic treatment plan for moving the patient's tooth from an initial position to a final position as described above. The image or rendering 4808 can be generated based on an imaging perspective. For example, one or more of an imaging distance, a focal length of an imaging system, and a size of the patient's tooth in an initial facial image can be used to generate the image or rendering.

[0202] The blurred image 4810 of the patient's tooth can be generated using one or more blurring algorithms (e.g., a Gaussian blurring algorithm). In some embodiments, the Gaussian blur can have a large radius, e.g., a radius of at least 5, 10, 20, 40, or 50 pixels. In some embodiments, the blur is large enough such that tooth structure cannot be discerned in the blurred image.

[0203] The color-coded model 4812 of the patient's tooth can be a red-green-blue (RGB) color-coded image of the patient's tooth model, where each color channel corresponds to a different quality or feature of the model. For example, the green color channel, which can be an 8-bit color channel, can indicate the brightness of the blurred image 4810, e.g., on a scale of 0 to 255, as overlaid on the 3D model.

[0204] The red color channel can be used to distinguish each tooth and the gum from one another. In such an embodiment, the gum 4813 can have a red color channel value of 1, the upper left central incisor 4814 can have a red value of 2, the lower right canine can have a red color channel of 3, and portions of the model that are not teeth or gums have a red color channel value of 0, and so on, such that the red color channel value of each pixel identifies the tooth anatomy associated with the pixel.

[0205] The blue color channel can be used to identify the angle of the tooth and / or the gum relative to the facial plane. For example, at each pixel location, an angle perpendicular to the surface of the tooth structure is determined and a value between 0 - 255 (for an 8-bit color channel) is assigned to the pixel. For example, such information allows the neural network to model the light reflectivity from the tooth surface.

[0206] The neural network then uses the input and its training to render a realistic image 4806 of the patient's teeth in the final position. This realistic image is then integrated into the open mouth of the facial image, and an alpha channel blur is applied.

[0207] Figure 12 FIG. 1200 depicts a system for simulating estimated and / or expected results of orthodontic treatment according to some embodiments. In Figure 12 an example, system 1200 includes a computer-readable medium 1210, a dental scanning system 1220, a dental treatment planning system 1230, a dental treatment simulation system 1240, and an image capture system 1250. One or more elements of system 1200 may include elements such as those described in the computer system description as referenced Figure 20 FIG. 1. One or more elements of system 1200 may also include one or more computer-readable media that include instructions that, when executed by a processor (e.g., the processor of any of systems 1220, 1230, 1240, and 1250), cause the corresponding one or more systems to perform the processes described herein.

[0208] The dental scanning system 1220 may include a computer system configured to capture one or more scans of a patient's dentition. The dental scanning system 1220 may include a scanning engine for capturing 2D images of the patient. These images may include images of the patient's teeth, face, and jaw. These images may also include x-rays or other subsurface images of the patient. The scanning engine may also capture 3D data representative of the patient's teeth, face, gums, or other aspects of the patient.

[0209] The dental scanning system 1220 may also include a 2D imaging system, such as a still camera or video camera, an x-ray machine, or other 2D imagers. In some embodiments, the dental scanning system 1220 also includes a 3D imager, such as an intraoral scanner or an impression scanner. As described herein with reference to Figures 1 - 11 FIG. 1, the dental scanning system 1220 and associated engines and imagers may be used to capture historical scan data for determining historical average parameters of a 3D parametric dental model. For example, as described herein with reference to Figures 1 - 11 FIG. 1, the dental scanning system 1220 and associated engines and imagers may be used to capture 2D images of the patient's face and dentition for constructing a 3D parametric model of the patient's teeth.

[0210] The dental treatment simulation system 1240 may include a computer system configured to simulate one or more estimated and / or expected outcomes of a dental treatment plan. In some embodiments, the dental treatment simulation system 1240 obtains photographs and / or other 2D images of a customer / patient. The dental treatment simulation system 1240 may also be configured to determine teeth, lips, gums, and / or other margins related to the teeth in the 2D images. As described herein, the dental treatment simulation system 1240 may be configured to match tooth and / or dental arch parameters to the teeth, lips, gums, and / or other margins. The dental treatment simulation system 1240 may also render a 3D tooth model of the patient's teeth. The dental treatment simulation system 1240 may gather information related to a historical dental arch and / or an ideal dental arch representing an estimated treatment outcome. In various embodiments, the dental treatment simulation system 1240 may insert the 3D tooth model into the patient's 2D image, align it with the patient's 2D image, etc., to render a 2D simulation of the estimated outcome of orthodontic treatment. The dental treatment simulation system 1240 may include a photo parameterization engine, which may also include an edge analysis engine, an EM analysis engine, a rough tooth alignment engine, and a 3D parameterization conversion engine. The dental treatment simulation system 1240 may also include a parametric treatment prediction engine, which may also include a treatment parameterization engine, a scanned tooth normalization engine, and a treatment plan remodeling engine. The dental treatment simulation system 1240 and its associated engines may perform the processes described above with reference to Figures 5 - 9 the process described.

[0211] The dental treatment planning system 1230 may include a computer system configured to implement a treatment plan. The dental treatment planning system 1230 may include a rendering engine and an interface for visualizing or otherwise displaying the simulation results of a dental treatment plan. For example, the rendering engine may render visualizations of the 3D models described herein, such as, Figure 1 at block 140 of Figure 6 at blocks 640, 650, and 660 of Figure 8 process 800 described with reference to Figure 9 at blocks 910, 920, and 930 of Figure 11 block 1150 of Figure 20 block 1150 and Figure 24As described above. The dental treatment planning system 1230 can also determine an orthodontic treatment plan for moving a patient's teeth from an initial position to a final position, for example, based in part on 2D images of the patient's teeth. The dental treatment planning system 1230 can be operable to provide image viewing and manipulation such that the rendered images can be scrolled, pivoted, zoomed, and interactive. The dental treatment planning system 1230 can include graphics rendering hardware, one or more displays, and one or more input devices. Some or all of the dental treatment planning system 1230 can be implemented on a personal computing device such as a desktop computing device or on a handheld device such as a mobile phone. In some embodiments, at least a portion of the dental treatment planning system 1230 can be implemented on a scanning system such as the dental scanning system 1220. The image capture system 1250 can include devices configured to acquire images, including images of patients. The image capture system can include any type of mobile device (iOS devices, iPhones, iPads, iPods, etc., Android devices, portable devices, tablets), PCs, cameras (DSLR cameras, film cameras, video cameras, still cameras, etc.). In some implementations, the image capture system 1250 includes a set of stored images, such as images stored on a storage device, a network location, a social media website, etc.

[0212] Figure 13 Shows an example of one or more elements of a dental treatment simulation system 1240 according to some embodiments. In Figure 13 the example, the dental treatment simulation system 1240 includes a photo collection engine 1310, a photo data warehouse 1360, a photo parameterization engine 1340, a case management engine 1350, a reference case data warehouse 1370, and a treatment rendering engine 1330.

[0213] The photo collection engine 1310 can implement one or more automated agents configured to retrieve photos of a selected patient from the photo data warehouse 1360, the image capture system 1250, and / or scans from the dental scanning system 1220. The photo collection engine can then provide one or more photos to the photo parameterization engine 1340. The photo collection engine can then provide one or more photos to the photo parameterization engine 1340 and / or other modules of the system.

[0214] The photo data warehouse 1360 may include a data warehouse configured to store photos of patients (e.g., their facial photos). In some embodiments, these photos are 2D images that include one or more images of the patient's mouth as well as the face, head, neck, shoulders, torso, or the patient as a whole. The 2D images of the patient may include images of the patient with the mouth in one or more positions. For example, the patient's mouth may be in a smiling position (such as a social smiling position), a resting position with relaxed muscles and slightly open lips, or a retruded anterior open bite or anterior closed bite position. In some embodiments, images of the patient are taken using an image capture system. The images may be captured using a lens with a predetermined focal length at a certain distance from the patient. The images may also be captured remotely and then received for processing. The images may be a series of images from a video taken from one or more perspectives. For example, the images may include one or more of a frontal face image and a side image, and the side image includes one or more three-quarter side images and full side images.

[0215] The photo parameterization engine 1340 may implement one or more automated agents configured to build a parametric 3D model based on 2D images of the patient from the photo data warehouse 1360 and parametric models and average data from the parametric treatment prediction engine. The edge analysis engine 1342 may implement one or more automated agents configured to determine the edges of teeth, lips, and gums within the patient photo, e.g., as described in reference Figure 6 Particularly, an initial step of identifying the patient's lips (e.g., the inner edge of the lips defining an open mouth), teeth, and gum contours may be performed by a machine learning algorithm such as a convolutional neural network.

[0216] Then, the initial contours may be extracted from the image. Then, the pixels representing the contours may undergo binarization. The binarized tooth contours are thinned, reducing the thickness of the contours to, for example, a single pixel width, thereby forming thinned contours.

[0217] Using the thinned or otherwise obtained contours, the parametric 3D tooth and dental arch models are matched to each tooth of the patient depicted in the 2D image of the patient. This matching is based on the contours determined above.

[0218] The rough alignment engine 1344 may implement one or more automated agents configured to receive the 2D image and / or associated edges and perform a rough alignment between the identified edges and the average tooth model, e.g., as described in reference Figure 7AAs described, the corresponding centers of the teeth in the 2D image and the silhouette of the parametric teeth can be aligned before sending the 2D image and / or associated edges along with the silhouette and its associated tooth parameters to the EM engine 1346.

[0219] The Expectation Maximization (EM) engine 1346 can implement one or more automated agents configured to perform an expectation maximization analysis between the edges of the 2D image and the parameters of the 3D parametric model, e.g., with reference to Figure 7A , and specifically as described in boxes 720, 730, and 740.

[0220] Once the EM analysis engine 1346 has completed the matching of the parametric 3D model to the patient's 2D image, the parametric results are sent to the 3D parametric conversion engine 1348, which can convert the principal component analysis output by the EM analysis engine into case-specific parameters that define a 3D model of the patient's teeth according to the parametric model for use by the parametric treatment prediction engine 1350, as described above.

[0221] The gingival parameterization engine 1345 can implement one or more automated agents configured to build a gingival line model based on patient-specific input parameters and optionally simulated input parameters as described with respect to Figure 20 and specifically as described in boxes 2031 and 2041 with respect to Figure 20 . The gingival line model is generated using inputs from an initial 2D image of the patient's teeth, a 3D parametric model of the patient's teeth, and optionally ideal tooth shape parameters. The gingival line model can then be rendered onto the 2D image using the treatment rendering engine as described in box 1330.

[0222] The gingival line leveling engine 1347 can implement one or more automated agents configured to level the gingival line based on the gingival line parameters generated at box 1345, the gingival parameterization engine. Optionally, the gingival line leveling is performed as described with respect to Figure 20 , and specifically as described in box 2051. The leveled gingival line can then be rendered onto the 2D image using the treatment rendering engine as described in box 1330.

[0223] The case management engine 1350 may include a case parameterization engine 1352, a scanned tooth normalization engine 1354, and a treatment plan simulation engine 1356. The case management engine 1350 may implement one or more automated agents that are configured to define a parametric model for representing a patient's teeth and dental arches, determine average data that parametrically represents the average positions of the patient's teeth and dental arches based on a treatment plan retrieved from the treatment plan data warehouse 1370, and simulate the treatment of a particular patient's teeth based on the parametric model, the average tooth positions, and the particular teeth of the patient.

[0224] The case parameterization engine 1352 may implement one or more automated agents that are configured to define a parametric 3D model for use in the process of modeling teeth. For example, the definition of the parametric 3D model may be as described above with reference to equation (2). The treatment parameterization engine 1352 not only defines equations, but may also define a schema for the parameters. For example, T τ representing tooth position may be defined as a 4*4 matrix including tooth position and 3-axis rotation. Similarly, representing tooth shape may be defined as a 2500*3 matrix, where each vertex of the sphere is mapped to a position on the surface of the tooth model, and the position of each vertex is a special position of x, y, and z. For lower or higher resolution models, fewer or more vertices may be used. Further discussion of the parameters defined by the treatment parameterization engine 1352 is described elsewhere in this document (e.g., reference Figures 2 - 5 ).

[0225] The scanned tooth normalization engine 1354 may implement one or more automated agents that are configured to parameterize the scanned teeth of a set of treatment plans collected from the treatment plan data warehouse 1370, and then determine an average set of general parameters for the parametric model. The scanned tooth normalization engine 1354 may perform the process 350 described in reference Figure 3B . For example, the scanned tooth normalization engine 1354 may align the dental arches within historical cases retrieved from the data warehouse 1370. Align each dental arch in the data warehouse at the following three positions: between the central incisors, and at each distal end on the left and right sides of each dental arch. Then, determine the position and orientation of each tooth based on the position and orientation of each tooth relative to the alignment points.

[0226] The scanned tooth normalization engine 1354 may implement one or more automated agents that are configured to determine the distribution of parameters within a case. For example, each historical dental arch may be rescaled to determine Φ, and then averaged across each dental arch to determine Then determine the local deformation of each tooth and compare it with to determine T τ . Then, after aligning each tooth, determine β τ .

[0227] From this data, determine the relative shape, position, and rotation of each tooth in order to construct a distribution of each case-specific parameter. For example, determine the coefficients of the principal components of the surface model of each corresponding tooth in all retrieved models distribution.

[0228] The position and orientation of each tooth can be averaged to determine the average position of each tooth, and the orientation of each tooth can be averaged to determine the average orientation of each tooth. The average position and average orientation are used to determine

[0229] In some embodiments, a template data warehouse of a parameterized template model can be generated using a set of scanned teeth of treatment plans collected from a treatment plan data warehouse. The scanned tooth normalization engine 1354 can align the dental arches in the historical cases retrieved from the reference case data warehouse 1370. Align each dental arch in the reference case data warehouse at the following three positions: between the central incisors, and at each distal end on the left and right sides of each dental arch. Then, determine the position and orientation of each tooth based on the position and orientation of each tooth relative to the alignment points. The parameters of the parameterized template model in the data warehouse are searchable. The data warehouse can be used to match a patient 3D model (e.g., generated from a 2D model or from a 3D scan) to the closest template in the data warehouse based on tooth shape and dental arch shape to identify a matching template model. The closest matching template model can be used to determine the final tooth position and orientation of the patient's teeth.

[0230] The treatment plan simulation engine 1356 can implement one or more automated agents configured to use the parameter model defined by the treatment parameterization engine 1352, the general parameters from the scanned tooth normalization engine, and the case-specific parameters from the photo parameterization engine 1340 to simulate the estimated and / or expected results of a dental treatment plan for a specific patient's teeth, e.g., as described with reference to FIGS. 10 and Figure 11 described.

[0231] The treatment plan simulation engine 1356 retrieves or otherwise determines the average shape In some embodiments, the average shape can be retrieved from the scanned tooth normalization engine 1354. In some embodiments, the average shape can be loaded into memory or retrieved from memory. The average shape can also be initialized as a set of matrices, each matrix corresponding to a tooth.

[0232] The treatment planning simulation engine 1356 determines the average shape of the teeth Performs principal component analysis shape adjustment. For each tooth in the model, the case-specific coefficients determined from the photo parameterization engine 1340 for the principal components Are applied to the principal components After completing the shape adjustment, in some embodiments, for example, the adjusted shape can be rendered on a screen for viewing by the patient or dental professional. In some embodiments, the adjusted shape can be stored in memory. The adjusted shape can also be stored as a set of matrices, each matrix corresponding to a tooth

[0233] The treatment planning simulation engine 1356 determines the average tooth pose. In some embodiments, as discussed above, the average tooth pose can be based on the average tooth pose of a set of scanned dental arches. In some embodiments, each adjusted tooth is placed in its corresponding average position and orientation determined by the average dental arch. In some embodiments, for example, the average tooth pose can be rendered on a screen for viewing by the patient or dental professional. In some embodiments, the average tooth pose can be loaded into memory or retrieved from memory. The average tooth pose can also be initialized as a set of matrices, each matrix corresponding to a tooth in a tooth pose

[0234] In some embodiments, the treatment planning remodeling engine 1356 uses the tooth position and orientation coordinates matched from the ideal parameterization data warehouse and applies the shape and texture of the patient's 3D model generated from the 2D image to the tooth position and orientation to generate a dental arch

[0235] The treatment planning simulation engine 1356 scales the dental arch so that the generated dental arch is based on the patient's tooth and dental arch dimensions. In some embodiments, as discussed above, the dental arch scaling factor Φ is based on the specific teeth and dental arch of a particular patient. In some embodiments, as discussed herein, the dental arch is adjusted so that the scaled dental arch matches the dimensions of the patient's dental arch, the dimensions of which are determined, for example, by the dimensions of the teeth and dental arch in the scanned patient dental arch or otherwise. In some embodiments, for example, the scaled dental arch can be rendered on a screen for viewing by the patient or dental professional. In some embodiments, the scaled dental arch can be stored in memory. The scaled dental arch can also be stored as a set of matrices, each matrix corresponding to a tooth in a tooth pose. The scaled dental arch can be sent to the treatment rendering engine 1330

[0236] The treatment rendering engine 1330 renders the teeth and 2D images of the patient in the final position as determined, for example, by the parametric treatment prediction engine 1350 and, in particular, by the treatment plan simulation engine 1356 located within the parametric treatment prediction engine 1350. The treatment rendering engine 1330 may perform some of the processes described with reference to Figure 9 , FIG. 10, and Figure 11 described processes. In particular, with reference to Figure 9 , FIG. 10, Figure 11 and the rendering processes described elsewhere herein.

[0237] Figure 14An exemplary tooth repositioning instrument or appliance 1500 is shown that can be worn by a patient to effect progressive repositioning of individual teeth 1502 in the jaws. The instrument can include a housing (e.g., a continuous polymeric housing or a segmented housing) having tooth receiving cavities that receive the teeth and resiliently reposition the teeth. The instrument or one or more portions thereof can be indirectly fabricated using a physical model of the teeth. For example, the instrument (e.g., a polymeric instrument) can be formed using a physical model of the teeth and appropriate layers of polymeric material sheets. The physical model of the teeth (e.g., a physical mold) can be formed by various techniques including 3D printing. The instrument can be formed by thermoforming the instrument over the physical model. In some embodiments, the physical instrument is fabricated directly from a digital model of the instrument, e.g., using additive manufacturing techniques. In some embodiments, the physical instrument can be fabricated by various direct forming techniques such as 3D printing. The instrument can be mounted on all the teeth present in the upper or lower jaw or not on all the teeth. The instrument can be specifically designed to conform to the patient's teeth (e.g., the topography of the tooth receiving cavities matches the topography of the patient's teeth) and can be fabricated based on positive or negative models of the patient's teeth generated by impression, scanning, etc. Alternatively, the instrument can be a general purpose instrument that is configured to receive the teeth but not necessarily shaped to match the topography of the patient's teeth. In some cases, only certain teeth received by the instrument are repositioned, while other teeth can provide a base or anchorage area for holding the instrument in place when the instrument applies a force to one or more teeth that are the target of repositioning. In some cases, during treatment, some or most or even all of the teeth are repositioned at some point. The teeth being moved can also serve as a base or anchorage for holding the instrument in place when the instrument is worn by the patient. In some embodiments, wires or other devices for holding the instrument in place on the teeth are no longer provided. However, in some cases, it may be desirable or necessary to provide separate attachments or other anchoring elements 1504 on the teeth 1502 using corresponding sockets or apertures 1506 in the instrument 1500 such that the instrument can apply a selected force to the teeth. Many patents and patent applications assigned to Align Technology, Inc. (e.g., including U.S. Patent No. 6,450,807 and U.S. Patent No. 5,975,893) and the company website accessible on the World Wide Web (see, e.g., the url "invisalign.com") describe including Exemplary instruments, including instruments used in the system. Examples of tooth-mounted attachments suitable for use with orthodontic instruments are also described in patents and patent applications assigned to Align Technology, Inc., such as, for example, U.S. Patent No. 6,309,215 and U.S. Patent No. 6,830,450.

[0238] Optionally, in cases involving more complex movements or treatment plans, the use of auxiliary components (e.g., features, accessories, structures, devices, components, etc.) in combination with an orthodontic instrument may be advantageous. Examples of such accessories include, but are not limited to, rubber bands, wires, springs, rods, arch expanders, palatal expanders, bicuspid plates, bite blocks, bite ramps, mandibular advancement splints, bite plates, pontics, hooks, brackets, headgear tubes, springs, bumper tubes, palatal bars, frameworks, pin and tube devices, buccal shields, buccinator bows, wire shields, flange and pads, lip pads or lip bumpers, protrusions, divots, etc. In some embodiments, the instruments, systems, and methods described herein include improved orthodontic instruments having features integrally formed and shaped to couple with or replace these auxiliary components.

[0239] Figure 15Illustrated is a tooth repositioning system 1510 that includes a plurality of instruments 1512, 1514, 1516. Any of the instruments described herein can be designed and / or provided as part of a set of a plurality of instruments for use in a tooth repositioning system. Each instrument can be configured such that the tooth receiving cavity has a geometry corresponding to an intermediate tooth alignment or a final tooth alignment intended for the instrument. By placing a series of progressively positioned adjustment instruments on a patient's teeth, the patient's teeth can be gradually repositioned from an initial tooth alignment toward a target tooth alignment. For example, the tooth repositioning system 1510 can include a first instrument 1512 corresponding to the initial tooth alignment, one or more intermediate instruments 1514 corresponding to one or more intermediate alignments, and a final instrument 1516 corresponding to the target alignment. The target tooth alignment can be the planned final tooth alignment selected for the patient's teeth at the end of all planned orthodontic treatment. Alternatively, the target alignment can be one of some intermediate alignments of the patient's teeth during the course of orthodontic treatment, which can include a variety of different treatment regimens, including but not limited to the following examples: surgical intervention is recommended, interproximal reduction (IPR) is appropriate, scheduling an inspection, optimal anchor placement, palatal expansion is required, dental restorations are involved (e.g., inlays, onlays, crowns, bridges, implants, veneers, etc.), and so on. For example, it should be understood that the target tooth alignment can be any planned resultant alignment for the patient's teeth after one or more progressive repositioning phases. Similarly, the initial tooth alignment can be any initial alignment of the patient's teeth, followed by one or more progressive repositioning phases.

[0240] Figure 16A method 1550 of orthodontic treatment using multiple instruments in accordance with many embodiments is shown. Method 1550 may be practiced using any of the instruments or groups of instruments described herein. In block 1560, a first orthodontic instrument is applied to a patient's teeth to reposition the teeth from a first tooth alignment to a second tooth alignment. In block 1570, a second orthodontic instrument is applied to the patient's teeth to reposition the teeth from the second tooth alignment to a third tooth alignment. Method 1550 may be repeated as needed using any suitable number of ordered instruments and any suitable combination of ordered instruments to progressively reposition the patient's teeth from an initial alignment to a target alignment. The instruments may all be generated at the same stage, or in groups or batches (e.g., at the beginning stage of treatment, at the intermediate stage of treatment, etc.), or one instrument may be manufactured at a time, and the patient may wear each instrument until the pressure of each instrument on the teeth can no longer be felt, or until the maximum amount of tooth movement exhibited for that given stage has been achieved. The multiple instruments may be designed and even manufactured before the patient wears any orthodontic appliance in a plurality of different instruments (e.g., a set). After the instrument has been worn for an appropriate period of time, the patient may replace the current instrument with the next instrument in the series until no instruments remain. These instruments are generally not adhered to the teeth, and the patient may install and replace the instrument at any time during the procedure (e.g., patient-removable instruments). The final instrument or some of the instruments in the series may have one or more geometries selected for overcorrection of the tooth alignment. For example, one or more instruments may have a geometry that will cause the movement of each tooth beyond the tooth alignment selected as "final" (if fully achieved). Such overcorrection may be desirable to counteract potential relapse after the repositioning method ends (e.g., to allow each tooth to move back towards its pre-corrected position). Overcorrection is also beneficial for accelerating the correction speed (e.g., an instrument whose geometry is positioned beyond the desired intermediate or final position can shift each tooth towards that position at a faster speed). In this case, the use of the instrument may be stopped before the teeth reach the position defined by the instrument. Additionally, overcorrection may be intentionally applied to compensate for any inaccuracies or limitations of the instrument.

[0241] The various embodiments of the orthodontic instruments presented herein may be manufactured in a variety of ways. In some embodiments, the orthodontic instruments (or portions thereof) herein may be produced using direct manufacturing techniques such as additive manufacturing techniques (also referred to herein as "3D printing") or subtractive manufacturing techniques (e.g., milling). In some embodiments, direct manufacturing involves forming an object (e.g., an orthodontic instrument or a portion thereof) without using a physical template (e.g., a mold, a mask, etc.) to define the geometry of the object.

[0242] In some embodiments, the orthodontic appliances herein can be manufactured using a combination of direct manufacturing techniques and indirect manufacturing techniques such that different parts of the appliance can be manufactured using different manufacturing techniques and assembled to form the final appliance. For example, the appliance housing can be formed by indirect manufacturing (e.g., thermoforming), and one or more structures or components described herein (e.g., auxiliary components, power arms, etc.) can be added to the housing by direct manufacturing (e.g., printed onto the housing).

[0243] The configuration of the orthodontic appliances herein can be determined based on a patient's treatment plan, e.g., the treatment plan involves the sequential implementation of multiple appliances for progressively repositioning teeth. Computer-based treatment planning and / or appliance manufacturing methods can be used to facilitate the design and manufacture of the appliances. For example, with the aid of computer-controlled manufacturing equipment (e.g., computer numerical control (CNC) milling, computer-controlled additive manufacturing such as 3D printing, etc.), one or more appliance components described herein can be digitally designed and manufactured. The computer-based methods proposed herein can improve the accuracy, flexibility, and convenience of appliance manufacturing.

[0244] In some embodiments, a computer-based 3D planning / design tool (e.g., Treat TM software from Align Technology, Inc.) can be used to design and manufacture the orthodontic appliances described herein.

[0245] Figure 17 Method 1800 for designing an orthodontic appliance to be manufactured according to an embodiment is shown. Method 1800 can be applied to any embodiment of the orthodontic appliances described herein. Some or all of the operations of Method 200 can be performed by any suitable data processing system or device (e.g., one or more processors configured with suitable instructions).

[0246] In block 1810, a movement path for moving one or more teeth from an initial alignment to a target alignment is determined. The initial alignment can be determined from a mold or scan of the patient's teeth or oral tissues, e.g., using wax bite registration, direct contact scanning, x-ray imaging, tomography, ultrasound imaging, and other techniques for obtaining information about the position and structure of teeth, jaws, gums, and other orthodontically relevant tissues. A digital data set can be obtained from the acquired data, which represents the initial (e.g., pre-treatment) alignment of the patient's teeth and other tissues. Optionally, the initial digital data set is processed to segment tissue constituents from each other. For example, a data structure representing each tooth crown can be generated. Advantageously, a digital model of the entire tooth can be generated, including measured or inferred hidden surfaces and root structures, as well as surrounding bone and soft tissue.

[0247] The target alignment of teeth (e.g., the desired and anticipated end result of orthodontic treatment) can be received from a clinician in the form of a prescription, can be calculated based on fundamental orthodontic principles, and / or can be inferred computationally from a clinical prescription. With a detailed description of the desired final positions of the teeth and a digital representation of the teeth themselves, the final position and surface geometry of each tooth can be specified to form a complete model of the tooth alignment at the end of the desired treatment.

[0248] Each tooth has both an initial position and a target position, and a movement path can be defined for the movement of each tooth. In some embodiments, these movement paths are configured to move the teeth in the fastest way with a minimum amount of round - tripping to move the teeth from their initial positions to their desired target positions. Optionally, the tooth paths can be segmented, and these segments can be calculated such that the movement of each tooth within a segment remains within the threshold limits of linear translation and rotational translation. In this way, the endpoints of each path segment can constitute a clinically viable repositioning, and the set of segment endpoints can constitute a clinically viable sequence of tooth positions such that the movement from one point to the next in the sequence does not cause tooth collisions.

[0249] In block 1820, a force system is determined for moving one or more teeth along the movement path. The force system can include one or more forces and / or one or more torques. Different force systems cause different types of tooth movement, such as tipping, translation, rotation, extrusion, intrusion, root movement, etc. Biomechanical principles, modeling techniques, force calculation / measurement techniques, etc. (including knowledge and protocols commonly used in orthodontics) can be used to determine the appropriate force system to be applied to the teeth to achieve tooth movement. Resources that can be considered when determining the force system to be applied include literature, force systems determined through experiments or virtual modeling, computer - based modeling, clinical experience, minimizing unwanted forces, etc.

[0250] A force system can be determined in various ways. For example, in some embodiments, the force system is determined on a patient-by-patient basis (e.g., using patient-specific data). Alternatively or in combination, the force system can be determined based on a generalized model of tooth movement (e.g., based on experiments, modeling, clinical data, etc.), and thus not necessarily using patient-specific data. In some embodiments, determining the force system includes calculating the specific force values to be applied to one or more teeth to produce a specific movement. Alternatively, the determination of the force system can be performed at a high level without calculating the specific force values of the teeth. For example, block 1820 can include determining a specific type of force to be applied (e.g., extrusion force, intrusion force, translational force, rotational force, tipping force, torque force, etc.) without calculating the specific magnitude and / or direction of the force.

[0251] In block 1830, the instrument geometry and / or material composition of the orthodontic instrument configured to produce the force system is determined. The instrument can be any embodiment of the instruments discussed herein, such as an instrument having variable local properties, integrally formed components, and / or power arms.

[0252] For example, in some embodiments, the instrument includes a heterogeneous thickness, a heterogeneous hardness, or a heterogeneous material composition. In some embodiments, the instrument includes two or more of a heterogeneous thickness, a heterogeneous hardness, or a heterogeneous material composition. In some embodiments, the instrument includes a heterogeneous thickness, a heterogeneous hardness, and a heterogeneous material composition. The heterogeneous thickness, hardness, and / or material composition can be configured to produce a force system for moving teeth, e.g., by preferentially applying force at certain locations on the teeth. For example, an instrument having a heterogeneous thickness can include a thicker portion that applies a greater force on the tooth than a thinner portion. As another example, an instrument having a heterogeneous hardness can include a harder portion that applies a greater force on the tooth than a more resilient portion. As described herein, the change in hardness can be achieved by changing the instrument thickness, material composition, and / or degree of photopolymerization.

[0253] In some embodiments, determining the instrument geometry and / or material composition includes determining the geometry and / or material composition of one or more integrally formed components to be directly manufactured with the instrument housing. The integrally formed component can be any embodiment described herein. The geometry and / or material composition of the (one or more) integrally formed components can be selected to facilitate the application of the force system to the patient's teeth. The material composition of the integrally formed component can be the same as or different from the material composition of the housing.

[0254] In some embodiments, determining the instrument geometry includes determining the geometry of a variable gable bend.

[0255] The block 1830 may include analyzing a desired force system to determine the instrument geometry and material composition that will produce the force system. In some embodiments, the analysis includes determining the instrument properties (e.g., stiffness) at one or more locations that will produce the desired force at one or more locations. The analysis then includes determining the instrument geometry and material composition at one or more locations to achieve the specified properties. The determination of the instrument geometry and material composition may be performed using a treatment or force application simulation environment. The simulation environment may include, for example, a computer modeling system, a biomechanical system or device, etc. Optionally, a digital model of the instrument and / or the teeth may be made, such as a finite element model. Computer program application software that can be purchased from various vendors may be used to create the finite element model. To create a solid geometry model, a computer-aided engineering (CAE) or computer-aided design (CAD) program may be used, such as the software product that can be purchased from Autodesk, Inc. in San Rafael, Canada. To create finite element models and analyze them, program products from some vendors may be used, which include the finite element analysis software package from ANSYS, Inc. in Canonsburg, Pennsylvania, and the SIMULIA (Abaqus) software product from Dassault Systèmes in Waltham, Massachusetts.

[0256] Optionally, one or more instrument geometries and material compositions may be selected for testing or force modeling. As described above, the desired tooth movement and the force system required or desired to cause the desired tooth movement may be identified. By using the simulation environment, the geometry and composition of the candidate instrument may be analyzed or modeled to determine the actual force system produced by using the candidate instrument. Optionally, one or more modifications may be made to the candidate instrument, and the force modeling may be further analyzed (as described), for example, in order to iteratively determine the instrument design that produces the required force system.

[0257] Optionally, block 1830 may also include determining the geometry of one or more auxiliary components to be used in combination with the orthodontic appliance to apply a force system to one or more teeth. These auxiliary devices may include one or more of the following: tooth-mounted attachments, elastics, wires, springs, bite blocks, arch expanders, wire-bracket appliances, shell appliances, headgear, or any other orthodontic device or system that may be used in combination with the orthodontic appliances herein. The use of these auxiliary components may be advantageous in cases where the appliance alone is difficult to generate a force system. Additionally, auxiliary components may be added to the orthodontic appliance to provide other desired functions in addition to generating a force system, such as a mandibular advancement splint for treating sleep apnea, a pontic for improving aesthetics, and the like. In some embodiments, the auxiliary components are manufactured and provided separately from the orthodontic appliance. Additionally, the geometry of the orthodontic appliance may be modified to include one or more auxiliary components that are formed as an integral part.

[0258] In block 1840, instructions are generated for manufacturing an orthodontic appliance having the appliance geometry and material composition. These instructions may be configured to control a manufacturing system or device to produce an orthodontic appliance having the specified appliance geometry and material composition. In some embodiments, these instructions may be configured to manufacture the orthodontic appliance using direct manufacturing (e.g., stereolithography, selective laser sintering, fused deposition modeling, 3D printing, continuous direct manufacturing, multi-material direct manufacturing, etc.). Optionally, as discussed above and herein, these instructions may be configured to cause the manufacturing machine to directly manufacture an orthodontic appliance having a tooth receiving cavity with a variable roof-shaped curvature. In alternative embodiments, the instructions may be configured to indirectly manufacture the appliance (e.g., by thermoforming).

[0259] While the above blocks illustrate a method 1800 of designing an orthodontic appliance according to some embodiments, those of ordinary skill in the art will recognize some variations in accordance with the teachings described herein. Some blocks may include sub-blocks. Some blocks may be repeated as frequently as needed. One or more blocks of method 200 may be performed using any suitable manufacturing system or device, such as the embodiments described herein. Some blocks are optional, and the order of the blocks may be changed as needed. For example, in some embodiments, block 1820 is optional such that block 1830 includes determining the appliance geometry and / or material composition directly based on the tooth movement path rather than based on the force system.

[0260] Figure 18 A method 1900 for digitally planning orthodontic treatment and / or the design or manufacture of an appliance according to an embodiment is shown. Method 1900 can be applied to any treatment procedure described herein and can be performed by any suitable data processing system.

[0261] In block 1910, a digital representation of a patient's teeth is received. The digital representation can include surface topography data of the patient's oral cavity (including teeth, gingival tissue, etc.). The surface topography data can be generated by directly scanning the oral cavity, a physical model (positive or negative) of the oral cavity, or an impression of the oral cavity using a suitable scanning device (e.g., a handheld scanner, a desktop scanner, etc.).

[0262] In block 1920, one or more treatment phases are generated based on the digital representation of the teeth. These treatment phases can be successive repositioning phases of an orthodontic treatment procedure that are designed to move one or more of the patient's teeth from an initial tooth alignment to a target alignment. For example, these treatment phases can be generated by determining the initial tooth alignment indicated by the digital representation, determining the target tooth alignment, and determining the movement paths required for one or more of the teeth in the initial alignment to achieve the target tooth alignment. The movement paths can be optimized based on minimizing the total movement distance, preventing collisions between teeth, avoiding difficult-to-achieve tooth movements, or any other suitable criteria.

[0263] In block 1930, at least one orthodontic appliance is manufactured based on the generated treatment phases. For example, a set of appliances can be manufactured, each appliance being shaped according to the tooth alignment specified by a particular treatment phase, such that the appliances can be worn by the patient in sequence to gradually reposition the teeth from the initial alignment to the target alignment. The set of appliances can include one or more of the orthodontic appliances described herein. The manufacture of the appliance can include creating a digital model of the appliance to be used as an input to a computer-controlled manufacturing system. As needed, the appliance can be formed using direct manufacturing methods, indirect manufacturing methods, or a combination of both.

[0264] In some cases, staging the various alignments or treatment phases may not be required for the design and / or manufacture of the appliance. As shown by the dashed line in Figure 18 , the design and / or manufacture of the orthodontic appliance (and possibly a particular orthodontic treatment) can include using a representation of the patient's teeth (e.g., receiving the digital representation 1910 of the patient's teeth), followed by designing and / or manufacturing the orthodontic appliance based on the representation of the patient's teeth in the alignment represented by the received representation.

[0265] Optionally, some or all of the blocks of method 1900 are performed locally at the site where the patient is being treated and during a single patient visit, referred to herein as "chairside fabrication". For example, chairside fabrication can include scanning a patient's teeth, automatically generating a treatment plan with various treatment phases, and immediately fabricating one or more orthodontic appliances using a chairside direct fabricator to treat the patient, all during a single appointment, in the office of a professional treatment provider. In embodiments where a series of appliances are used to treat a patient, the first appliance can be fabricated chairside for immediate delivery to the patient, while the remaining appliances are fabricated separately (e.g., off-site at a laboratory or central fabrication facility) and delivered at a later time (e.g., during a subsequent appointment, mailed to the patient). Alternatively, the methods herein can be adapted to generate and immediately deliver an entire series of appliances on-site during a single visit. Thus, by immediately beginning treatment of a patient in the physician's office rather than having to wait for appliances to be fabricated and delivered at a later date, chairside fabrication can improve the convenience and speed of the treatment procedure. In addition, chairside fabrication can provide improved flexibility and efficiency of orthodontic treatment. For example, in some embodiments, the patient is re-scanned at each consultation to determine the actual position of the teeth and the treatment plan is updated accordingly. Subsequently, new appliances can be fabricated and delivered immediately chairside to accommodate any changes or deviations in the treatment plan.

[0266] Figure 19 is a simplified block diagram of a data processing system 2000 that can be used to perform the methods and processes described herein. Data processing system 2000 generally includes at least one processor 2002 that communicates with one or more peripheral devices via a bus subsystem 2004. These peripheral devices typically include a storage subsystem 2006 (memory subsystem 2008 and file storage subsystem 2014), a set of user interface input and output devices 2018, and an interface to an external network 2016. The interface is schematically shown as the "network interface" block 2016 and is coupled via a communication network interface 2024 to a corresponding interface device in other data processing systems. Data processing system 2000 can include, for example, one or more computers, such as personal computers, workstations, mainframes, laptop computers, etc.

[0267] The user interface input devices 2018 are not limited to any particular device and can generally include, for example, a keyboard, a pointing device, a mouse, a scanner, an interactive display, a touchpad, a joystick, etc. Similarly, a variety of user interface output devices can be used in the systems of the present invention and can include, for example, one or more of a printer, a display (e.g., visual, non-visual) system / subsystem, a controller, a projection device, an audio output, etc.

[0268] The storage subsystem 2006 maintains programming for basic requirements, including computer-readable media having instructions (e.g., operation instructions, etc.) and data structures. The program modules discussed herein are typically stored in the storage subsystem 2006. The storage subsystem 2006 generally includes a memory subsystem 2008 and a file storage subsystem 2014. The memory subsystem 2008 generally includes multiple memories (e.g., RAM 2010, ROM 2012, etc.), which include computer-readable memories for storing fixed instructions, instructions and data during program execution, basic input / output systems, etc. The file storage subsystem 2014 permanently (non-volatilely) stores program and data files and may include one or more removable or fixed drives or media, hard disks, floppy disks, CD-ROMs, DVDs, optical disk drives, etc. One or more of the storage systems, drives, etc. may be located at a remote location, e.g., coupled via a server on a network or via the Internet / World Wide Web. In context, the term "bus subsystem" is generally used to include any mechanism for enabling the various components and subsystems to communicate with each other as expected and may include various suitable components / systems that would be considered or recognized as suitable for use therein. It should be recognized that the various components of the system may but need not be in the same physical location, but may be connected via various local area or wide area network media, transmission systems, etc.

[0269] The scanner 2020 includes any device for obtaining a digital representation (e.g., an image, surface topography data, etc.) of a patient's teeth (e.g., by scanning a physical model of the teeth such as a cast 2027, by scanning an impression taken from the teeth, or by directly scanning the oral cavity), the digital representation being obtainable from the patient or a professional caregiver (e.g., an orthodontist), and the scanner 2020 further includes a device for providing the digital representation to the data processing system 2000 for further processing. The scanner 2020 may be located at a remote location relative to the other components of the system and may transmit image data and / or information to the data processing system 2000, e.g., via a network interface 2024. The fabricator 2022 fabricates the device 2023 based on a treatment plan that includes dataset information received from the data processing system 2000. For example, the fabricator 2022 may be located at a remote location and receive the dataset information from the data processing system 2000 via the network interface 2024. The camera 2025 may include any image capture device configured to capture still images or movies. The camera 2025 may capture various perspectives of the patient's dentition. In some embodiments, the camera 2025 may capture images at different focal lengths at different distances from the patient.

[0270] The data processing aspects of the methods described herein can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or suitable combinations thereof. A data processing apparatus can be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Data processing blocks can be performed by a programmable processor executing program instructions to perform functions by operating on input data and generating output. The data processing aspects can be implemented on one or more computer programs that can be executed on a programmable system including one or more programmable processors operatively coupled to a data storage system. Generally, a processor will receive instructions and data from a read-only memory and / or a random-access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as: semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices); magnetic disks (such as internal hard disks and removable disks); magneto-optical disks; and CD-ROM disks.

[0271] Although the detailed description contains many specific details, these specific details should not be construed as limiting the scope of the disclosure, but merely as illustrating different examples and aspects of the disclosure. It should be understood that the scope of the disclosure includes other embodiments not discussed in detail above. Without departing from the spirit and scope of the invention described herein, various other modifications, variations, and alterations that are obvious to those skilled in the art can be made in the arrangement, operation, and details of the methods, systems, and apparatuses of the disclosure provided herein.

[0272] As used herein, the terms “dental instrument” and “tooth-receiving instrument” are considered synonymous. As used herein, “dental positioning instrument” or “orthodontic instrument” can be considered synonymous and can include any dental instrument configured to change the position of a patient's teeth according to a plan such as an orthodontic treatment plan. As used herein, “patient” can include any person, including those seeking dental / orthodontic treatment, those undergoing dental / orthodontic treatment, and those who have previously received dental / orthodontic treatment. “Patient” can include customers or potential customers of orthodontic treatment, such as those using the visualization tools herein to inform their decision to fully accept orthodontic treatment or to select a specific orthodontic treatment plan. As used herein, “dental positioning instrument” or “orthodontic instrument” can include a set of dental instruments configured to gradually change the position of a patient's teeth over time. As described herein, a dental positioning instrument and / or an orthodontic instrument can include a polymeric instrument configured to move a patient's teeth according to an orthodontic treatment plan.

[0273] As used herein, the term "and / or" may be used as a functional word to indicate that two words or phrases are taken together or separately. For example, the phrase "A and / or B" includes A alone, B alone, and A and B together. Depending on the context, the term "or" does not exclude one of multiple words / phrases. For example, the phrase "A or B" does not exclude A and B together.

[0274] The terms "torque" and "moment" as used herein are considered synonymous.

[0275] The "moment" as used herein may include a force acting on an object, such as a tooth that is at a certain distance from the center of resistance. For example, the moment can be calculated by the vector cross product of a vector force applied to a location that corresponds to the displacement vector from the center of resistance. The moment may include a vector pointing in one direction. A moment opposite to another moment may include one of a moment vector oriented towards a first side of an object (such as a tooth) and another moment vector oriented towards the opposite side of the object (such as a tooth). Any discussion herein regarding applying a force to a patient's tooth equally applies to applying a moment to a tooth, and vice versa.

[0276] The "multiple teeth" as used herein may include two or more teeth. Multiple teeth may include adjacent teeth, but do not necessarily include adjacent teeth. In some embodiments, one or more posterior teeth include one or more of molars, premolars, or canines, and one or more anterior teeth include one or more of central incisors, lateral incisors, molars, first bicuspids, or second bicuspids.

[0277] The embodiments disclosed herein may be well suited for moving one or more teeth in a first set of one or more teeth or moving one or more teeth in a second set of one or more teeth and combinations thereof.

[0278] The embodiments disclosed herein may be well suited for combination with one or more commercially available tooth movement components (such as attachments and polymer shell appliances). In some embodiments, the appliance and one or more attachments are configured to move one or more teeth along a tooth movement vector that includes six degrees of freedom, where three degrees of freedom are rotational degrees of freedom and three degrees of freedom are translational degrees of freedom.

[0279] The repositioning of teeth can be accomplished using a series of removable elastic positioning appliances, such as those available from Align Technology, Inc., the assignee of the present disclosure. Systems. These devices can have a thin shell made of an elastic material that generally conforms to the patient's teeth but is slightly misaligned with the initial or immediately preceding tooth configuration. Placing the device on the teeth can apply a controlled force at specific locations to gradually move the teeth into a new configuration. Repeating this process with successive devices that incorporate the new configuration ultimately moves the teeth through a series of intermediate configurations or alignment patterns to the final desired configuration. Tooth repositioning can be accomplished with other series of removable orthodontic devices and / or dental devices (including polymer shell devices).

[0280] The computer systems used herein are intended to be interpreted broadly. Generally speaking, a computer system includes a processor, a memory, a non-volatile memory, and an interface. A typical computer system usually includes at least a processor, a memory, and a device (e.g., a bus) that couples the memory to the processor. For example, the processor can be a general-purpose central processing unit (CPU) such as a microprocessor or a special-purpose processor such as a microcontroller.

[0281] By way of example and not limitation, the memory can include random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed. The bus can also couple the processor to the non-volatile memory. The non-volatile memory is typically a magnetic floppy disk or a magnetic hard disk, a magneto-optical disk, an optical disk, a read-only memory (ROM) such as a CD-ROM, EPROM, or EEPROM, a magnetic card or an optical card, or other forms of memory for large amounts of data. During the process of executing software on a computer system, some of this data is typically written into the memory through a direct memory access process. The non-volatile memory can be local, remote, or distributed. The non-volatile memory is optional because all applicable data available in the memory can be used to create the system.

[0282] The software is typically stored in the non-volatile memory. In fact, for large programs, it may even be impossible to store the entire program in the memory. Nevertheless, it should be understood that in order for the software to run, if necessary, the software is moved to a computer-readable location suitable for processing, and for the purposes of illustration, this location is referred to herein as the memory. Even when the software is moved to the memory for execution, the processor typically uses hardware registers to store values associated with the software and, ideally, local caches to speed up execution. As used herein, when a software program is said to be "implemented in a computer-readable storage medium," it is assumed that the software program is stored in a suitable known or convenient location (from non-volatile memory to hardware registers). When at least one value associated with the program is stored in a register readable by the processor, the processor is considered to be "configured to execute the program."

[0283] In one operating example, a computer system can be controlled by operating system software, which is a software program including a file management system such as a disk operating system. An example of operating system software with associated file management system software is the well-known operating system family and its associated file management system from Microsoft Corporation of Redmond, Washington. Another example of operating system software with its associated file management system software is the Linux operating system and its associated file management system. The file management system is typically stored in non-volatile memory and causes the processor to perform various actions required by the operating system to input and output data and store data in memory, including storing files on non-volatile memory.

[0284] The bus can also couple the processor to an interface. The interface can include one or more input and / or output (I / O) devices. By way of example and not limitation, depending on the considerations of a particular implementation or other considerations, the I / O devices can include a keyboard, a mouse or other pointing device, a disk drive, a printer, a scanner, and other I / O devices including a display device. By way of example and not limitation, the display device can include a cathode ray tube (CRT), a liquid crystal display (LCD), or some other suitable known or convenient display device. The interface can include one or more of a modem or a network interface. It will be appreciated that a modem or a network interface can be considered part of the computer system. The interface can include an analog modem, an ISDN modem, a cable modem, a token ring interface, a satellite transmission interface (such as “DirectPC”), or other interfaces for coupling the computer system to other computer systems. The interface enables the computer system and other devices to be coupled together in a network.

[0285] The computer system can be compatible with a cloud-based computing system or implemented as part of or via a cloud-based computing system. As used herein, a cloud-based computing system is a system that provides virtualized computing resources, software, and / or information to end-user devices. Computing resources, software, and / or information can be virtualized by maintaining centralized services and resources that can be accessed by edge devices via a communication interface (such as a network). The “cloud” can be a marketing term and, for the purposes of this disclosure, can include any network described herein. A cloud-based computing system can involve a subscription service or a usage-based pricing model. A user can access the protocol of a cloud-based computing system via a web browser or other container application located on their end-user device.

[0286] A computer system can be implemented as an engine, a part of an engine, or through multiple engines. As used herein, an engine includes one or more processors or a part of one or more processors. A part of one or more processors can include some, but not all, of the hardware that includes any given one or more processors, such as a subset of registers, a part of a processor dedicated to one or more threads of a multi-threaded processor, a time slice during which a processor is dedicated in whole or in part to executing a portion of the functions of an engine, and so on. Thus, a first engine and a second engine can have one or more dedicated processors, or the first engine and the second engine can share one or more processors with each other or with other engines. Depending on the considerations of a particular implementation or other considerations, an engine can be centralized, or its functions can be distributed. An engine can include hardware, firmware, or software embodied in a computer-readable medium for execution by a processor. For example, as described with reference to the figures herein, a processor uses implemented data structures and methods to transform data into new data.

[0287] The engines described herein, or the engines that can implement the systems and devices described herein, can be cloud-based engines. As used herein, a cloud-based engine is an engine that can run applications and / or functions using a cloud-based computing system. All or a part of an application and / or function can be distributed across multiple computing devices and need not be restricted to only one computing device. In some embodiments, a cloud-based engine can execute functions and / or modules accessed by an end user via a web browser or a container application without locally installing the functions and / or modules on the end user's computing device.

[0288] As used herein, a data warehouse is intended to include a repository having any applicable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. For example, a data warehouse can be implemented as software embodied in a physical computer-readable medium located on a dedicated machine, in firmware, in hardware, in their combination, or in an applicable known or convenient device or system. Components associated with a data warehouse (e.g., a database interface) can be considered a "part" of the data warehouse, a part of some other system component, or a combination thereof, although the physical location and other characteristics of the components associated with a data warehouse are not critical for understanding the techniques described herein.

[0289] A data warehouse may include data structures. As used herein, a data structure is associated with a particular way of storing and organizing data in a computer so that it can be effectively utilized in a given context. Data structures are generally based on the computer's ability to retrieve data located at any position in its memory and store data at any position in its memory, where the position is specified by an address, which is a bit string that can itself be stored in memory and manipulated by a program. Thus, some data structures are based on using arithmetic operations to calculate the address of a data item; while other data structures are based on storing the address of a data item in the structure itself. Many data structures use both principles, sometimes in non-trivial combinations. The implementation of a data structure typically involves writing a set of programs that create and manipulate instances of the structure. The data warehouse described herein may be a cloud-based data warehouse. A cloud-based data warehouse refers to a data warehouse that is compatible with cloud-based computing systems and engines.

[0290] Although reference has been made to instruments including polymer housing instruments, the embodiments disclosed herein are well suited for use with many instruments for accommodating teeth (e.g., instruments that do not have one or more of a polymer or a housing). For example, an instrument can be made of one or more of many materials, such as metal, glass, reinforcing fibers, carbon fibers, composite materials, reinforced composite materials, aluminum, biomaterials, and combinations thereof. For example, an instrument can be formed in a variety of ways, such as by thermoforming or direct manufacturing as described herein. Alternatively or in combination, an instrument can be manufactured by machining, e.g., an instrument manufactured from a block of material by computer numerical control machining. Additionally, although orthodontic instruments are mentioned herein, at least some of the techniques described herein are applicable to prosthetic instruments and / or other dental instruments, including but not limited to dental crowns, dental ornaments, tooth whitening instruments, tooth protection instruments, etc.

[0291] While the preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. Many variations, changes, and substitutions can now be made by those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention. The following claims are intended to define the scope of the present invention, and methods and structures within the scope of these claims and their equivalents will thereby be covered.

Claims

1. A computer - implemented method for simulating orthodontic treatment, the computer - implemented method comprising: Capture a first 2D image that includes a representation of a patient's face and the patient's teeth; Identify one or more shapes associated with at least one tooth of the patient's teeth; Construct a parametric 3D model of the patient's teeth based on the first 2D image using one or more case-specific parameters of the one or more shapes associated with the at least one tooth of the patient's teeth; Simulate the position of the patient's teeth by rendering the parametric 3D model of the patient's teeth in a predetermined position; Simulate the patient's gums by constructing a parametric gum line that includes patient-specific input parameters; And Render a second 2D image representing the patient's face, the second 2D image representing the patient's teeth according to the predetermined position and the patient's gums according to the parametric gum line.

2. The computer - implemented method according to claim 1, wherein, Constructing the parametric gum line includes: Determine one or more edges of the teeth, one or more edges of the gums, and one or more edges of the lips in the first 2D image; Determine the patient-specific input parameters from the one or more edges of the teeth, the one or more edges of the gums, and the one or more edges of the lips.

3. The computer - implemented method according to claim 2, wherein, Constructing the parametric gum line further includes modifying the patient-specific input parameters based on simulated input parameters.

4. The computer - implemented method according to claim 1, wherein, The patient-specific input parameters include one or more of the following: the maximum distance between the lips and the gum line, the minimum distance between the lips and the gum line, or the width of the teeth at the incisal edge.

5. The computer - implemented method according to claim 3, wherein, The simulated input parameters include one or more of the following: gum tip parameters, ideal tooth height, tooth thickness, distance from the gum line to the lips, or visible tooth height.

6. The computer - implemented method according to claim 3, wherein, Constructing the parametric gum line includes: Model the parametric 3D model of the patient's teeth as a cylinder; Identify coordinates on the cylinder defined by the simulated input parameters, the simulated input parameters including gum tip parameters, visible tooth height, and distance from the gum line to the lips; Perform a cross-section cut on the cylinder by an inclined plane; and Simulate the parametric gum line as the intersection line between the inclined plane and the cylinder projected onto the parametric 3D model.

7. The computer - implemented method according to claim 1, further comprising: Render the leveled gum line, wherein rendering the leveled gum line includes aligning the coordinates of the parametric gum line.

8. The computer - implemented method according to claim 7, wherein, The coordinates of the parametric gum line are aligned with a quadratic polynomial.

9. The computer - implemented method according to claim 2, further comprising: Locate the edges of the teeth, gums, and lips in the first 2D image; And Align the parametric 3D tooth model with the edges of the teeth, gums, and lips in the first 2D image.

10. The computer - implemented method according to claim 1, wherein, The first 2D image includes a side image representing the profile of the patient's face.

11. The computer - implemented method according to claim 1, wherein, The predetermined position includes an initial position.

12. The computer - implemented method according to claim 1, wherein, The predetermined position includes a final position.

13. The computer-implemented method according to claim 12, wherein, The final position includes the position after an orthodontic treatment plan, a restorative treatment plan, or a combination of an orthodontic treatment plan and a restorative treatment plan.

14. The computer-implemented method according to claim 1, wherein, Capturing the first 2D image includes: instructing a mobile phone or camera to image the patient's face, or collecting the first 2D image from a storage device or network system.

15. The computer-implemented method according to claim 1, wherein, Constructing a parametric 3D model of a patient's teeth based on the first 2D image using one or more case - specific parameters of one or more shapes associated with at least one of the patient's teeth includes: Roughly aligning the teeth represented in the parametric 3D model with the patient's teeth represented in the first 2D image; and Performing a desired step to first determine the probability that the projection of the silhouette of the parametric 3D model matches one or more edges of the first 2D image.

16. The computer-implemented method according to claim 15, wherein, Constructing a parametric 3D model of a patient's teeth based on the first 2D image using one or more case - specific parameters of one or more shapes associated with at least one of the patient's teeth includes: Performing a maximization step using a small - angle approximation to linearize the rigid transformation of the teeth in the parametric 3D model; and Performing the desired step to second - determine the probability that the projection of the silhouette of the parametric 3D model matches the edges of the 2D image.

17. The computer-implemented method according to claim 16, further comprising: Performing a first round of multiple loop traversals of the desired step and the maximization step using a first subset of the parameters; And After performing the first round of multiple loop traversals of the desired step and the maximization step using the first subset of the parameters of the parametric 3D model, performing a second round of multiple loop traversals of the desired step and the maximization step using the first subset of the parameters and a second subset of the parameters.

18. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

19. A system comprising: A photo - parameterization engine configured to generate a parameterized gingival line and a 3D parametric dental arch model from 2D images of a patient's face, gums, and teeth, the 3D parametric dental arch model including case - specific parameters of the shape of at least one of the patient's teeth, the parameterized gingival line including patient - specific input parameters of the shape of the gums; A parametric treatment prediction engine configured to simulate an orthodontic treatment of a patient based on the 3D parametric dental arch model and historical models of multiple patients; And A treatment projection rendering engine configured to render the 3D parametric dental arch model, wherein the photo - parameterization engine, the parametric treatment prediction engine, and the treatment projection rendering engine are together configured to perform the method according to claim 1.

Images