Systems, methods, and devices for facial and oral static and dynamic analysis

Abstract

The disclosure relates to systems and methods for determining static and dynamic characteristics of a patient. In some embodiments, the systems and methods herein may be used in dental treatment planning. In some embodiments, a method may include receiving, by a computer system, facial scan data of a patient, the facial scan data including image data and depth data. A method may include determining, by the computer system, a plurality of reference points based on the face scan data. A method may include determining, by the computer system, one or more reference points, lines, or planes. A method may include determining, by the computer system, one or more ratios associated with a dental treatment plan. A method may include determining, by the computer system, a dynamic feature.

Description

[0001] Incorporation by reference into any priority application

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 363,135, filed on April 18, 2022, entitled “Systems, Methods, and Devices for Dental Diagnosis and Treatment,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application relates to systems, methods and devices that can be used to assist in dental diagnosis and treatment. Some embodiments relate to capturing and manipulating three-dimensional images of a patient. Some embodiments relate to capturing or tracking tooth movement after alignment with the patient's face. Some embodiments relate to three-dimensional modeling of a patient's face. Background Art

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0005] Proper placement of a patient's own teeth, artificial teeth, or both may be important for both aesthetic and functional reasons. Often, when a dental practitioner is planning or performing orthodontic surgery, prosthetic surgery, or both, the practitioner may lack some information that would be helpful in positioning teeth or selecting artificial teeth. Furthermore, in some cases, the practitioner may rely on flawed information.

[0006] Often, practitioners may rely on limited views of the patient's anatomy that give the practitioner limited insight, making it difficult for the practitioner to consider the patient's overall architecture and instead focus on the positioning or repositioning of individual teeth, which can result in poor patient outcomes and / or higher costs as treatment may be prolonged.

[0007] When developing a treatment plan, practitioners often consider the positioning of the patient's teeth relative to each other (which can be obtained using, for example, a dental mold or an intraoral scanner) as well as information about the patient's facial structure, such as the location of certain features of the patient's skull, and the positioning of the teeth within the skull. However, collecting such information can be time consuming, uncomfortable, and prone to error. Various mechanical, electronic, electromechanical, optical, opto-mechanical, and radiographic devices and methods have been developed, but their use can be cumbersome. The difficulty in using such devices and methods can lead to errors, and patient treatment outcomes can vary significantly based on the practitioner's skill in using such devices and methods. Summary of the invention

[0008] For purposes of the present disclosure, certain aspects, advantages, and novel features are described herein. It is to be understood that not all of these advantages may be achieved according to any particular embodiment. Thus, for example, one skilled in the art will recognize that the present invention may be embodied or practiced in a manner that achieves one advantage or a group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.

[0009] In some aspects, the technology described herein relates to a method for determining characteristics of a patient, the method comprising: receiving, by a computer system, facial scan data of the patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and determining, by the computer system, one or more ratios associated with a dental treatment plan.

[0010] In some aspects, the technology described herein relates to a method, wherein the plurality of reference points includes at least one of an infraorbital point, a condyle point, a pupil point, a nasal wing point, a subnasal point, a submental point, a midpoint of the hairline, a midpoint of the glabella, a mandibular angle point, a nasal protrusion point, an upper lip point, a lower lip point, an outer corner of the eye point, a tragus point, a skin nasion point, or a tragus angle apex.

[0011] In some aspects, the techniques described herein relate to a method in which determining the plurality of reference points includes: generating a low-dimensional representation of the patient's face based on facial scan data; and determining the one or more reference points using a reference point recognition model.

[0012] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

[0013] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on depth data.

[0014] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on image data.

[0015] In some aspects, the technology described herein relates to a method wherein determining the plurality of reference points includes: applying, by a computer system, a deformable mask to facial scan data; and deforming the deformable mask, wherein deforming the deformable mask includes adjusting the deformable mask to reduce differences between the deformable mask and the facial scan data.

[0016] In some aspects, the technology described herein relates to a method, wherein the facial scan data includes motion information, wherein the method further includes: determining, by a computer system, dynamic characteristics of the patient.

[0017] In some aspects, the technology described herein relates to a method in which determining a dynamic feature includes detecting, by a motion detection model, movement of a mandible of a patient.

[0018] In some aspects, the technology described herein relates to a method further comprising: receiving, by a computer system, a dental model of a patient; and co-registering the dental model with facial scan data.

[0019] In some aspects, the technology described herein relates to a method wherein the dental model comprises an upper jaw model and a lower jaw model.

[0020] In some aspects, the technology described herein relates to a method further comprising: generating a facial model of a patient; co-registering the dental model and the facial model; and determining a range of motion limit of the patient's mandible, wherein the range of motion limit is determined by determining an amount of closure when the maxillary model collides with the mandibular model.

[0021] In some aspects, the technology described herein relates to a method further comprising determining, by a computer system, a condition associated with a patient.

[0022] In some aspects, the technology described herein relates to a method in which generating a low-dimensional representation includes determining a set of eigenfaces and a set of associated weights, wherein a patient's face is described by a linear combination of the eigenfaces and their associated weights.

[0023] In some aspects, the techniques described herein relate to a method in which deforming a deformable mask includes one or more of cage deformation, skeletal animation, or mesh interpolation.

[0024] In some aspects, the technology described herein relates to a method for determining features of a patient, the method comprising: receiving, by a computer system, facial scan data of the patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and determining, by the computer system, dynamic features of the patient.

[0025] In some aspects, the technology described herein relates to a method, wherein the plurality of reference points includes at least one of an infraorbital point, a condyle point, a pupil point, a nasal wing point, a subnasal point, a submental point, a midpoint of the hairline, a midpoint of the glabella, a mandibular angle point, a nasal protrusion point, an upper lip point, a lower lip point, an outer corner of the eye point, a tragus point, a skin nasion point, or a tragus angle apex.

[0026] In some aspects, the techniques described herein relate to a method in which determining the plurality of reference points includes: generating a low-dimensional representation of the patient's face based on facial scan data; and determining the one or more reference points using a reference point recognition model.

[0027] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

[0028] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on depth data.

[0029] In some aspects, the techniques described herein relate to a method in which the low-dimensional representation is based on image data.

[0030] In some aspects, the technology described herein relates to a method wherein determining the plurality of reference points includes: applying, by a computer system, a deformable mask to facial scan data; and deforming the deformable mask, wherein deforming the deformable mask includes adjusting the deformable mask to reduce differences between the deformable mask and the facial scan data.

[0031] In some aspects, the technology described herein relates to a method in which determining a dynamic feature includes detecting, by a motion detection model, movement of a mandible of a patient.

[0032] In some aspects, the technology described herein relates to a method further comprising: determining a facial model of a patient; receiving a skeletal model of the patient; and co-registering the skeletal model and the facial model.

[0033] In some aspects, the technology described herein relates to a method further comprising: determining contact relationships between bones of a skeletal model.

[0034] In some aspects, the technology described herein relates to a method, further comprising: determining a facial model of a patient; receiving a dental model of the patient, the dental model comprising maxillary teeth and mandibular teeth; and co-registering the dental model with the facial model, wherein co-registering the dental model with the facial model produces an orofacial model.

[0035] In some aspects, the technology described herein relates to a method further comprising: determining an occlusal plane.

[0036] In some aspects, the technology described herein relates to a method further comprising determining a function generating surface, the function generating surface indicating an envelope of a function of dental arch motion.

[0037] In some aspects, the technology described herein relates to a method further comprising: determining a hinge axis.

[0038] In some aspects, the technology described herein relates to a method further comprising: determining a condylar slope, the determination being based at least in part on a protrusive movement.

[0039] In some aspects, the techniques described herein relate to a method further comprising determining a left Bennett angle, the determination based at least in part on a right side shift.

[0040] In some aspects, the techniques described herein relate to a method further comprising determining a right Bennett angle, the determination based at least in part on a left side shift.

[0041] In some aspects, the technology described herein relates to a method for training a machine learning model, the method comprising: receiving a plurality of facial scans associated with a plurality of individuals, the facial scans comprising image data and depth data, at least one of the facial scans being labeled to indicate the location of one or more reference points; generating a low-dimensional representation for each of the plurality of facial scans; providing the generated low-dimensional representation to the machine learning model; and training the machine learning model, wherein training the machine learning model comprises adjusting one or more weights of the machine learning model.

[0042] In some aspects, the techniques described herein relate to a method in which generating a low-dimensional representation includes computing one or more weights of one or more eigenfaces.

[0043] In some aspects, the technology described herein relates to a method, further comprising, before generating a low-dimensional representation, performing the following operations: determining the location of one or more features to be excluded using different machine learning models; and removing the one or more features to be excluded, wherein the removing comprises one or more of blurring the one or more features to be excluded or placing a solid object over the one or more features to be excluded.

[0044] In some aspects, the technology described herein relates to a method wherein generating a low-dimensional representation includes generating a first two-dimensional representation, the method further including: generating a second two-dimensional representation for each facial scan in the plurality of facial scans, the second two-dimensional representation being different from the first two-dimensional representation; generating a second low-dimensional representation for each second two-dimensional representation; and performing the following operations after training the machine learning model: providing the second low-dimensional representation to the second machine learning model; providing at least one of the one or more weights of the machine learning model to the second machine learning model; and training the second machine learning model, wherein training the second machine learning model includes adjusting one or more weights of the second machine learning model.

[0045] In some aspects, the technology described herein relates to a system for determining characteristics of a patient, the system comprising: one or more processors; and a non-volatile storage medium having instructions embodied thereon, the instructions, when executed by the one or more processors, causing the system to perform the following steps: receiving facial scan data of the patient by a computer system, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and determining, by the computer system, one or more ratios associated with a dental treatment plan.

[0046] In some aspects, the technology described herein relates to a system wherein the plurality of reference points include at least one of an infraorbital point, a condyle point, a pupil point, a nasal wing point, a subnasal point, a submental point, a midpoint of the hairline, a midpoint of the glabella, a mandibular angle point, a nasal process point, an upper lip point, a lower lip point, an outer corner of the eye point, a tragus point, a skin nasion point, or a tragus angle apex.

[0047] In some aspects, the techniques described herein relate to a system wherein determining the plurality of reference points includes: generating a low-dimensional representation of the patient's face based on facial scan data; and determining the one or more reference points using a reference point recognition model.

[0048] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

[0049] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on depth data.

[0050] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on image data.

[0051] In some aspects, the technology described herein relates to a system wherein determining the plurality of reference points includes: applying, by a computer system, a deformable mask to facial scan data; and deforming the deformable mask, wherein deforming the deformable mask includes adjusting the deformable mask to reduce differences between the deformable mask and the facial scan data.

[0052] In some aspects, the technology described herein relates to a system wherein the facial scan data includes motion information, wherein the steps further include: determining, by a computer system, dynamic characteristics of the patient.

[0053] In some aspects, the technology described herein relates to a system in which determining a dynamic feature includes detecting, by a motion detection model, movement of a mandible of a patient.

[0054] In some aspects, the technology described herein relates to a system wherein the steps further include: receiving, by a computer system, a dental model of a patient; and co-registering the dental model with the facial scan data.

[0055] In some aspects, the technology described herein relates to a system wherein the dental model includes an upper jaw model and a lower jaw model.

[0056] In some aspects, the technology described herein relates to a system wherein the steps further include: generating a facial model of the patient; co-registering the dental model and the facial model; and determining the range of motion limits of the patient's mandible, wherein the range of motion limits are determined by determining the amount of closure when the maxillary model collides with the mandibular model.

[0057] In some aspects, the technology described herein relates to a system wherein the steps further include determining, by the computer system, a condition associated with the patient.

[0058] In some aspects, the techniques described herein relate to a system wherein generating a low-dimensional representation includes determining a set of eigenfaces and a set of associated weights, wherein a face of a patient is described by a linear combination of the eigenfaces and their associated weights.

[0059] In some aspects, the techniques described herein relate to a system wherein deforming a deformable mask includes one or more of cage deformation, skeletal animation, or mesh interpolation.

[0060] In some aspects, the technology described herein relates to a system for determining features of a patient, the system comprising: one or more processors; and a non-volatile storage medium having instructions embodied thereon, the instructions, when executed by the one or more processors, causing the system to perform the following steps: receiving facial scan data of a patient by a computer system, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and determining, by the computer system, dynamic features of the patient.

[0061] In some aspects, the technology described herein relates to a system wherein the plurality of reference points include at least one of an infraorbital point, a condyle point, a pupil point, a nasal wing point, a subnasal point, a submental point, a midpoint of the hairline, a midpoint of the glabella, a mandibular angle point, a nasal process point, an upper lip point, a lower lip point, an outer corner of the eye point, a tragus point, a skin nasion point, or a tragus angle apex.

[0062] In some aspects, the techniques described herein relate to a system wherein determining the plurality of reference points includes: generating a low-dimensional representation of the patient's face based on facial scan data; and determining the one or more reference points using a reference point recognition model.

[0063] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

[0064] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on depth data.

[0065] In some aspects, the techniques described herein relate to a system in which the low-dimensional representation is based on image data.

[0066] In some aspects, the technology described herein relates to a system wherein determining the plurality of reference points includes: applying, by a computer system, a deformable mask to facial scan data; and deforming the deformable mask, wherein deforming the deformable mask includes adjusting the deformable mask to reduce differences between the deformable mask and the facial scan data.

[0067] In some aspects, the technology described herein relates to a system in which determining a dynamic feature includes detecting, by a motion detection model, movement of a mandible of a patient.

[0068] In some aspects, the technology described herein relates to a system wherein the steps further include: determining a facial model of the patient; receiving a skeletal model of the patient; and co-registering the skeletal model and the facial model.

[0069] In some aspects, the technology described herein relates to a system wherein the steps further include: determining contact relationships between bones of the skeletal model.

[0070] In some aspects, the technology described herein relates to a system wherein the steps further include: determining a facial model of a patient; receiving a dental model of the patient, wherein the dental model includes maxillary teeth and mandibular teeth; and co-registering the dental model with the facial model, wherein co-registering the dental model with the facial model produces an orofacial model.

[0071] In some aspects, the technology described herein relates to a system wherein the steps further comprise: determining an occlusal plane.

[0072] In some aspects, the technology described herein relates to a system wherein the steps further include determining a function generating surface, the function generating surface indicating an envelope of the function of the dental arch motion.

[0073] In some aspects, the technology described herein relates to a system wherein the steps further include determining a hinge axis.

[0074] In some aspects, the technology described herein relates to a system wherein the steps further include determining a condylar slope, the determination being based at least in part on the protrusive movement.

[0075] In some aspects, the techniques described herein relate to a system wherein the steps further include determining a left Bennett angle, the determination being based at least in part on a right side shift.

[0076] In some aspects, the techniques described herein relate to a system wherein the steps further include determining a right Bennett angle, the determination being based at least in part on a left side shift.

[0077] In some aspects, the technology described herein relates to a system for training a machine learning model, the system comprising: one or more processors; and a non-volatile storage medium having instructions embodied thereon, the instructions, when executed by the one or more processors, causing the system to perform the following steps: receiving a plurality of facial scans associated with a plurality of individuals, the facial scans comprising image data and depth data, at least one of the facial scans being marked to indicate the location of one or more reference points; generating a low-dimensional representation for each of the plurality of facial scans; providing the generated low-dimensional representation to a machine learning model; and training the machine learning model, wherein training the machine learning model comprises adjusting one or more weights of the machine learning model.

[0078] In some aspects, the techniques described herein relate to a system wherein generating a low-dimensional representation includes computing one or more weights of one or more eigenfaces.

[0079] In some aspects, the technology described herein relates to a system wherein the steps further include, before generating the low-dimensional representation, performing the following operations: determining the location of one or more features to be excluded using different machine learning models; and removing the one or more features to be excluded, wherein the removing includes one or more of blurring the one or more features to be excluded or placing a solid object over the one or more features to be excluded.

[0080] In some aspects, the technology described herein relates to a system wherein generating a low-dimensional representation includes generating a first two-dimensional representation, wherein the steps further include: generating a second two-dimensional representation for each facial scan in the plurality of facial scans, the second two-dimensional representation being different from the first two-dimensional representation; generating a second low-dimensional representation for each second two-dimensional representation; and performing the following operations after training the machine learning model: providing the second low-dimensional representation to the second machine learning model; providing at least one of the one or more weights of the machine learning model to the second machine learning model; and training the second machine learning model, wherein training the second machine learning model includes adjusting one or more weights of the second machine learning model.

[0081] All such embodiments are intended to fall within the scope of the present disclosure.These and other embodiments will become apparent to those skilled in the art from the following detailed description, which proceeds from the accompanying drawings, which are incorporated into and form a part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] These and other features, aspects and advantages of the present disclosure are described with reference to the drawings of certain embodiments, which are intended to illustrate but not limit the present disclosure. It should be understood that the drawings incorporated in and constituting a part of this specification are for the purpose of illustrating the concepts disclosed herein and may not be drawn to scale.

[0083] Figure 1-5 Examples are shown of patients using a smartphone to capture their faces in various poses, such as mouth closed, mouth open, smiling, etc., in accordance with some embodiments.

[0084] Figure 6 An example interface showing a patient from a side view is shown in accordance with some embodiments.

[0085] Figure 7 is an example diagram showing the positions of various primary and secondary points on a face.

[0086] Figure 8is a diagram illustrating the locations of selected reference points according to some embodiments.

[0087] Fig. 9 is a flowchart for training an artificial intelligence or machine learning model according to some embodiments.

[0088] Fig.10 An example process for training and using an AI / ML model according to some embodiments is shown.

[0089] Fig.11 Examples of various ratios that may be relevant to dental treatment and planning are shown.

[0090] Fig.12 is a flow chart illustrating an example process for generating a low-dimensional representation according to some embodiments.

[0091] Fig.13 is a flow chart illustrating an example process according to some embodiments.

[0092] Fig.14 is a flow chart illustrating an example process according to some embodiments.

[0093] Figures 15A-15B is a flow chart illustrating an example process according to some embodiments.

[0094] Figure 16-17 An example user interface is shown in accordance with some embodiments.

[0095] Fig.18 is a block diagram depicting an embodiment of a computer hardware system configured to run software for implementing one or more embodiments disclosed herein. DETAILED DESCRIPTION

[0096] Embodiments of the present disclosure will now be described with reference to the accompanying drawings. The terms used in the description presented herein are not intended to be interpreted in any limiting or restrictive manner, but are utilized solely because they are combined with the detailed description of the embodiments of the present disclosure. In addition, embodiments of the present disclosure may include several novel features, no single feature of which is solely responsible for its desirable properties or is necessary for practicing the embodiments of the present disclosure described herein. For the purposes of the present disclosure, certain aspects, advantages, and novel features of various embodiments are described herein. It should be understood that, according to any particular embodiment, not all of these advantages may be achieved. Therefore, for example, a person skilled in the art will recognize that an embodiment may be implemented in a manner that achieves an advantage or a group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein.

[0097] introduction

[0098] Proper placement of a patient's own teeth, artificial teeth, or both may be important for both aesthetic and functional reasons. Often, when a dental practitioner is planning or performing an orthodontic or prosthetic procedure, the practitioner may lack some information that would be helpful in positioning teeth, selecting artificial teeth, and the like. For example, the practitioner may lack information about the patient's jaw movements, the patient's face when making actions such as smiling or eating, the patient's appearance when viewed from different viewpoints, and the like. In some cases, the practitioner may rely on limited or even incorrect information about the patient because it may be challenging to collect and accurately represent and manipulate the data.

[0099] In some cases, practitioners may rely on limited views of the patient's anatomy that give them limited insight, making it difficult for them to consider the patient's overall architecture and kinematics, and instead focus on the positioning or repositioning of individual teeth, which can result in poor patient outcomes, prolonged treatment time, increased costs, etc.

[0100] When developing a treatment plan, practitioners often consider both the facial structure and the positioning of the patient's teeth, which can be obtained in various ways, such as by collecting dental molds or using intraoral scanners. Practitioners can consider the patient's facial structure, such as the position of certain features of the patient's skull, the position of the nose, the position of the lips, etc. However, collecting such information can be time consuming, uncomfortable, expensive, and prone to error. For example, it can be difficult to determine the position of a patient's teeth within the oral cavity relative to other facial structures. With limited information, practitioners may not be able to develop a treatment plan that properly accounts for the patient's skeletal structure, movement, tooth positioning, etc.

[0101] Various mechanical, electronic, electromechanical, optical, optomechanical, and radiographic devices and methods have been developed to assist in dental treatment planning. However, the use of these devices and methods can be cumbersome. Coordinating data from different sources can be difficult. For example, there can be significant challenges in determining the relative positioning of teeth and skull, determining the positioning of teeth relative to the lip line, etc. The difficulty in using such devices and methods can lead to errors, and patient treatment outcomes can vary significantly depending on the practitioner's skill in using such devices and methods.

[0102] In some cases, practitioners may use models (e.g., mechanical models) to represent the morphology and dynamic structure of the patient. However, these efforts have often relied on simple models or average faces, which may not accurately reflect the actual patient anatomy. To be suitable for dental treatment, it may be important that the model represents the actual structure and dynamic features of the patient with a high level of accuracy. Without this accuracy, treatment planning may achieve suboptimal results.

[0103] Therefore, there is a need for systems and methods that can take into account patient-specific parameters, that are easier to use, less prone to error, and that can give a more complete picture of the patient's static and dynamic structure.

[0104] An additional limitation of existing systems and methods may be the high cost of the equipment used to collect the data. This may present a significant barrier to many practitioners, who may have difficulty affording such equipment. The cost to the patient may be significant, which may make dental treatment out of reach for some patients. Therefore, systems and methods that can be implemented at a relatively low cost are desirable. Preferably, such systems and methods can be relatively easy to use.

[0105] Facial data capture

[0106] Preferably, when making a diagnosis, a treatment plan, or both, the actual facial structure and jaw movement of the patient are considered. In some cases, a motion capture system can be used to map the movement of the patient's face (which, in some embodiments, may include the patient's jaw) during actions such as speaking, smiling, eating, drinking, etc. The patient's jaw may or may not include the patient's mandibular teeth. In some embodiments, the capture of the patient's face may include at least a partial capture of the patient's teeth, but in some other embodiments, the patient's teeth may not be captured during the facial data capture process. In some embodiments, a marker may be applied to the patient's face, and an infrared camera or other optical tracker may be used to track movement. However, some providers may not have access to dedicated equipment for facial motion capture. Therefore, it would be beneficial for providers to be able to capture facial structure and movement without the need for dedicated equipment. In some embodiments, only static information may be considered. In some embodiments, only dynamic information may be considered. In some embodiments, both dynamic and static information may be considered. Some embodiments described herein may use off-the-shelf equipment to achieve data collection, which can be obtained at a low cost compared to the cost of dedicated equipment. For example, in some embodiments, consumer electronic devices such as smartphones, tablets, or laptops equipped with depth sensing cameras may be used to collect data. In some embodiments, the external depth sensing camera may be connected to the consumer electronic device, for example via USB.

[0107] As consumer imaging hardware becomes more advanced, the need for professional or specialized equipment has decreased. For example, the imaging system in a smartphone, tablet computer, laptop computer, etc. may have features that allow depth data to be captured. For example, such imaging systems may use structured light, time of flight, stereology, light detection and ranging (LIDAR) or other depth sensing technologies. For example, an imaging system may project an infrared dot onto the user's face to determine depth information. Alternatively or in addition, the depth may be estimated by, for example, using an infrared emitter and a dual camera, thereby allowing the depth to be determined based on the stereoscopic parallax between the images captured by each camera. In some cases, a stereo camera detected within the visible range may be used. Therefore, instead of using a dedicated motion capture camera and marker, it is possible to capture the 3D shape and texture of the patient's face using consumer electronics hardware. In some embodiments, the movement of the patient's face may be captured using commonly available consumer hardware. In some embodiments, a smartphone, tablet computer, laptop computer, etc. may be used to capture data about the patient's face, and another system (e.g., a desktop or laptop computer) may be used to process the captured data. However, in some cases, the data may be manipulated directly on a smartphone, tablet computer, or other capture device. Advantageously, in some embodiments, this processing can be effectively performed due to the presence of dedicated integrated circuits, instructions, etc. available in some consumer hardware. In some embodiments, the application for processing the captured data can be written to take advantage of this dedicated functionality, but in other embodiments, the application may not take advantage of this dedicated functionality. For example, if greater device or software compatibility is required, the use of specialized instructions can be avoided. In some embodiments, different applications can be used depending on the hardware or software (e.g., operating system) on which the application is to be run. In some embodiments, a single application can be written to take advantage of any available acceleration features, while having a fallback implementation that can be used when certain features are not available.

[0108] In some embodiments, the capture of the patient's face and movements can be used in conjunction with a 3D representation of the patient's teeth (referred to herein as a dental model) as part of the process of determining the optimal placement of the teeth. The dental model can be a single model or multiple models (e.g., a maxillary model and a mandibular model). The dental model of the patient's teeth can be obtained in a variety of ways, such as by using an intraoral scanner, by digitizing a mold of the patient's teeth, based on cone beam computed tomography (CBCT) data, etc. A dental practitioner can typically use at least one of these methods.

[0109] In some embodiments, teeth, gums, or both may be segmented. This may be desirable for placement of individual teeth or arrays of teeth. In some embodiments, the teeth and gums of a patient may be treated separately. In some embodiments, the teeth and gums data may be stored in a single file or model. In some embodiments, separate files or models may be used for the gums and each tooth. In some embodiments, a segment may contain multiple teeth, such as two teeth, three teeth, four teeth, etc. Although segmentation may be performed in some embodiments, segmentation is not necessary. In some embodiments, segmentation may not be performed.

[0110] Figure 1-5 Examples of patients using smartphones to capture their faces in various poses are shown, such as mouth closed, mouth open, smiling, etc. Figure 1-5 As shown, a digital representation of the captured data can be provided in a user interface. As discussed in more detail below, the captured data can be manipulated in various ways and / or can be used as a basis for generating a model representing the patient's head and teeth (referred to herein as an orofacial model) or a facial model. Figure 6 As shown, in some captures, the patient may not be facing the camera. For example, the patient may capture a side view of the patient's face. In some embodiments, a side view may be constructed based on a frontal image and corresponding depth data. In some embodiments, the patient may capture a still photo. In some embodiments, the patient may capture a video. In some embodiments, the patient may capture both a still photo and a video. In some embodiments, the captured photo or video may include depth information. In some embodiments, the captured photo or video may not include depth information.

[0111] In some embodiments, the patient may capture images, videos, facial scans, etc. of themselves, which may or may not contain depth information. In some embodiments, the practitioner may capture images, videos, facial scans, etc. of the patient, which may or may not contain depth information. In some embodiments, both the patient and the practitioner may capture images, videos, facial scans, etc. In some embodiments, the captured images (or videos) and depth information may be manipulated, displayed, or manipulated and displayed by an application running on the computing system, as described in more detail below.

[0112] Feature determination

[0113] As discussed above, determining features of the patient's face, teeth, or both may be important in developing a treatment plan. It may be important to consider both dynamic and static features of the patient's face, but in some embodiments, only dynamic features or only static features may be considered. Considering the patient's skeletal structure and the movement of the patient's skeleton may be important for both functional and aesthetic reasons, such as to provide a desired appearance and avoid premature wear of the tooth surfaces.

[0114] In some embodiments, the patient's skin texture, the shape of the patient's face, etc. may be considered when determining a treatment plan. For example, the treatment plan may vary depending on the location of the patient's lips (e.g., how far the lower edge of the upper lip is from the bottom of the patient's nose), the shape of the patient's cheeks, etc. Patients may have different skin laxity, different cheek fat levels, etc., which may significantly affect the patient's appearance; therefore, in some embodiments, the patient's skin may be considered when determining a dental treatment plan. Such considerations may be helpful, but are not required.

[0115] As used herein, “facial scan data” may include image data and / or video data captured at least in part in the visible spectrum and its corresponding depth data (e.g., images captured with a depth-sensitive camera of a smartphone or tablet).

[0116] Static Features

[0117] The facial scan data can be used to determine the location of various reference points that can be used to develop a dental treatment plan. The reference points may include, for example, locations on the patient's head (e.g., locations of features of facial bones, skulls, ears, lips, noses, foreheads, etc.). In some embodiments, the reference points may be points on bones. In some embodiments, the reference points may be points on soft tissues. In some embodiments, the reference points may be, for example, joints. In some embodiments, the captured facial data may be used to construct a 3D model of the patient's face. In some embodiments, the captured facial data may be used in conjunction with other data such as radiological data (e.g., x-rays, computed tomography (CT), cone beam computed tomography (CBCT), etc.). In some embodiments, the radiological data may include contour teleradiography data. Using teleradiography (e.g., radiography using an x-ray source approximately two meters from a detector or patient) may be superior to some other forms of x-ray data because, for example, it uses relatively parallel x-rays, thereby reducing geometric distortions that may exist in x-ray images captured using non-parallel x-rays. In some embodiments, the contours of the face may be easily determined from the radiological image. In some embodiments, the contours of the face may be determined from the captured facial scan data. In some embodiments, a 3D model can be generated from the facial scan data, as described in more detail herein. In some embodiments, the outline of the face can be determined from the 3D model. In some embodiments, the system can be configured to co-register the outline with the x-ray image data. In some embodiments, the registration can be performed in two dimensions. In some embodiments, for example, if 3D CT or CBCT images are used, the registration can be performed in three dimensions.

[0118] In some embodiments, radiological data may not be considered. In some embodiments, radiological data may be considered. For example, radiological data indicating the position of facial bones may be important for determining bone collisions, movement restrictions, etc.

[0119] In some embodiments, the positioning of the patient's dental arch relative to the patient's face can be determined, for example, using the system and method described in U.S. Patent No. 10,265,149, entitled "Method and system for modeling the mandibular kinematics of a patient", the contents of which are incorporated herein by reference in their entirety. In some embodiments, the positioning of the patient's dental arch relative to the patient's face can be determined, for example, using the system and method described in U.S. Publication No. 2022 / 0156953, filed on March 12, 2020, entitled "Method for registering virtual models of the dental arches of an individual with a digital model of the face of said individual", the contents of which are incorporated herein by reference in their entirety.

[0120] In some embodiments, the positioning of the patient's teeth can be determined in whole or in part based on data captured during a facial scan. For example, if the patient's teeth are at least partially visible during a facial scan, the facial scan data can be used to determine the relative positioning of the patient's teeth and features of the patient's face.

[0121] After capturing the patient's facial scan, the facial scan data may be analyzed for use in determining a dental treatment plan. In some embodiments, as described in more detail herein, a 3D model (dental model) of the patient's teeth may also be used.

[0122] In some embodiments, the analysis process can be divided into three stages: an extraction stage, a combination stage, and an analysis stage. In the extraction stage, the facial scan data can be analyzed to determine reference points of interest for dental treatment planning. For example, an algorithm trained or otherwise configured to recognize reference points can be used to identify reference points on one or more images, facial scans, etc. of a patient. In some embodiments, the reference points can be determined directly based on the facial scan data. In some embodiments, the reference points can be determined indirectly, such as using a low-dimensional representation such as an eigenface or Fisherface, or using a deformable mask, as described in more detail herein. In the combination stage, subsets of reference points can be associated with each other. In the analysis stage, various calculations such as angles and distances can be performed based at least in part on underlying anatomical features. In some embodiments, more, fewer, or different stages can be used in accordance with the present disclosure.

[0123] The performance of the facial recognition algorithm, AI / ML model, etc. may depend on various factors, such as the resolution of the facial scan, image, video, etc., lighting, etc. In some embodiments, the performance of the facial recognition algorithm, AI / ML model, etc. may depend on the compression of the image, facial scan, video, etc. For example, a highly compressed image may lack detail, resulting in a less accurate analysis. Other factors may also affect the analysis, such as changes in the patient's posture compared to other images, such as training images.

[0124] In some embodiments, a 3D scan of the face can be used in conjunction with a photograph of the patient to improve analysis. For example, an image processing algorithm can combine information from the facial scan and the patient's photograph. For example, the photograph can be aligned with features of the facial scan, such as eyes, nose, lips, hairline, ears, etc. In some embodiments, the photograph can be transformed (e.g., resized, rotated, skewed, etc.) to better fit the facial scan.

[0125] Extraction phase

[0126] As described herein, various methods can be used to determine reference points, planes, etc. In some embodiments, existing methods for tasks such as facial recognition can be modified for dental treatment planning. In some embodiments, artificial intelligence (AI) / machine learning (ML) algorithms can be used. In some embodiments, various methods can be combined to achieve the desired results, as described herein. In some embodiments, a deformable mask, etc. can be used.

[0127] In some embodiments, reference points, planes, axes, landmarks, and the like may be determined using techniques associated with facial recognition, such as described in U.S. Pat. No. 10,265,149. Facial recognition uses image processing algorithms to identify unique features of a face (e.g., features of a 2D or 3D image or model of a face, or a representation of a face such as a low-dimensional representation). In some embodiments, physical features may be extracted in the form of various nodes, which may be used to determine a feature vector for the face. Facial recognition algorithms may identify unique biometric and facial features. In some embodiments, specific features may be identified. For example, a facial recognition algorithm may be trained or otherwise configured to identify features such as a nose, a mouth, eyes, and the like. In some embodiments, a facial recognition algorithm may be configured to determine the relative positioning of specific facial features (e.g., the distance from the nose to the mouth, from the nose to the eyes, from the eyes to the mouth, and the like).

[0128] The use of conventional facial recognition algorithms can provide valuable insights. However, such algorithms may not be well suited for dental treatment applications. For example, a facial recognition algorithm can determine many nodes on a human face, such as about 80 nodes. In some cases, a node may correspond directly to a reference point, but this is not necessary. In some embodiments, a reference point may be defined relative to one or more nodes. In some embodiments, a reference point may be determined by, for example, interpolation between nodes or other estimation methods. In some cases, not all nodes may be of interest in a dental treatment plan, and in some cases, a conventional facial recognition algorithm may not be able to identify nodes of interest for a dental treatment plan. For example, for a dental treatment plan, it may be particularly important to have a relatively large amount of information related to the eyes, ears, and / or lips, while points related to some other facial features (such as eyebrows) may be rarely used or not used in a dental treatment plan. Therefore, in some embodiments, a facial recognition algorithm may be written or modified so that the features of interest for dental treatment are more fully characterized, while other features are weakened or completely omitted. Conventional algorithms may not include features that are important for dental treatment. For example, conventional algorithms may not be suitable for identifying the locations of condylar points, midpoints between eyebrows, and the like. Some conventional algorithms may lack information about the overall structure of the face.

[0129] Therefore, there is a need for algorithms that can be used to determine points of interest for a dental treatment plan. For example, an algorithm or AI / ML model for facial recognition can be created or modified so that it better identifies features that are important for a dental treatment plan than a conventional algorithm or AI / ML model for facial recognition. In some embodiments, a custom model (e.g., an AI / ML model) or algorithm can be created. In some embodiments, an existing model or algorithm can be trained to identify points that are important for a dental treatment plan. In some embodiments, an AI / ML model can be trained to identify features of a patient's face directly based on captured facial scan data (e.g., based on images, depth information, or both). This approach may be relatively computationally intensive because both the image data and the depth data may include a large amount of information, but it may be relatively simple to implement and may have other advantages, such as a relatively large amount of detail, which may better determine reference points.

[0130] In some embodiments, a low dimensional representation may enable faster processing. Thus, in some embodiments, it may be beneficial to generate a low dimensional representation of the patient's face. In some embodiments, a facial recognition algorithm may be used to generate a low dimensional representation of the patient.

[0131] In some embodiments, existing facial recognition algorithms may be used or modified when developing a dental treatment plan. Examples of low-dimensional algorithms may include, for example, but not limited to, Fisher face or eigenface methods. The eigenface method is a conventional method of facial recognition using principal component analysis (PCA). In some embodiments, the algorithm may use eigenfaces to recognize faces, features of faces, or both. A set of eigenfaces may include eigenvectors. A set of eigenfaces may be generated by performing principal component analysis on a set of images of different faces. The eigenfaces may form a basis set that can be used to represent any face, such as a patient's face. After generating the eigenfaces, the image of a particular person may be represented as a linear combination of the eigenfaces. For example, the face of a particular person may be represented by a linear combination, such as 0.4EF1+0.1EF2+0.3EF3-0.2EF4, or more generally ∑ i w i EF i , where EF i represents the i-th eigenface, and w i Denotes the ith weight. It should be appreciated that, in practice, there may be any number of eigenfaces used to represent the face of an individual.

[0132] In some cases, an eigenface may represent a specific attribute of a person's face, such as symmetry, mouth size, hairline location, etc. Other eigenfaces may not have a clear correspondence to a specific facial feature or metric.

[0133] The use of eigenfaces can reduce the computational burden and speed up facial recognition by reducing the image of a person's face to a combination of a relatively small number of eigenvectors. However, some information may be lost when constructing an eigenface representation of an individual's face. As discussed above, it may be important to use highly accurate information when formulating a dental treatment plan. In some embodiments, information loss can be reduced by using more eigenfaces to represent the patient's face. For example, instead of 100 eigenvectors, 200, 300, 400, 500 or more eigenvectors may be used. However, this may result in slow performance of a system implementing this method and may complicate further analysis. Therefore, it may be advantageous to modify conventional methods such as eigenfaces or Fisher faces to be more suitable for dental treatment planning.

[0134] One difficulty associated with using eigenfaces for dental treatment planning is that the eigenface approach cannot represent the global structure of the face. In some embodiments, similar approaches (e.g., Fisher faces) can retain more information about the global structure of the face, which may be important for dental treatment and planning. Fisher faces can use linear discriminant analysis. However, as described below, facial recognition algorithms do not necessarily retain global structural information. For example, an eigenface representation of a patient can be run through an AI / ML model, which can determine relevant reference points.

[0135] In some embodiments, a modified eigenface method can be used that focuses on the features that are most relevant to the dental treatment plan, such as the location, shape, size, etc. of the eyes, nose, condyles, lips, etc. For example, eigenfaces can be used to determine facial points that are particularly relevant to dental treatment. For example, the eigenface algorithm can be configured to be suitable for identifying condyles. As an example, in some embodiments, a facial image with irrelevant features blurred or removed (e.g., by cropping irrelevant features, placing solid shapes on irrelevant features, etc.) can be used to generate eigenfaces. This method can help exclude such features or limit their impact on the eigenface representation of the patient. In some embodiments, additional details related to reference points and other information that is particularly important for dental treatment planning can be partially or completely offset by the increased loss in the representation of other parts of the patient's face. For example, the location of the patient's hairline may not be particularly relevant to determining a dental treatment plan, so accurately representing the hairline is not as important as other more important features of the patient's face.

[0136] In some embodiments, a facial recognition model can be used to pre-process an image before determining a feature face. For example, an AI / ML model can be trained using labeled images (e.g., using supervised training) to identify areas that are less important to a dental treatment plan so that they can be removed or otherwise masked. In some embodiments, the AI / ML model can output a modified image. In some embodiments, the AI / ML model can output the locations of less important features, and further processing can be performed. For example, using the location information and the corresponding original image, a new image can be generated that masks less important areas of the face. In some embodiments, this AI / ML model can take into account both depth and image data, but in some embodiments, only image data or only depth data may be used.

[0137] In some embodiments, modular eigenfaces may be used. As described herein, modular eigenfaces may be an extension of eigenfaces that enable processing of images with different postures, facial expressions, or both. The modular eigenface method may use an array of eigenfaces generated for each posture, facial expression, or both. For example, one set of eigenfaces may be generated using images of individuals facing the camera with a neutral expression, while another set of different eigenfaces may be generated using images of individuals with their heads tilted to one side while smiling. These are merely examples, and an array of eigenfaces may be generated using any desirable combination of posture and facial expression (e.g., using forward-facing images and profile images). In some embodiments, the facial expression may be a smile, a frown, etc., but the facial expression is not limited thereto. For example, a facial expression may include the appearance of an individual's face while chewing, laughing, talking, etc.

[0138] This approach may be particularly beneficial in dental treatment planning, as it may be important to ensure that a dental treatment plan will produce desirable aesthetics from various viewing angles under a variety of facial expressions and / or during various actions (eg, speaking and chewing).

[0139] The methods described above may rely on image information, which may be two-dimensional or three-dimensional. For example, eigenfaces may be generated from a set of images, and an image of a patient may be described as a linear combination of eigenfaces. According to some embodiments, the above methods may be improved or extended by using depth information captured during a facial scanning process. For example, an array of eigenfaces (or another low-dimensional representation method) may be generated based at least in part on depth information, and depth information associated with a facial capture of a particular patient may be represented as a linear combination of such eigenfaces. In some embodiments, further processing as described herein may operate only on eigenface representations using image data, only on eigenface representations using depth data, or both.

[0140] Using the methods described above, one or more eigenface representations of a patient can be generated, which can be used to more accurately represent features important to dental treatment and planning, while limiting the number of eigenfaces required to adequately represent the patient, which can shorten training time, reduce computational burden, etc.

[0141] In some embodiments, the extraction stage may include determining reference points of interest. In some embodiments, the extraction stage may include locating reference points on the patient's face. In some embodiments, the reference points may correspond to the locations of anatomical points known in the literature. The locations of some reference points may generally be known to those skilled in the art. However, since anatomical features vary from patient to patient, it may be important not to assume the locations of anatomical points. Some anatomical points may be easily discernible after visually inspecting the patient. However, some anatomical points may not be visible. For example, the general location of the condyles is known, but their actual location on the body of an individual patient may differ from the published literature, and the condyles may lack precise visual indications. Therefore, in some embodiments, an artificial intelligence or machine learning model may be used to identify reference points. For example, a labeled image indicating the location of the reference points may be used to train an AI / ML model, as described herein.

[0142] In some embodiments, reference points may be divided into primary points and secondary points. In some embodiments, primary points may indicate the location of the reference point of interest. However, in some cases, it may be difficult for an artificial intelligence or machine learning (AI / ML) model to identify the primary points. For example, if the primary points are covered or otherwise obscured by the patient's hair or facial hair, or if there is no clear visual indication, the AI / ML model may not identify the primary points or may incorrectly identify the primary points. Alternatively or in addition, other features of the patient, such as skin laxity, glasses, piercings, tattoos, cosmetic implants, etc., may make it difficult to identify reference points. In some embodiments, secondary points may be points that can be used to infer the location of one or more primary points. A non-exhaustive list of primary points and secondary points can be found, for example, in U.S. Pat. No. 12,265,149, the entire contents of which are incorporated herein by reference for all purposes.

[0143] In some embodiments, the primary points may include (for example but not limited to):

[0144] ·Infraorbital point

[0145] ·Condylar point

[0146] Pupil point

[0147] Alar point (wing of the nose)

[0148] ·Subnasal point

[0149] ·Submental point

[0150] ·Midpoint of hairline

[0151] ·Midpoint between eyebrows

[0152] Mandibular angle point

[0153] ·Nasal protrusion

[0154] Upper lip point

[0155] Lower lip point.

[0156] The locations of these and other points may generally be known in the literature. However, for dental treatment planning, it may be important to ensure that these points are accurately positioned for a particular patient.

[0157] Figure 7 is an example diagram showing the locations of various principal and secondary points on the face. The dotted line represents the median plane (eg, midsagittal plane). It should be understood that Figure 7 The points shown do not necessarily represent an exhaustive list of primary and secondary points. Figure 7 The general locations of the various primary and secondary points are shown, but it should be understood that the precise positioning of the primary and secondary points may vary to some extent from patient to patient.

[0158] In some embodiments, the infraorbital point can be determined in various ways. For example, the infraorbital point can be determined as the most oblique point of the lower border of the eye socket (e.g., the left or right eye socket) based on a 3D scan or 3D model of the face. In some embodiments, to construct the axial-orbital plane, the left or right infraorbital point can be used. In some embodiments, the average of the left and right infraorbital points can be used. Depending on the facial anatomy, it may be difficult to locate the infraorbital point. For example, a protruding cheek may obscure the infraorbital point. Therefore, in some embodiments, a secondary point can be used. For example, published studies have shown that the infraorbital point is typically located approximately 25.4 mm below the nasion. However, this is merely a typical value, and the actual location on the body of an individual patient may vary. In some embodiments, even when the primary points are obscured, the AI / ML model can be trained to identify those primary points, for example by considering the patient's overall facial structure and / or other points that can be determined (e.g., other primary points and / or secondary points). The use of secondary points is described in more detail below.

[0159] In some embodiments, the condylar point can be determined from the secondary point, as described in more detail below. In some embodiments, various definitions of the condylar point can be used. For example, published studies have shown that the Duminil condylar point can be 9.5 mm forward (from the tragus angle) and 7.5 mm vertically downward from a straight line formed between the lateral angle of the eye and the tragus angle. As mentioned above with respect to the infraorbital point, this is merely a typical value and may vary from patient to patient. In some embodiments, dynamic data can be used to locate the hinge axis. In some embodiments, the position of the hinge axis can be used to determine the condylar point position. The use of dynamic data is discussed in more detail below. In some embodiments, the AI / ML model can be configured to locate the condylar point in a manner similar to or identical to that described above with respect to the infraorbital point. Other points, such as Guichet's point, Gysi's point, and / or Monson's point, can be similarly determined.

[0160] Figure 8 is a diagram showing the location of Guichet's point, Gysi's point, Monson's point, and Duminil's point with reference to a line segment extending between the tragus and the medial canthus. It will be appreciated that these points can give an approximate location of the condyle. The precise location of the condyle can vary from patient to patient and can be more accurately determined, for example, using an AI / ML model trained to recognize features relevant to dental treatment and planning as described herein.

[0161] In some embodiments, the pupil point can be relatively easy to locate. For example, an AI / ML model can be trained to recognize the position of the eye, and the pupil point can be located at the center of the eye.

[0162] In some embodiments, the alar point (wing of the nose) can be the point where the ala meets the upper lip. There can be two alar points, one on the left and one on the right. In some embodiments, the AI / ML model can easily determine the location of the alar point, for example due to disturbances in the contour line. In some embodiments, color information, brightness information, etc. can be used to determine the location of the alar point. For example, the presence of a shadow can be used to determine the location of the alar point.

[0163] In some embodiments, the subnasal point may be located between the two nostrils at the base of the nose. In some embodiments, the subnasal point may be located at the highest and most distal point of the nasolabial notch. In some embodiments, the submental point may be the lowest point of the skin chin. In some embodiments, the midpoint of the hairline may be located at the front hairline. In some embodiments, the midpoint of the glabella may be located at the upper edge of the eyebrows. In some embodiments, the mandibular angle point may be located at the mandibular angle. In some embodiments, the nasal process point may be the tip of the nose (e.g., the most anterior point of the nose). In some embodiments, the upper lip point may be on the midsagittal line and at the intersection of the upper lip and the facial skin. In some embodiments, the lower lip point may be on the midsagittal line and at the boundary between the lower lip and the facial skin. In some embodiments, the AI / ML model may be trained to recognize some or all of these points. In some embodiments, the AI / ML model may be trained to recognize additional points, such as secondary points.

[0164] In some embodiments, an AI / ML model can be trained to identify various secondary points. For example, secondary points can be used to assist in determining the location of one or more primary points. Secondary points may include, for example but not limited to:

[0165] ·Outer corner of eye

[0166] ·Tragus point

[0167] Apex of the tragus angle

[0168] Skin nasal root point.

[0169] In some embodiments, the outer corner of the eye can be defined by the point where the outer ends of the upper eyelid and the lower eyelid meet. In some embodiments, the tragus can be the highest point of the tragus of the ear in the sagittal plane. In some embodiments, the skin nasion can be the saddle point of the nose. For example, in the sagittal plane, the skin nasion can be the farthest point of the frontonasal notch on the skin contour.

[0170] In some embodiments, orientation information can be used to assist in locating points. For example, if the patient's head is tilted rather than in a neutral position, the AI / ML model should preferably have this information during training and / or during deployment, or the information can be used in a pre-processing step to orient the facial scan or model. In some embodiments, sensor data collected during capture of the patient's face (e.g., accelerometer, magnetometer, and / or gyroscope data) can be used to determine the orientation of the facial scan or model relative to gravity or another reference axis.

[0171] As discussed above, in some embodiments, an AI / ML model may be used to determine a reference point. Fig. 9900 may be run on a computing system. At block 901, a system may receive a data set containing eigenfaces, Fisher faces, and the like. In some embodiments, some or all of the data set may be annotated to indicate various landmarks, primary points, secondary points, and the like. In some embodiments, additional information may be included, such as gender, age, sex, ethnicity, race, condition (e.g., malocclusion), and the like. Including such data during the training process may produce a trained model that may more accurately determine the patient's features by considering factors that may affect facial features. At block 902, one or more transformations may be performed on the data. For example, the data may need to be transformed to conform to an expected input format, such as converting an eigenface, Fisher face, and the like to a specific size, removing color, and the like. In some embodiments, the data may undergo a transformation to prepare it for use in training an AI or ML algorithm. For example, the categorical data may be encoded in a particular manner. The nominal data may be encoded using one-hot encoding, binary encoding, feature hashing, or other suitable encoding methods. Ordinal data can be encoded using ordinal encoding, polynomial encoding, Helmert encoding, etc. Numerical data can be normalized, for example, by scaling the data to a maximum value of 1 and a minimum value of 0 or -1. These are only examples, and skilled technicians will readily appreciate that other transformations are possible. At box 903, the system can create training, tuning, and test / validation data sets based on the received data set. Training data set 904 can be used to determine the features used to form a model during training. Tuning data set 905 can be used to select the final model (e.g., final weights) and prevent or correct overfitting that may occur during training with training data set 904, because the trained model should be generally applicable to a wide range of patients. After training and tuning, a test data set 906 can be used to evaluate the model. For example, test data set 906 can be used to check whether the model is overfitted to the training data set. In the training loop 914, the system can use the training data set 904 to train the model at box 907. Training can be performed in a supervised, unsupervised, or partially supervised manner. According to some embodiments, supervised training may be preferred because it may be necessary to identify reference points that are generally known to technicians in the field. At 908, the system can evaluate the model according to one or more evaluation criteria. For example, the evaluation can include determining how often the model correctly determines the location of primary points, secondary points, landmarks, etc. At 909, the system can determine whether the model meets the one or more evaluation criteria. If the model does not pass the evaluation, the system can tune the model using the tuning data set 905 at 910, repeating the training at box 907 and the evaluation at box 908 until the model passes the evaluation at 909. Once the model passes the evaluation at box 909, the system can exit the model training loop 914.The test data set 906 can be run through the trained model 911, and at box 912, the system can evaluate the results. If the evaluation does not pass at box 913, the system can re-enter the training loop 914 for additional training and tuning. If the model passes, the system can stop the training process, thereby generating the trained model 911. In some embodiments, the training process can be modified. For example, the system may not use the tuning data set 905. In some embodiments, the model may not use the test data set 906.

[0172] Although described with respect to detecting landmarks, primary points, secondary points, etc., it should be understood that the same or similar processes can be used to make other determinations. For example, the model can be trained to identify potential conditions such as malocclusion. For example, the model can be trained using a set of labeled images showing patients with different types of malocclusion or without malocclusion. For example, the images can be labeled according to an angular classification method, where Class I indicates normal molar alignment, Class II indicates an anteroposterior overbite or an upper and lower overbite, and Class III indicates an anteroposterior underbite or an upper and lower underbite.

[0173] Fig.10 An example process for training and using an AI / ML model according to some embodiments is shown. Fig.10 The process depicted in can be used for various purposes, such as identifying primary points, identifying secondary points, determining a condition that a patient may have (e.g., malocclusion), etc. The training data store 1002 can store data for training a model. For example, the training data store 1002 can store facial images, facial scan data, 3D models, low-dimensional representations (e.g., weights) of eigenfaces, Fisher faces, etc. In some embodiments, the training data can be annotated to include information for training a model. For example, images, facial scans, 3D models, etc. can be annotated to indicate the locations of landmarks, primary points, secondary points, etc. At box 1004, the system can be configured to prepare the training data if the training data has not been previously prepared for training a model. Preparing the training data can include performing one or more normalization procedures, standardization procedures, etc. For example, image data, facial scan data, 3D model data, etc. can be normalized to a specific size, color can be removed or normalized, orientation can be adjusted, etc. At box 1006, the system can extract features from the training data, and at box 1008, the model can be trained using the training data to produce a model 1010. At block 1012, the system may evaluate the model to determine whether it passes one or more criteria (e.g., success criteria for correctly identifying landmarks or points on the patient's face). At decision point 1014, if the model does not pass, the system may perform additional training. If the model passes at decision point 1014, the system may make available a trained model 1016, which may be model 1010 after training is complete.

[0174] The trained model 1016 can be used to evaluate a particular patient. The patient data 1018 can relate to a particular patient for whom the output of the training model 1016 (e.g., the location of landmarks, primary points, secondary points, etc.) is desired. At block 1020, the system can prepare the data, e.g., as described above with respect to stored training data. At block 1022, the system can extract features from the prepared user data. The system can be configured to feed the extracted features to the trained model 1016 to produce results 1024.

[0175] In some embodiments, patient data 1018 (and optionally, results 1024) may be stored for training. At box 1026, the system may prepare patient data 1018 and results 1024 for use in training. Preparing data may include, for example, anonymizing the data (e.g., if a name or patient ID is associated with the data). In some embodiments, the system may store the prepared data in a training data storage area 1002. In some embodiments, the prepared data may be stored in another database or data storage area in addition or alternatively. In some embodiments, the system may retrain the model periodically, continuously, or whenever an operator indicates to the system that the model should be retrained. Therefore, in some embodiments, the trained model 1016 may evolve over time, which may, for example, improve the determination made over time as the model is trained based on additional data. In some embodiments, patient data 1018 may not be stored in the training data storage area 1002. In some embodiments, once the model is trained, it may not be updated, or may only undergo occasional updates. For example, regulatory requirements may limit changes to the model.

[0176] As described above, extraction may include receiving facial scan data of a patient, generating a representation of the patient (e.g., an eigenface or Fisher face representation), and applying an AI / ML model to the representation to determine the locations of various reference points, landmarks, and the like. In some embodiments, alternative methods may be used. For example, in some embodiments, the system may be configured to determine the locations of reference points, for example, by directly processing facial scan data, without generating a low-dimensional representation of the patient. In some embodiments, the facial scan data may undergo some processing before being processed by the AI / ML model. For example, in some embodiments, the AI / ML model may be configured to use three-dimensional facial scan data. In some embodiments, the AI / ML model may be configured to use one or more two-dimensional projections of the patient's face. The AI / ML model may be trained and deployed in a manner similar to that described above, but instead of using an eigenface representation (or another low-dimensional representation) as training data and instead of using an eigenface representation (or another low-dimensional representation) of a patient for model deployment, facial scan data may be used for training and model deployment. Facial scan data may be labeled in a manner similar to or the same as that used to train a model that works using an eigenface representation.

[0177] In some embodiments, multiple models may be used. For example, different data may be used to train different models. For example, one model may be trained using frontal two-dimensional images, while another model may be trained using profile images. In some cases, a model may be trained using depth information, and another model may be trained using image information. These are merely examples, and the specific inputs to different models may vary to include different poses, different facial expressions, different types of data (e.g., image data and / or depth data, which may be two-dimensional or three-dimensional), etc.

[0178] In some embodiments, using multiple models can provide several benefits. For example, in some embodiments, the location of reference points can be determined more accurately. In some embodiments using multiple models, transfer learning can be used, which can enable the models to be trained together. For example, a model trained to determine reference points using a head on a facial image can be re-applied to another model to identify reference points in a contour image. For example, in the case of computer vision, the lower layers of an AI / ML model typically identify edges, shapes, etc. The higher layers of the AI / ML can identify specific reference points. Thus, for example, a layer that identifies edges and shapes can be re-applied while a higher layer can be trained to identify reference points in a contour image. For example, a similar approach can be applied to identify reference points using depth data. These are just examples, and as will be appreciated by those skilled in the art, transfer learning can be used in a variety of ways.

[0179] Combination stage

[0180] After determining the positions of the reference points, the system can be configured to combine the points in a way that is useful for dental treatment planning. For example, the reference points can be used to determine planes, lines, or other points that are useful in dental treatment planning.

[0181] In the combination stage, points can be associated with each other to describe an axis, a plane, a landmark, or other surfaces or features of interest. For example, two reference points can be associated with each other to form a line. As another example, three reference points can be associated with each other to form a plane. In some embodiments, more than three reference points can form a subset and can be associated with each other, and a surface having a shape passing through each reference point in the subset can be defined. In some embodiments, various restrictions can be imposed on the shape of the surface. For example, without any restrictions, an infinite number of surfaces passing through the reference points in the subset can be defined, but almost all of these surfaces are anatomically meaningless. Therefore, restrictions that limit the curvature of the surface, the slope of the surface, etc. can be imposed. In some embodiments, restrictions can vary according to the patient. For example, restrictions can ensure that the surface representing the patient's cheek will not appear concave. However, some patients may have concave cheeks, and restrictions can be modified accordingly. In some embodiments, restrictions can be automatically adjusted based on information about the patient (e.g., biological sex, gender identity, age, race, ethnicity, etc.).

[0182] In some embodiments, the reference point can be associated with other reference points or vectors. For example, the reference point can be associated with the gravity axis of the earth. In some embodiments, the reference point can be associated with a standard orthogonal reference system.

[0183] In some embodiments, reference points, landmarks, etc. may be used to generate a model of the patient's face. The model may include, for example, a mesh or point cloud representing the patient's face.

[0184] In some embodiments, the combined stage may determine the axial orbital plane, which is a plane passing through the condylar point and one of the left infraorbital point, the right infraorbital point, or the average position of the left and right infraorbital points. In some embodiments, the combined stage may determine the Camper plane, which is defined as a plane passing through the tragus and the subnasal point. In some embodiments, the combined stage may determine the tragus-alla plane.

[0185] In some embodiments, other reference planes may include a mid-sagittal plane passing through the nasion, the subnasal point, and a point equidistant from the condyle point. A reference axis may include a bi-condylar axis defined by a line passing through the left condyle point and the right condyle point. A reference axis may include a bi-pupillary axis defined by a line passing through the left pupil point and the right pupil point. In some embodiments, a lower facial contour may be used to generate one or more reference axes, contours, and the like. In some embodiments, a lower facial contour may be defined by a path passing through the tragus, the mandibular angle point, and the submental point. In some embodiments, an aesthetic line may connect the tip of the nose and the premental point.

[0186] Analysis Phase

[0187] During the analysis phase, various measurements relevant to the dental treatment plan may be determined. For example, the system may be configured to determine the mandibular angle (e.g., the angle formed by the junction of the posterior boundary of the mandibular ramus and the lower boundary of the mandibular body at the mandibular angle point). The mandibular angle may be calculated as the angle between the tragus-mandibular angle point line segment and the mandibular angle point-submental point line segment. In some cases, the mandibular angle may indicate a growth defect. In some cases, normalization of the mandibular angle may be a goal of surgical intervention and may inform the surgical procedure. As another example, in some embodiments, the system may determine the ratios of different phases of the face. For example, the system may be configured to determine the ratios between the midpoint of the hairline-midpoint of the glabella, the midpoint of the glabella-subnasal point, and / or the subnasal point-submental point. By studying these relationships between different phases of the face, the selection of new vertical distances for occlusion may be facilitated. The positioning of the dental arch may be adjusted to coordinate lower horizontal planes (e.g., the distance between the submental point and the subnasal point) with other horizontal planes.

[0188] In another example, the system can determine one or more ratios (e.g., aesthetic ratios) that can inform dental treatment planning. In some embodiments, the system can be configured to determine how far forward or backward the lip should be positioned. For example, the lip can be positioned based on the Ricketts line. The positioning of the dental arch can play an important role in the positioning of the lip. For example, the mandibular arch position can significantly affect how far forward or backward the lower lip is placed. Analysis of this aesthetic ratio (and / or other ratios) can be beneficial for dental treatment planning.

[0189] Fig.11 Examples of various ratios that may be relevant to dental treatment and planning are shown. For example, the distance from the top of the head to the midpoint of the eye may preferably be about the same as the distance from the midpoint of the eye to the submental point. As another example, the distance from the subnasal point to the midpoint of the lips may preferably be about the same as the distance from the midpoint of the lips to the midpoint between the midpoint of the lips and the submental point. As another example, the midpoint of the hairline-midpoint of the glabella distance may be about the same as the midpoint of the glabella-subnasal point distance and the subnasal point-submental point distance.

[0190] Once the feature points and planes have been found using the methods described herein, a single coordinate system can be created for the dental arch model, the facial capture system (e.g., a smartphone, tablet, laptop, etc.), and the feature points and planes. For example, the device used for facial capture (e.g., a mobile phone, tablet, camera, etc.) can have one or more sensors built in or attached thereto (e.g., accelerometers, gyroscopes, magnetometers, etc.) that can be used for orientation determination and creation of a single standard orthogonal coordinate system. The feature points and planes can have positions within a coordinate system defined using orientation data associated with the facial capture device. The maxillary and mandibular dental arches can be defined according to different second coordinate systems. After the dental arches are aligned with the facial scan data, the maxillary and mandibular dental arches can be defined in the same coordinate system as the feature points and planes.

[0191] In some embodiments, multiple coordinate systems may be used. For example, in some embodiments, the mandibular arch may have its own coordinate system. For example, a separate coordinate system for the mandibular arch may facilitate relatively simple transformations, rotations, etc. of the mandibular arch.

[0192] During the analysis phase, it may be important to have a well-defined model of the patient's face. For example, the model preferably has high resolution and represents the patient's entire face. If relevant points cannot be identified or are misidentified during the extraction phase, the analysis may be impaired. In some embodiments, the model may include reference points. In some embodiments, the model may be a point cloud or mesh generated based on reference points or using other methods as described herein (e.g., deformable masks).

[0193] In some embodiments, alternative approaches may be used instead of or in addition to the three-stage and two-stage approaches described above. For example, in some embodiments, a deformable mask (also referred to as a deformable model) may be used to determine static features of a patient.

[0194] The use of a deformable mask may be attractive for a variety of reasons. For example, a deformable mask may have well-defined reference points, equipment for manipulation, and the like. Advances in augmented reality enable the use of a deformable mask in real time or near real time. For example, a deformable mask may be easily overlaid onto a capture of a patient, either live or after facial capture. Augmented reality solutions may provide surface detection, image recognition, facial recognition, and the like, enabling the deformable mask to be easily applied to the patient's face. In some embodiments, an AR solution may include motion tracking, which may be important for dynamic feature determination, as described below. A deformable mask may be deformed to more closely represent the face of an actual patient. Since a deformable mask may have well-defined reference points, the reference point recognition step may be eliminated or significantly simplified. In some embodiments, predefined reference points of a deformable mask may be used to associate reference points, planes, landmarks, surfaces, and the like with features of a patient's face. In some embodiments, bone structures, dental models, and the like may be used to refine the deformable mask to define motion restrictions, and the like. In some embodiments, customized equipment specifically related to dental treatment planning may be developed. For example, for dental treatment planning, it may be important to have a high level of control over the positioning of teeth.

[0195] In some embodiments, a customized AR solution can be provided. In some embodiments, the AR solution can utilize existing libraries, applications, application programming interfaces (APIs), etc. For example (but not limited to), the AR solution can utilize Vuforia, Unity AR Foundation, Wikitude, EasyAR, ARToolKit, ARKit, ARCore, etc. Some of these solutions can work across platforms (e.g., on different operating systems, different central processing unit (CPU) architectures, etc.). In some embodiments, the AR solution can be specific to an operating system, CPU architecture, graphics processing unit (GPU) architecture, etc. Visualization can be performed using various technologies such as, but not limited to, OpenGL, OpenGLES, Metal, WebGL, Vulkan, etc.

[0196] In some embodiments, the facial scan data can be used to directly generate a 3D model of the patient's face. In some embodiments, the model can be generated based on, for example, the determined reference points as described above. Such a model can be referred to as a primary model. For example, the primary model can be constructed based on depth information associated with the facial scan. In some embodiments, the texture of the model can be determined from the captured facial image, the facial scan, or both. Preferably, the facial image data is captured simultaneously with the depth data, although this is not strictly necessary.

[0197] In some embodiments, the primary model may be a point cloud or mesh representing the patient's face, and may or may not contain texture. However, the primary facial model may be missing some information. For example, limitations of the capture technology may result in missing data, erroneous data, etc. In some cases, hair or other obstructions may cause errors in the primary model. Therefore, it may be desirable not to use the primary model for further analysis. However, the primary model may be detailed enough so that it can be used to generate a secondary model. The secondary model may be, for example, a deformable model or mask. A deformable model or mask may be a mathematical model containing one or more controllable parameters. In some embodiments, the primary model may not be used. As described above, AR technology can be used to deform a deformable mask to fit the patient's face, which can avoid any need to create a primary model separately. In some embodiments, an augmented reality solution can be used to superimpose feature points and planes on a deformable model.

[0198] In some embodiments, even if there are no obvious problems with the primary model, it may be preferable to use a secondary model. For example, the secondary model may be standardized at least to some extent and may have well-defined reference points, landmarks, manipulations, etc. The software may be optimized to work with the secondary model. Thus, in some cases, the secondary model may be manipulated with fewer errors and more realistic manipulations than the primary model.

[0199] In some embodiments, the secondary model can be defined at least in part by the representation or location of points of interest (e.g., landmarks, primary points, secondary points, etc.). In some embodiments, one or more transformations can be applied to the secondary model, for example, to make the secondary model more closely represent the patient's face. Such transformations can include, for example, adjusting the position of the ears, eyes, nose, lips, etc. In some embodiments, the jaw can be moved back or forward and / or left or right. In some embodiments, cheek roundness, nose width, nose shape, etc. can be adjusted.

[0200] In some embodiments, in a first transformation, the secondary model can be superimposed onto the primary model or onto the patient's face. In some embodiments, the secondary model can be superimposed onto the patient's face (e.g., a facial capture or a live view of the patient). In some embodiments, the fit may not be exact. In some embodiments, the secondary model can be deformed to minimize the difference between the primary model and the secondary model (or between the patient's face and the secondary model), for example by minimizing the mean squared error or the mean absolute error. For example, if outliers are very adverse, the mean squared error can be used because it is more sensitive to outliers, while if the severity of the outliers is roughly linear with the absolute error, the mean absolute error may be a better metric.

[0201] Various methods may be used to deform the secondary model. Deforming the secondary model may be performed using, for example, but not limited to, mesh interpolation, skeletal animation, cage deformation, or any combination of these methods. Such methods are known to those skilled in the art. Mesh interpolation methods, also known as linear mesh deformation, may include creating a series of intermediate shapes between an initial mesh and a target mesh. The intermediate shapes may be obtained by linearly interpolating vertex coordinates between the initial mesh and the target mesh. This method enables the secondary model to be transformed in a smooth manner and without large deformations. For example, mesh interpolation may use linear transformations, such as enlargement, reduction, and / or rotation.

[0202] Skeleton animation methods can use a skeleton or rig to deform the mesh in a controlled manner. The bones of the skeleton can be associated with vertices or vertex groups of the mesh, and the displacement of the bones may cause deformation of the associated vertices. In some embodiments, skeletal animation methods may be preferred for accurately modeling the movement of the patient's face.

[0203] With the cage deformation method, the initial mesh can be "wrapped" in a deformable cage. Deforming the cage can produce a modification of the shape of the initial mesh. For example, as the vertices of the cage move, the initial mesh can deform accordingly. This method can be particularly attractive when more drastic transformations are required.

[0204] In some embodiments, the deformable model or mask may include or have associated therewith one or more points on a grid that may be associated with a reference point for an augmented reality application. In some embodiments, the grid may be animated. For example, the grid may be overlaid onto a patient's face and may deform to match the patient's facial movements, expressions, or both.

[0205] In some embodiments, various data processing steps may be performed. For example, a dental arch may be matched to a primary face or a secondary face as described herein. In some embodiments, a secondary model may be matched to a primary model or a facial scan. In some embodiments, a dental arch may be co-registered with a secondary model. In some embodiments, the system may be configured to orient the deformable model using gravity. In some embodiments, the system may determine the glabella, the furthest portion located above the nose and at the beginning of the forehead. In some embodiments, after determining the glabella, the infraorbital point may be found. After locating the infraorbital point, the system may locate the condyle point. This process may be repeated for the left and right sides of the patient's face. In some embodiments, data regarding the movement of the patient's jaw may be used to refine the location of the condyle point, as described below with reference to dynamic features.

[0206] Any of the methods described above can be performed with or without the inclusion of the patient's teeth (e.g., using a facial model or orofacial model). For example, determining reference points, lines, and planes can be performed without having any knowledge of the patient's teeth. However, co-registering the patient's teeth (e.g., a dental model) and the patient's face can have many benefits. For example, the positioning of the patient's teeth may significantly affect the positioning of the patient's lips. As another example, the patient's teeth may limit the movement of the patient's jaw (e.g., once the patient's maxillary and mandibular teeth collide, the jaw cannot continue to close).

[0207] In the subsequent figures, various examples for generating facial representations, training AI / ML models, and analyzing facial captures, dental models, or both (e.g., orofacial models) are shown. It should be understood that these are merely examples. Not all steps are required, and the steps may be performed in an order different from that depicted in the subsequent figures. In some embodiments, the process described below may be modified to include additional steps, such as orienting and / or co-registering facial scans, facial models, dental models, etc.

[0208] The following description and accompanying drawings refer to images. However, it should be understood that the processes described below can be applied to various types of data. For example, unless the context clearly dictates otherwise, the following processes can be applied to facial scans (e.g., 3D facial scans), facial models (e.g., 3D facial models), two-dimensional images, depth data, etc.

[0209] Fig.121 is a flowchart illustrating an example process for generating a low-dimensional representation according to some embodiments. The low-dimensional representation may include, for example, a feature face, a Fisher face, etc. At box 1202, a computer system may receive a set of images for generating a low-dimensional representation. In some embodiments, the images may be standardized (e.g., all in color or all in grayscale, all with individuals depicted from the same distance and positioned at the same position within the frame, etc.). In some embodiments, the images may not be standardized, and processing (e.g., resizing, rotating, de-skewing, cropping, color normalization, etc.) may be performed before implementing additional steps. At box 1204, the system may detect features that are less important for dental treatment planning (e.g., eyebrows, portions of the outer ear other than the tragus, etc.). At box 1206, the system may remove a subset of facial features that are less important for dental treatment planning. For example, the system may blur features, place solid colors on top of features, etc. In some embodiments, blurred areas, solid color areas, etc. may be standardized in size, position, etc., so that such modifications do not vary from image to image. At box 1208, the system may perform principal component analysis on the modified image. At block 1210, the system may generate a set of eigenfaces (or the like). The generated set of eigenfaces may form a base set that may be used to represent the patient. In some embodiments, linear discriminant analysis may be used, for example, when using Fisher faces (or a similar method). At block 1212, the system may store the eigenfaces (or the like) for future use, for example, when determining the characteristics of a particular patient.

[0210] In some embodiments, Fig.12 The illustrated process may be performed on a single set of images, which may include individuals of different ages, genders, sexes, ethnicities, races, etc. In some embodiments, Fig.12The process shown can be performed multiple times to generate multiple sets of eigenfaces (or the like). For example, the process can be performed for different groups of people who tend to share common features. For example, it may be beneficial to have different sets of eigenfaces for men and women, different sets of eigenfaces for children, teenagers and adults, and so on. In these cases, when performing analysis on a particular patient, a set of eigenfaces (or the like) can be selected based on the patient's specific features. As discussed above, in some embodiments, the image set may include 2D images, such as 2D projections of facial scan data. In some embodiments, the 2D image may be from a specific angle (e.g., head-on, profile, etc.). In some embodiments, the 2D image may be an "unfolded" image determined, for example, from a facial capture. For example, a 2D image may be prepared by performing a projection operation similar to a projection operation used to make a map of the earth. In some embodiments, instead of a two-dimensional image, a facial capture (which may include, for example, three dimensions) may be used to perform the process. In some embodiments, a full facial capture (e.g., capturing the entire head) may be used, while in other embodiments, only a partial capture (e.g., from the left ear to the right ear) may be used.

[0211] Fig.13 is a flow chart illustrating an example process according to some embodiments. Fig.13 The process shown uses eigenfaces (or similar methods) to represent patients. Fig.13 The process shown can be run on a computer system. At box 1302, the system can receive a facial scan of a patient. In some embodiments, the system can perform additional steps directly using the facial scan. In some embodiments, as described above, the system can generate one or more two-dimensional projections, and can use the two-dimensional projections to perform additional steps. In some embodiments, multiple two-dimensional projections can be used, as described in more detail below. At box 1304, the system can determine a low-dimensional representation, such as an eigenface representation, a Fisher face representation, etc. At box 1306, the system can determine reference points, such as using an AI / ML model configured to determine reference points from a low-dimensional representation of the patient's face. At box 1308, the system can determine any combination of lines, planes, surfaces, or additional points (e.g., points not determined at box 1306). Lines, planes, surfaces, or additional points can be determined based on, for example, reference points determined at box 1306. At box 1310, the system can perform facial analysis. For example, the system can determine one or more ratios, as described in more detail herein. In some embodiments, at block 1310, the system may determine a condition affecting the patient, such as a malocclusion.

[0212] Although described above with reference to a single low-dimensional representation, it should be understood that in some embodiments, multiple low-dimensional representations may be used. For example, blocks 1304, 1306, and 1308 may be performed multiple times, such as once for each received image or for multiple views generated from facial capture. In some embodiments, the steps performed at blocks 1306, 1308, and 1310 may take into account different low-dimensional representations. For example, at block 1306, different AI / ML models trained to use different low-dimensional representations (e.g., one using a front view and one using a cross-sectional view) to identify features may provide different results for the location of the feature. In some embodiments, the system may incorporate feedback or collaboration mechanisms to reach consensus between models. In some embodiments, an average position (e.g., an average of the positions determined by different models of a particular reference point) may be used. In some embodiments, if several models (e.g., more than two) are used, the position of the reference point may be governed by consensus. For example, once a defined number of models agree on the position of the reference point, the reference point may be identified. For example, consensus may be reached when a defined number of models agree on the location of a reference point within defined limits (e.g., within 0.1 mm, within 0.5 mm, within 1 mm, within 2 mm, within 3 mm, etc.) In some embodiments, limits may vary according to a particular reference point, e.g., based on the importance of the reference point to dental treatment planning, the difficulty of locating the reference point, etc.

[0213] Similar techniques can be applied to other embodiments of the present invention. For example, similar or identical techniques can be applied to Fig.14 The process described in .

[0214] Fig.14 is a flow chart illustrating an example process according to some embodiments. Fig.13 Unlike the process depicted in , a low-dimensional representation of the patient's face may not be generated. At box 1402, the system may receive a facial scan. In some embodiments, the facial scan may be used directly. In other embodiments, a two-dimensional projection as described above may be generated, and the two-dimensional projection may be used to perform additional steps. In some embodiments, the system may generate a simplified facial model (e.g., a 3D mesh or point cloud) generated based on the facial scan data. At box 1404, the system may determine reference points, for example using an AI / ML model. At box 1406, the system may determine any of a line, plane, surface, or additional points, for example based on the reference points determined at box 1404. At box 1408, the system may perform facial analysis as described above.

[0215] Fig.15A is a flow chart illustrating an example process according to some embodiments. Fig.15AThe process depicted in utilizes a deformable mask. At block 1502, the system may receive a facial scan of a patient. At block 1504, the system may determine one or more reference points based on the facial scan. At block 1506, the system may deform the deformable mask relative to the face. The locations of the reference points may be used to assist in positioning and / or deforming the mask relative to the patient's face. This approach may be attractive because, for example, it may help ensure that the points of the mask that are most relevant to the dental treatment plan are accurately mapped to the patient's face. However, as described below with respect to Fig. 15B As described herein, this method may not be used, and a simpler process may be used instead. At block 1508, the system may use the deformable mask to determine various landmarks, planes, lines, surfaces, additional points, etc. At block 1510, the system may perform facial analysis (e.g., lip position relative to Ricketts lines, ratios, etc., as described herein).

[0216] Fig. 15B is a flow chart illustrating another example process according to some embodiments. At block 1520, the system may receive a facial scan. At block 1522, the system may deform a deformable mask relative to the patient's face (e.g., based on the facial scan). Advantageously, Fig.15A In contrast, this can be accomplished without generating a separate facial model, determining reference points, and the like. In fact, surface detection, facial recognition, feature recognition, and the like are provided in some AR solutions. At box 1524, the system can extract reference points from the deformable mask. As discussed above, the deformable mask can have well-defined reference points. Thus, in some embodiments, reference points can be easily extracted without the need for additional detection or determination processes. At box 1526, the system can extract and / or calculate landmarks, planes, lines, surfaces, and / or additional reference points. Advantageously, such extraction and / or calculation can be based on information already contained in the deformable mask. At box 1528, the system can perform facial analysis as described herein (e.g., lip position relative to Ricketts lines, ratios, etc., as described herein).

[0217] Figure 13-15B A process is described that utilizes facial scan data but does not utilize dental model data. As discussed above, the inclusion of dental model data may be beneficial for various reasons. For example, if dental model data is included, it may impose limitations on the deformable model, such as to avoid collisions that may occur if the jaw is allowed to close too much. Inclusion of teeth may be important when developing a dental treatment plan. For example, while the above method may reveal a dental condition or facial imbalance without information about the teeth, it may be difficult for a practitioner to develop a treatment plan.

[0218] Dynamic Features

[0219] While static information can provide useful insights, it may be important to also consider the patient's dynamic movements (e.g., movement of the patient's jaws, teeth, and / or face). Dynamic features may be important for both functionality and aesthetics. For example, the positioning of the patient's teeth may be optimized to achieve a functional effect (e.g., chewing without causing excessive wear, biting without excessive difficulty) and / or an aesthetic effect (e.g., appearance when smiling).

[0220] In some embodiments, the patient's movements can be recorded directly, for example, using a camera or other motion capture device and one or more markers placed on the patient's body (e.g., markers on the patient's forehead, chin, etc.). However, as discussed above, the use of markers or dedicated equipment may be challenging due to the difficulty of using such equipment, the cost of obtaining such equipment, etc. In some embodiments, markers may not be used. In some embodiments, the movement may include movement of the patient's face. In some embodiments, the movement may include movement of the patient's jaw. In some embodiments, the movement may include movement of the patient's teeth. In some embodiments, motion capture may be performed using a smartphone or other device equipped with depth sensing technology. In some embodiments, a series of still images (with or without depth information) may be used to determine the dynamic movement of the patient's skeletal features, skin, etc. In some embodiments, a video (with or without depth information) may be used to determine dynamic movement. As described above, in some embodiments, a deformable mask may be used for dynamic analysis.

[0221] In some embodiments, a large amount of movement may be represented in a facial capture. In some embodiments, various methods may be used to improve tracking of jaw movement without markers, such as by training an AI / ML model to detect movement. Motion detection using an AI / ML model will be readily understood by those skilled in the art. In some embodiments, an AI / ML model may be trained to select the most relevant points of a deformable model for animating a dental arch model, mandible, etc. The appropriate selection may depend on a variety of factors, such as patient morphology, skin laxity, age, biological sex, social gender, ethnicity, race, weight, etc. In some embodiments, only a few factors may be considered. In some embodiments, such considerations may not be taken into account and the model may instead represent a general or average animation, but it may be beneficial for the model to be able to accurately represent a specific patient.

[0222] In some embodiments, the motion tracking model can be an AI / ML model. In some embodiments, the motion tracking model can be trained to recognize the movement of the patient's jaw, the movement of the patient's lips, etc. Generally, this model can be trained in a similar manner as described above. In some embodiments, the model can be configured for time series analysis.

[0223] In some embodiments, the training data set used to train the motion tracking model may include reference movements obtained from motion capture using optical, photoelectric, accelerometer markers (fixed or movable), etc. In some embodiments, the markers can be fixed on the maxillary arch, the mandibular arch, or both. In some embodiments, tracking can be improved based on multiple alignments with real markers. This method can enable selection of the most useful parts of a deformable mask, point cloud, mesh, etc. for animation. This method can achieve accurate jaw tracking of actual patients without the use of facial motion-based markers. For example, reference movements can be used to initially train an AI / ML model. Transfer learning can be used to train a model to track jaw motion without the use of markers. For example, weights determined using markers can serve as a starting point for the weights of a model that does not rely on markers.

[0224] In some embodiments, the motion tracking model may not be trained using motion capture data as described above. For example, the motion tracking model may be trained using captured video, captured image frames, etc., without using markers.

[0225] In some embodiments, the motion tracking model can be trained using labeled data (eg, in a supervised manner). In some embodiments, other training methods can be used, such as partially supervised, unsupervised, etc.

[0226] In some embodiments, the AI / ML model (e.g., as described above) can use the facial capture data and the dynamic data to determine the best deformation or manipulation to apply to the secondary model. For example, if a deformable mask is used to represent the patient, the movement of the deformable mask can be selected to more closely match the actual movement of the patient.

[0227] Orofacial visualization and manipulation

[0228] As described above, in some cases it may be important to consider the face and teeth together. For example, while facial analysis may provide information that aids in dental treatment planning, it may be difficult to determine how to manipulate the teeth to achieve desired facial features if the teeth are not adequately considered. For example, a static feature determination process as described above may reveal that the lips are too far forward or too far backward from the Ricketts line (the line extending from the tip of the nose to the tip of the chin), but this information alone may not be sufficient to develop a treatment plan.

[0229] As discussed above, in some embodiments, the AI / ML model can be trained to identify problems and recommend dental treatments. In some embodiments, such identification and recommendations can be based solely on facial analysis. In some embodiments, the AI / ML model can be trained using an orofacial model (e.g., a model representing both the teeth and the patient's face, such as by combining facial scan data, a deformable mask, etc., and a dental model (or multiple dental models) of the patient).

[0230] Although the orofacial model can be analyzed by the AI / ML model to at least partially determine the dental treatment plan, it may be important for the practitioner to be able to visualize the patient and the results of the dental treatment plan. In some embodiments, the practitioner may follow a recommended dental treatment plan created by a computer system using the AI / ML model. In some embodiments, the practitioner may wish to change the dental treatment plan or develop their own dental treatment plan. Therefore, it may be important to provide systems and methods for visualizing orofacial models. In some embodiments, the systems and methods herein can enable manipulation of the orofacial model, for example, by moving teeth, extending or reducing the mandible, and the like.

[0231] The practitioner may conclude that movement of the teeth is required to improve the appearance. If the dental model (or models) are co-registered with the facial model, the practitioner can better determine how to adjust the teeth to achieve the desired result.

[0232] As discussed above, there are various methods for generating a facial model. For example, the facial model can be a deformable mask generated at least in part based on facial scan data. As another example, the facial model can be a mesh or point cloud model based on the facial scan data. In some embodiments, the facial scan data itself can be used directly as a model. By applying the static and dynamic analysis methods described herein, a model can be created.

[0233] Although the above disclosure can be used to generate a model of a patient's face that accurately reflects the patient's structure and movement, it may be important that the teeth are correctly oriented in the model. As described above, dental models can be created using intraoral scans, x-ray technology, dental impressions or molds, etc.

[0234] In some embodiments, the placement of the dental model relative to the facial model (e.g., primary or secondary model) can be based on measurements performed by a practitioner, for example, using the techniques described in U.S. Pat. No. 10,265,149. For example, a tracker can be attached to the patient and the rod, and information about specific points on the patient's teeth can be collected. However, dedicated equipment can be used in this method.

[0235] In some embodiments, automatic registration may be performed. For example, if a profile x-ray of the patient or a 3D x-ray scan of the patient is available, it may be used to register the facial model and the dental model as described above.

[0236] The co-registered facial model and dental model may be presented to the practitioner for viewing and / or manipulation in a user interface.

[0237] Over the past few years, augmented reality (AR) and virtual reality (VR) solutions have been developed. These solutions may be available in the form of applications, libraries, application programming interfaces (APIs), etc. In some cases, AR solutions may utilize motion tracking, surface detection, image recognition, or any combination of these and other features.

[0238] AR solutions may include, for example, but not limited to, Vuforia, Unity AR Foundation, Wikitude, EasyAR, ARToolKit, ARKit, and ARCore. Some solutions may work across platforms, such as on computers, smartphones, tablets, head-mounted devices, etc. running various operating systems (e.g., Windows, macOS, Linux, Android, iOS, and iPadOS, etc.), while other solutions may be designed to operate on a limited number of devices or operating systems.

[0239] Augmented reality solutions can provide many advantages for dental treatment planning. For example, surface detection and image and facial recognition can be combined with real-time rendering, which can enable the association of feature points and planes with facial features. In some embodiments, a three-dimensional model (or multiple models) of the dental arch can be superimposed on the face. In some embodiments, the 3D model of the dental arch can be placed in the same standard orthogonal reference system as the face. In some embodiments, the placement of the dental arch model can be used to provide reference points for feature points and planes.

[0240] In some embodiments, the three-dimensional rendering may include the patient's face, teeth, and feature points and planes. In some embodiments, such three-dimensional rendering may be generated using 3D APIs such as, but not limited to, OpenGL, OpenGL ES, Metal, WebGL, Vulkan, etc.

[0241] In some embodiments, the oral-facial model may include a deformable model. There are several techniques that can be used to manipulate the deformable model. In some embodiments, blendshapes may be used. In some embodiments, geometric deformations may be used.

[0242] Blend Shapes (also called Morph Targets) can be deformable 3D models that have been pre-built to represent different facial expressions, such as smiling, frowning, blinking, etc. In some embodiments, developers can use these models to create facial animations by combining different blend shapes. In some embodiments, this approach can achieve real-time or near real-time animation. For example, the model can be deformed in real-time or near real-time to provide a live animation of the patient's facial movements.

[0243] In some embodiments, geometric deformation can be used. Geometric deformation is a deformation technique that changes the geometry of a 3D model. In some embodiments, geometric deformation can be performed in real time or near real time. Geometric deformation uses a deformable 3D mesh that can be adjusted according to the shape of the user's face. The 3D mesh can be divided into triangles, and each triangle can be deformed to follow the patient's facial movements.

[0244] Whether using blend shapes or geometric deformations, it may be important to find nodes of the deformable mask that will achieve accurate animation of the dental arch, face, etc. For example, it may be important that the mandible moves in a manner consistent with the patient's actual jaw movements. In some embodiments, the most relevant nodes may be located primarily at the horizontal plane of the chin and / or the lower and / or lateral edges of the mandible.

[0245] In some embodiments, various steps can be taken to ensure more realistic motion. For example, when the patient's mouth is closed, the deformable model can be adjusted to more closely represent the patient. In some embodiments, blend deformations can present specific challenges because they are typically designed and calibrated based on facial expressions (e.g., smiling, frowning, surprise, etc.) rather than based on jaw position, dental contacts, etc., which may be more important for developing dental treatment plans. In some embodiments, the system can determine the position of the face (e.g., the position of the jaw) when the teeth are in an occluded position (e.g., when the mandibular teeth and the maxillary teeth touch). This position can be different from the patient's resting position. For example, in a resting or intermediate position, the teeth are typically not in contact and the mandible is slightly lower.

[0246] Reproducing dental contacts can be important for dental treatment planning because poor alignment of teeth when the mouth is fully closed can lead to significant problems such as premature wear, lack of contact between teeth, etc. For example, the rear mandibular molars and maxillary molars may be in contact, while the more anterior mandibular molars and maxillary molars may not be in contact.

[0247] It may be important to co-register the dental model and the facial model (which together may form the orofacial model) with high accuracy, since even small deviations from the patient's true anatomy may have a significant impact on the dental contacts.

[0248] In some embodiments, pre-made blend shapes can be used. For example, a third party can provide blend shapes to reflect facial expressions. In some embodiments, custom blend shapes can be used for dental treatment planning. For example, custom blend shapes can be created for low amplitude movements (such as lateral movement and thrust).

[0249] In some embodiments, the system may be configured to provide collision detection. The collision detection algorithm may ensure that mesh penetration does not occur, or that mesh penetration only occurs in a clinically acceptable manner. For example, teeth, which are typically rigid and non-deformable, cannot penetrate each other. However, soft tissue may deform, so a certain amount of mesh penetration may be allowed.

[0250] In some embodiments, the animation of the oral-facial model may not be smooth. For example, there may be jumps or other defects in the animation. In some embodiments, smoothing techniques can be applied to ensure smooth motion.

[0251] In some embodiments, the user interface can display a distance map. The distance map can show, for example, the proximity between different grids. For example, the distance map can show the proximity of mandibular teeth to maxillary teeth. A method for determining tooth contact is discussed in U.S. Pat. No. 10,582,992.

[0252] In some embodiments, condylar points attached to the mandibular arch can be used to draw lines, arcs, etc. and / or indicate values ​​such as condylar slope, Bennett's angle, etc. In some embodiments, the condylar slope can be analyzed based at least in part on the protrusive movement. In some embodiments, the left Bennett's angle can be determined based at least in part on the right side displacement. In some embodiments, the right Bennett's angle can be determined based at least in part on the left side displacement. These are merely examples, and a skilled technician will readily appreciate that analysis of dynamic features can be used to determine many parameters that can be used for dental treatment planning.

[0253] In some embodiments, the user interface can enable the orofacial model to deform over time. For example, this animation can depict the patient in its original state (e.g., before treatment). In some embodiments, the animation can show the final result. In some embodiments, the animation can show one or more intermediate results, such as how the patient will appear one month after starting treatment, three months after starting treatment, six months after starting treatment, etc.

[0254] exist Fig.16 and 17 In some embodiments, the interface may display points that differentiate between different regions and that may be used for model kinematics. For example, some points may be associated with the chin, while other points may be associated with the lip contour. In some embodiments, the software may be configured to automatically identify the points. In some embodiments, the points may be placed manually, or the user may refine or correct the automatic placement of the recommended points.

[0255] exist Fig.16 , the user interface 1600 has a main view 1602, a teeth view 1604, a drawing view 1606, buttons 1608, adjustment coefficients 1610, and a drop-down menu 1612. Button 1608 can be used to change the display. For example, the reset button can change the view to its initial position, and the axis button can be used to toggle the display of the axes in the main view of the user interface. The teeth button can be used to enable and disable the display of the teeth in the main view, and the landmark button can be used to enable and disable the display of landmarks in the main view 1602, such as showing or hiding vertices, upper lip landmarks, etc. used in calculating the mandibular position (determined at least in part based on the chinDist coefficients discussed below). The plane button can be used to toggle the display of the two main planes in the main view 1602. The condyle button can be used to enable or disable the display of the condyle in the main view 1602. The wireframe button can be used to switch between a wireframe view and other views, such as displaying a captured patient's face.

[0256] In some embodiments, the drawing view can be configured to display various drawings. For example, the drawing view can be used to show the movement of the left condyle, the right condyle, the incisal point, etc. In some embodiments, the drawing can be displayed from various perspectives (e.g., front view or sagittal view).

[0257] It should be appreciated that the user interface may have more or fewer features, and that features may be implemented in different manners (e.g., check boxes may be used to enable or disable display of individual components such as axes, planes, condyles, etc.), while still achieving substantially the same purpose.

[0258] exist Fig.17 , user interface 1700 can allow a user to perform various actions that may be beneficial in fitting the patient's teeth to the patient's face. For example, a user can use a deformation target list 1702 to apply various transformations to the patient's face. For example, a deformation target can be used to open and close the jaw, move the jaw forward or backward, or move the jaw left or right. The user can adjust the morphCoeff value within the adjustment coefficient 1610 to determine the intensity of the deformation (e.g., how far to open the jaw or how far to move the jaw forward). In addition, the user can adjust the zCoeff, scaleCoeff, and chinDist variables within the adjustment coefficient 1610 to control the position of the teeth relative to the face. For example, zCoeff can be manipulated to change the distance of the teeth from the lips, scaleCoeff can be used to control the size of the teeth relative to the patient's face, and chinDist can be used to change how the mandible moves with the mouth (e.g., by changing which vertices are used to calculate the movement of the teeth).

[0259] The deformation target list 1702 may also be used, for example, to cause the patient to smile or exhibit another facial expression, which may be helpful in determining the optimal placement of the patient's teeth.

[0260] The facial and dental capture data may further be used as input as part of the process of designing the patient's smile by changing the alignment and positioning of the patient's teeth or artificial teeth.

[0261] In some embodiments, the user interface can be configured to allow the practitioner to adjust the positioning of the teeth within the patient's mouth. For example, the practitioner can adjust the maxillary teeth, the mandibular teeth, all teeth, a subset of teeth, or individual teeth. In some embodiments, the practitioner can adjust the patient's posture, facial expression, etc. (e.g., a model representing the patient) to evaluate the placement of the teeth.

[0262] In some embodiments, the user interface may be configured to enable a user to record and / or replay movements. In some embodiments, the user interface may be used to display the contact relationship or occlusal surface between the mandibular teeth and the maxillary teeth. In some embodiments, color coding may be used to indicate the distance between the maxillary teeth and the mandibular teeth. In some embodiments, the user interface may enable visualization and / or generation of an occlusal reference sphere (e.g., a Monson sphere) and / or a Monson curve. In some embodiments, the user interface may enable a user to display and / or change a function generating surface. The function generating surface may indicate the envelope of a function defined by the dental arch movement.

[0263] In some embodiments, the user interface can be configured to enable the user to view and / or adjust various aesthetic parameters. For example, in some embodiments, the user can adjust the vertical distance of the bite. In some embodiments, such adjustments may be limited by the patient's known bone structure, the positioning of the teeth, etc. In some embodiments, the user can manipulate the positioning of the teeth. In some embodiments, the user can manipulate the bone structure, such as shortening or lengthening the mandible in the case where the patient suffers from an anteroposterior underbite or an anteroposterior overbite.

[0264] Computer Systems

[0265] Fig.18 is a block diagram depicting an embodiment of a computer hardware system configured to run software for implementing one or more embodiments disclosed herein. References to computing system 1820 may also refer to portable device 1815 unless the context clearly dictates otherwise.

[0266] In some embodiments, the systems, processes, and methods described herein are implemented using a computing system, such as Fig.18The example computer system 1802 communicates with one or more computing systems 1820 and / or one or more data sources 1822 via one or more networks 1818. Fig.18 An embodiment of a computing system 1802 is shown, but it should be appreciated that the functionality provided in the components and modules of the computer system 1802 may be combined into fewer components and modules, or further divided into additional components and modules.

[0267] The computer system 1802 may include a module 1814 that performs the functions, methods, acts and / or processes described herein. The module 1814 is executed on the computer system 1802 by the central processing unit 1806 discussed further below.

[0268] Generally speaking, the term "module" as used herein refers to logic embodied in hardware or firmware, or a collection of software instructions with entry and exit points. Modules are written in programming languages, such as JAVA, C or C++, Python or the like. Software modules can be compiled or linked into executable programs, installed in dynamic link libraries, or can be written in interpreted languages ​​such as BASIC, PERL, LUA or Python. Software modules can be called from other modules or from themselves, and / or can be called in response to detected events or interrupts. Modules implemented in hardware include connected logical units, such as gates and triggers, and / or can include programmable units, such as programmable gate arrays or processors.

[0269] Typically, the modules described herein refer to logical modules that can be combined with other modules or divided into submodules regardless of the physical organization or storage of the logical modules. The modules are executed by one or more computing systems and can be stored on or in any suitable computer-readable medium, or implemented in whole or in part in specially designed hardware or firmware. Although any of the methods, calculations, processes, or analyses described above can be facilitated by the use of a computer, not all calculations, analyses, and / or optimizations require the use of a computer system. In addition, in some embodiments, the process blocks described herein may be altered, rearranged, combined, and / or omitted.

[0270] The computer system 1802 includes one or more processing units (CPU) 1806, which may include a microprocessor. The computer system 1802 further includes a physical memory 1810, such as a random access memory (RAM) for temporary storage of information, a read-only memory (ROM) for permanent storage of information, and a mass storage device 1804, such as a backup memory, a hard drive, a rotating disk, a solid-state disk (SSD), a flash memory, a phase change memory (PCM), a 3D XPoint memory, a disk, or an optical media storage device. Alternatively, the mass storage device may be implemented in a server array. Typically, the components of the computer system 1802 are connected to the computer using a standard-based bus system. The bus system can be implemented using various protocols, such as peripheral component interconnect (PCI), micro channel (Micro Channel), SCSI, industry standard architecture (ISA) and extended ISA (EISA) architecture.

[0271] The computer system 1802 includes one or more input / output (I / O) devices and interfaces 1812, such as a keyboard, a mouse, a touchpad, and a printer. The I / O devices and interfaces 1812 may include one or more display devices, such as a monitor, that allow data to be visually presented to a user. More specifically, the display device provides, for example, a GUI as a presentation of application software data as well as multimedia presentation. The I / O devices and interfaces 1812 may also provide a communication interface to various external devices. For example, the computer system 1802 may include one or more multimedia devices 1808, such as a speaker, a graphics card, a graphics accelerator, and a microphone.

[0272] The computer system 1802 may be run on a variety of computing devices, such as servers, Windows servers, structured query language servers, Unix servers, personal computers, laptops, etc. In other embodiments, the computer system 1802 may be run on a cluster computer system, a large mainframe computer system, and / or other computing systems suitable for controlling and / or communicating with large databases, performing large-volume transaction processing, and generating reports based on large databases. The computing system 1802 is typically controlled and coordinated by operating system software, such as Windows XP, Windows Vista, Windows 7, Windows 8, Windows 10, Windows 11, Windows Server, Unix, Linux (and its variants, such as Debian, Linux Mint, Fedora, and Red Hat), SunOS, Solaris, Blackberry OS, z / OS, iOS, macOS, or other operating systems, including proprietary operating systems. The operating system controls and schedules computer processes for execution, performs memory management, provides file system, networking, and I / O services, and provides a user interface (e.g., a graphical user interface (GUI)), among others.

[0273] Fig.18 The computer system 1802 shown is coupled to a network 1818, such as a LAN, WAN, or the Internet, via a communication link 1816 (wired, wireless, or a combination thereof). The network 1818 communicates with various computing devices and / or other electronic devices. The network 1818 communicates with one or more computing systems 1820 and one or more data sources 1822. Via a web-enabled user access point, the module 1814 can access the computing system 1820 and / or the data source 1822, or can be accessed by the computing system 1820 and / or the data source 1822. The connection can be a direct physical connection, a virtual connection, and other connection types. The web-enabled user access point can include a browser module that uses text, graphics, audio, video, and other media to present data and allows interaction with the data via the network 1818.

[0274] Access to the module 1814 of the computer system 1802 by the computing system 1820 and / or the data source 1822 may be via a web-enabled user access point, such as a personal computer, cellular phone, smart phone, laptop, tablet computer, e-reader device, audio player, or another device capable of connecting to the network 1818 of the computing system 1820 or data source 1822. Such a device may have a browser module implemented as a module that presents data using text, graphics, audio, video, and other media and allows interaction with the data via the network 1818.

[0275] The output modules may be implemented as a combination of fully addressable displays, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, or other types of displays and / or combinations of displays. The output modules may be implemented to communicate with input devices 1812, and they also include software with appropriate interfaces that allow the user to access data through programmable screen elements such as menus, windows, dialog boxes, toolbars, and controls (e.g., radio buttons, check boxes, sliding scales, etc.). In addition, the output modules may communicate with a set of input and output devices to receive signals from the user.

[0276] The input device may include a keyboard, a roller ball, a pen and stylus, a mouse, a trackball, a voice recognition system, or a pre-specified switch or button. The output device may include a speaker, a display screen, a printer, or a speech synthesizer. In addition, a touch screen may serve as a hybrid input / output device. In another embodiment, the user may interact with the system more directly, for example, via a system terminal connected to the score generator, rather than communicating via the Internet, a WAN or LAN or similar network.

[0277] In some embodiments, system 1802 may include a physical or logical connection established between a remote microprocessor and a mainframe computer for the express purpose of uploading, downloading, or viewing interactive data and databases online in real time. The remote microprocessor may be operated by an entity operating computer system 1802 comprising a client server system or a host server system, and / or may be operated by one or more of data sources 1822 and / or one or more of computing systems 1820. In some embodiments, terminal emulation software may be used on the microprocessor for participating in the micro-mainframe link.

[0278] In some embodiments, the computing system 1820 within the entity operating the computer system 1802 may access the module 1814 internally as an application or process executed by the CPU 1806 .

[0279] In some embodiments, one or more features of the systems, methods, and devices described herein may utilize URLs and / or cookies, for example, for storing and / or transmitting data or user information. A uniform resource locator (URL) may include a web address and / or a reference to a web resource stored in a database and / or server. A URL may specify the location of a resource on a computer and / or a computer network. A URL may include a mechanism for retrieving a network resource. A source of a network resource may receive a URL, identify the location of a web resource, and transmit the web resource back to the requester. A URL may be converted to an IP address, and a domain name system (DNS) may look up a URL and its corresponding IP address. A URL may reference a web page, file transfer, email, database access, and other applications. A URL may include a character sequence identifying a path, domain name, file extension, host name, query, fragment, scheme, protocol identifier, port number, username, password, flag, object, resource name, etc. The system disclosed herein may generate, receive, transmit, apply, parse, serialize, render, and / or perform actions on a URL.

[0280] Cookies, also known as HTTP cookies, web cookies, Internet cookies, and browser cookies, may contain data sent from a website and / or stored on a user's computer. This data may be stored by the user's web browser while the user is browsing. Cookies may contain useful information for a website to remember previously browsed information, such as a shopping cart on an online store, clicks on a button, login information, and / or a record of web pages or network resources visited in the past. Cookies may also contain information entered by the user, such as name, address, password, credit card information, etc. Cookies may also perform computer functions. For example, an application (e.g., a web browser) may use an authentication cookie to identify whether a user has logged in (e.g., logged into a website). Cookie data may be encrypted to provide security for consumers. Tracking cookies may be used to compile individual browsing histories. The system disclosed herein may generate and use cookies to access individual data. The system may also generate and use JSONweb tokens to store authenticity information, HTTP authentication as an authentication protocol, IP addresses, URLs, etc. for tracking session or identity information.

[0281] The computing system 1802 may include one or more internal and / or external data sources (e.g., data source 1822). In some embodiments, one or more of the data repositories and data sources described above may use, for example, Sybase, Oracle, CodeBase, DB2, PostgreSQL, and SQL server, as well as other types of databases, such as NoSQL databases (e.g., Couchbase, Cassandra, or MongoDB), flat file databases, entity-relational databases, object-oriented databases (e.g., InterSystems Cache), cloud-based databases (e.g., Amazon RDS, Azure SQL, Microsoft Cosmos DB, Azure Database for MySQL, Azure Database for MariaDB, Azure Cache for Redis, Azure managed instance for Apache Cassandra, Google Bare Metal solution for Oracle on Google Cloud, Google Cloud SQL, Google Cloud Spanner, Google Cloud Big Tables, Google Firestore, Google Firebase Realtime Database, Google Memorystore, Google MongoDB Atlas, Amazon Aurora, Amazon DynamoDB, Amazon Redshift, Amazon ElastiCache, Amazon MemoryDB for Redis, Amazon DocumentDB, Amazon Keyspaces, Amazon Neptune, Amazon Timestream, or Amazon QLDB), non-relational databases, or record-based databases.

[0282] The computer system 1802 may also access one or more databases 1822. The databases 1822 may be stored in a database or data repository. The computer system 1802 may access the one or more databases 1822 via the network 1818, or may directly access the database or data repository via the I / O devices and interface 1812. The data repository storing the one or more databases 1822 may reside within the computer system 1802.

[0283] Additional Embodiments

[0284] In the foregoing description, the systems and processes have been described with reference to specific embodiments thereof. However, it is apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the embodiments disclosed herein. Therefore, the description and drawings should be considered illustrative rather than restrictive.

[0285] In fact, although the system and process have been disclosed in the context of certain embodiments and examples, it will be understood by those skilled in the art that the various embodiments of the system and process extend beyond the specifically disclosed embodiments to other alternative embodiments and / or uses of the system and process and its obvious modifications and equivalents. In addition, although several changes of the embodiments of the system and process have been shown and described in detail, other modifications within the scope of the present disclosure will be obvious to those skilled in the art based on the present disclosure. It is also carefully considered that various combinations or sub-combinations of the specific features and aspects of the embodiments may be performed, and these combinations or sub-combinations still fall within the scope of the present disclosure. It should be understood that the various features and aspects of the disclosed embodiments may be combined with each other or replace each other to form different modes of embodiments of the disclosed systems and processes. Any method disclosed herein does not have to be performed in the order described. Therefore, it is hoped that the scope of the system and process disclosed herein should not be limited by the specific embodiments described above.

[0286] It should be understood that the systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desired properties disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure.

[0287] Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in the form of any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features of the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination. No one feature or feature group is essential or indispensable for each embodiment.

[0288] It should also be understood that, unless otherwise explicitly stated, or otherwise understood in the context of use, conditional language used herein, such as "may", "could", "likely", "for example", etc., is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements, and / or steps are included in any particular embodiment or will be performed in any particular embodiment with or without author input or prompting. The terms "include", "comprise", "have", etc. are synonymous and are used inclusively in an open manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, unless otherwise specified, the articles "one", "an" and "said" used in the present application and the appended claims should be understood to mean "one or more" or "at least one". Similarly, although operations can be depicted in a particular order in the drawings, it should be appreciated that such operations are not necessarily performed in the particular order or sequential order shown, or all the operations shown are not necessarily performed to achieve the desired result. In addition, the drawings can schematically depict one or more example processes in the form of a flow chart. However, other operations not depicted can be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations can be performed before, after, simultaneously with, or between any of the illustrated operations. In addition, operations can be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the above-mentioned embodiments should not be understood as requiring such separation in all embodiments, but it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Therefore, other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0289] In addition, although the methods and devices described herein may be susceptible to various modifications and alternative forms, specific examples thereof have been shown in the drawings and described in detail herein. However, it should be understood that the embodiments should not be limited to the specific forms or methods disclosed, but on the contrary, the embodiments should cover all modifications, equivalents and alternatives within the spirit and scope of the various embodiments described and the appended claims. In addition, any specific features, aspects, methods, properties, characteristics, qualities, attributes, elements, etc. disclosed herein in conjunction with an embodiment or embodiment can be used in all other embodiments or embodiments described herein. Any method disclosed herein does not have to be performed in the order described. The methods disclosed herein may include certain actions taken by the practitioner; however, the methods may also include any third party's instructions for these actions, whether express or implied. The scope disclosed herein also covers any and all overlaps, sub-ranges and combinations thereof. For example, "at most", "at least", "greater than", "less than", "between..." and other languages ​​include the numbers described. Numbers preceded by terms such as "about" or "approximately" include the recited number and should be interpreted based on the circumstances (e.g., as accurate as reasonably possible under the circumstances, e.g., ±5%, ±10%, ±15%, etc.). For example, "about 3.5 mm" includes "3.5 mm". Phrases preceded by terms such as "substantially" include the recited phrase and should be interpreted based on the circumstances (e.g., as accurate as reasonably possible under the circumstances). For example, "substantially constant" includes "constant". Unless otherwise noted, all measurements are under standard conditions, including temperature and pressure.

[0290] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, connecting language such as the phrase "at least one of X, Y, and Z" is generally understood in context to express that an item, term, etc. can be at least one of X, Y, or Z. Therefore, such connecting language is generally not intended to imply that certain embodiments require that at least one of X, at least one of Y, and at least one of Z each be present. The headings provided herein, if any, are for convenience only and do not necessarily affect the scope or meaning of the apparatus and methods disclosed herein.

[0291] Thus, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and novel features disclosed herein.

Claims

1. A method for determining a characteristic of a patient, comprising: Receiving, by a computer system, facial scan data of a patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and One or more ratios associated with the dental treatment plan are determined by the computer system.

2. The method according to claim 1, characterized in that The multiple reference points include at least one of the infraorbital point, condyle point, pupil point, nasal wing point, subnasal point, submental point, midpoint of hairline, midpoint of glabellar space, mandibular angle point, nasal process point, upper lip point, lower lip point, outer corner of eye point, tragus point, skin nasion point or tragus angle vertex.

3. The method according to claim 1, characterized in that Determining the plurality of reference points comprises: generating a low-dimensional representation of the patient's face based on the facial scan data; and One or more reference points are determined using a reference point identification model.

4. The method according to claim 3, characterized in that The low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

5. The method according to claim 3, characterized in that: The low dimensional representation is based on the depth data.

6. The method according to claim 3, characterized in that The low dimensional representation is based on the image data.

7. The method according to claim 1, characterized in that Determining the plurality of reference points comprises: applying, by the computer system, a deformable mask to the facial scan data; and The deformable mask is deformed, wherein deforming the deformable mask includes adjusting the deformable mask to reduce a difference between the deformable mask and the facial scan data.

8. The method according to claim 1, characterized in that The facial scan data includes motion information, wherein the method further comprises: A dynamic characteristic of the patient is determined by the computer system.

9. The method according to claim 8, characterized in that Determining the dynamic feature includes: Movement of the patient's mandible is detected by a motion detection model.

10. The method according to claim 1, further comprising: receiving, by the computer system, a dental model of the patient; as well as The dental model and the facial scan data are co-registered.

11. The method according to claim 10, characterized in that The dental model includes an upper jaw model and a lower jaw model.

12. The method according to claim 11, further comprising: generating a facial model of the patient; co-registering the dental model with the facial model; as well as The range of motion limits of the patient's mandible are determined by determining the amount of closure when the maxillary model collides with the mandibular model.

13. The method according to claim 1, further comprising: A condition associated with the patient is determined by the computer system.

14. The method according to claim 3, characterized in that Generating the low-dimensional representation comprises: A set of eigenfaces and a set of associated weights are determined, wherein the patient's face is described by a linear combination of the eigenfaces and their associated weights.

15. The method according to claim 7, characterized in that Deforming the deformable mask includes one or more of cage deformation, skeletal animation, or mesh interpolation.

16. A method for determining a characteristic of a patient, comprising: Receiving, by a computer system, facial scan data of the patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and A dynamic characteristic of the patient is determined by the computer system.

17. The method according to claim 16, characterized in that The multiple reference points include at least one of the infraorbital point, condyle point, pupil point, nasal wing point, subnasal point, submental point, midpoint of hairline, midpoint of glabellar space, mandibular angle point, nasal process point, upper lip point, lower lip point, outer corner of eye point, tragus point, skin nasion point or tragus angle vertex.

18. The method according to claim 16, characterized in that Determining the plurality of reference points comprises: generating a low-dimensional representation of the patient's face based on the facial scan data; and One or more reference points are determined using a reference point identification model.

19. The method according to claim 18, characterized in that The low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

20. The method according to claim 18, characterized in that The low dimensional representation is based on the depth data.

21. The method according to claim 18, characterized in that The low dimensional representation is based on the image data.

22. The method according to claim 16, characterized in that Determining the plurality of reference points comprises: applying, by the computer system, a deformable mask to the facial scan data; and The deformable mask is deformed, wherein deforming the deformable mask includes adjusting the deformable mask to reduce a difference between the deformable mask and the facial scan data.

23. The method according to claim 16, characterized in that Determining the dynamic feature includes: Movement of the patient's mandible is detected by a motion detection model.

24. The method of claim 16, further comprising: determining a facial model of the patient; receiving a skeletal model of the patient; as well as The skeletal model and the facial model are co-registered.

25. The method of claim 24, further comprising: Determine the contact relationship between the bones of the skeleton model.

26. The method of claim 16, further comprising: determining a facial model of the patient; receiving a dental model of the patient, the dental model comprising maxillary teeth and mandibular teeth; as well as The dental model and the facial model are co-registered, wherein co-registering the dental model and the facial model produces an orofacial model.

27. The method of claim 26, further comprising: Determine the occlusal surface.

28. The method of claim 26, further comprising: A function generating surface is determined, the function generating surface being indicative of an envelope of the function of dental arch motion.

29. The method of claim 16, further comprising: Determine the hinge axis.

30. The method of claim 16, further comprising: A condylar slope is determined, the determination being based at least in part on the protrusive movement.

31. The method of claim 16, further comprising: A left Bennett angle is determined, the determination being based at least in part on the right side shift.

32. The method of claim 16, further comprising: A right Bennett angle is determined, the determination being based at least in part on the left side shift.

33. A method for training a machine learning model, comprising: receiving a plurality of facial scans associated with a plurality of individuals, the facial scans comprising image data and depth data, at least one of the facial scans being marked to indicate the location of one or more reference points; generating a low-dimensional representation for each facial scan of the plurality of facial scans; providing the generated low-dimensional representation to the machine learning model; as well as Training the machine learning model, wherein training the machine learning model comprises adjusting one or more weights of the machine learning model.

34. The method according to claim 33, characterized in that Generating the low-dimensional representation includes computing one or more weights for the one or more eigenfaces.

35. The method of claim 33, further comprising performing the following operations before generating the low-dimensional representation: determining the location of one or more features to be excluded using different machine learning models; and The one or more features to be excluded are removed, wherein the removing comprises one or more of blurring the one or more features to be excluded or placing a solid object over the one or more features to be excluded.

36. The method according to claim 33, characterized in that Generating the low-dimensional representation includes generating a first two-dimensional representation, the method further comprising: generating a second two-dimensional representation for each facial scan of the plurality of facial scans, the second two-dimensional representation being different from the first two-dimensional representation; generating a second low-dimensional representation for each second two-dimensional representation; and After training the machine learning model, the following operations are performed: providing the second low-dimensional representation to a second machine learning model; providing at least one of the one or more weights of the machine learning model to the second machine learning model; and Training the second machine learning model, wherein training the second machine learning model comprises adjusting one or more weights of the second machine learning model.

37. A system for determining a characteristic of a patient, comprising: one or more processors; as well as a non-volatile storage medium having instructions embodied thereon, which when executed by the one or more processors cause the system to perform the following steps: Receiving, by a computer system, facial scan data of a patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and One or more ratios associated with the dental treatment plan are determined by the computer system.

38. The system according to claim 37, characterized in that The multiple reference points include at least one of the infraorbital point, condyle point, pupil point, nasal wing point, subnasal point, submental point, midpoint of hairline, midpoint of glabellar space, mandibular angle point, nasal process point, upper lip point, lower lip point, outer corner of eye point, tragus point, skin nasion point or tragus angle vertex.

39. The system according to claim 37, characterized in that Determining the plurality of reference points comprises: generating a low-dimensional representation of the patient's face based on the facial scan data; and One or more reference points are determined using a reference point identification model.

40. The system according to claim 39, characterized in that The low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

41. The system according to claim 39, characterized in that The low dimensional representation is based on the depth data.

42. The system according to claim 39, characterized in that The low dimensional representation is based on the image data.

43. The system according to claim 37, characterized in that Determining the plurality of reference points comprises: applying, by the computer system, a deformable mask to the facial scan data; and The deformable mask is deformed, wherein deforming the deformable mask includes adjusting the deformable mask to reduce a difference between the deformable mask and the facial scan data.

44. The system according to claim 37, characterized in that The facial scan data includes motion information, wherein the steps further include: A dynamic characteristic of the patient is determined by the computer system.

45. The system according to claim 44, characterized in that Determining the dynamic feature includes: Movement of the patient's mandible is detected by a motion detection model.

46. ​​The system of claim 37, wherein: The steps further include: receiving, by the computer system, a dental model of the patient; and The dental model and the facial scan data are co-registered.

47. The system according to claim 46, characterized in that The dental model includes an upper jaw model and a lower jaw model.

48. The system according to claim 47, characterized in that The steps further include: generating a facial model of the patient; co-registering the dental model with the facial model; and The range of motion limits of the patient's mandible are determined by determining the amount of closure when the maxillary model collides with the mandibular model.

49. The system according to claim 37, characterized in that The steps further include: A condition associated with the patient is determined by the computer system.

50. The system according to claim 39, characterized in that Generating the low-dimensional representation comprises: A set of eigenfaces and a set of associated weights are determined, wherein the patient's face is described by a linear combination of the eigenfaces and their associated weights.

51. The system of claim 43, wherein: Deforming the deformable mask includes one or more of cage deformation, skeletal animation, or mesh interpolation.

52. A system for determining a characteristic of a patient, comprising: one or more processors; as well as a non-volatile storage medium having instructions embodied thereon, which when executed by the one or more processors cause the system to perform the following steps: Receiving, by a computer system, facial scan data of the patient, the facial scan data comprising image data and depth data; determining, by the computer system, a plurality of reference points based on the facial scan data; determining, by the computer system, one or more reference points, lines, or planes; and A dynamic characteristic of the patient is determined by the computer system.

53. The system according to claim 52, characterized in that The multiple reference points include at least one of the infraorbital point, condyle point, pupil point, nasal wing point, subnasal point, submental point, midpoint of hairline, midpoint of glabellar space, mandibular angle point, nasal process point, upper lip point, lower lip point, outer corner of eye point, tragus point, skin nasion point or tragus angle vertex.

54. The system according to claim 52, characterized in that Determining the plurality of reference points comprises: generating a low-dimensional representation of the patient's face based on the facial scan data; and One or more reference points are determined using a reference point identification model.

55. The system according to claim 54, characterized in that The low-dimensional representation is based on a two-dimensional projection of at least a portion of the facial scan data.

56. The system according to claim 54, characterized in that The low dimensional representation is based on the depth data.

57. The system according to claim 54, characterized in that The low dimensional representation is based on the image data.

58. The system according to claim 52, characterized in that Determining the plurality of reference points comprises: applying, by the computer system, a deformable mask to the facial scan data; and The deformable mask is deformed, wherein deforming the deformable mask includes adjusting the deformable mask to reduce a difference between the deformable mask and the facial scan data.

59. The system according to claim 52, characterized in that Determining the dynamic feature includes: Movement of the patient's mandible is detected by a motion detection model.

60. The system of claim 52, wherein: The steps further include: determining a facial model of the patient; receiving a skeletal model of the patient; and The skeletal model and the facial model are co-registered.

61. The system according to claim 60, characterized in that The steps further include: Determine the contact relationship between the bones of the skeleton model.

62. The system of claim 52, wherein: The steps further include: determining a facial model of the patient; receiving a dental model of the patient, the dental model comprising maxillary teeth and mandibular teeth; and The dental model and the facial model are co-registered, wherein co-registering the dental model and the facial model produces an orofacial model.

63. The system according to claim 62, characterized in that The steps further include: Determine the occlusal surface.

64. The system according to claim 62, characterized in that The steps further include: A function generating surface is determined, the function generating surface being indicative of an envelope of the function of dental arch motion.

65. The system of claim 52, wherein: The steps further include: Determine the hinge axis.

66. The system of claim 52, wherein: The steps further include: A condylar slope is determined, the determination being based at least in part on the protrusive movement.

67. The system of claim 52, wherein: The steps further include: A left Bennett angle is determined, the determination being based at least in part on the right side shift.

68. The system of claim 52, wherein: The steps further include: A right Bennett angle is determined, the determination being based at least in part on the left side shift.

69. A system for training a machine learning model, comprising: one or more processors; as well as a non-volatile storage medium having instructions embodied thereon, which when executed by the one or more processors cause the system to perform the following steps: receiving a plurality of facial scans associated with a plurality of individuals, the facial scans comprising image data and depth data, at least one of the facial scans being marked to indicate the location of one or more reference points; generating a low-dimensional representation for each facial scan of the plurality of facial scans; providing the generated low-dimensional representation to the machine learning model; as well as Training the machine learning model, wherein training the machine learning model comprises adjusting one or more weights of the machine learning model.

70. The system according to claim 69, characterized in that Generating the low-dimensional representation includes computing one or more weights for the one or more eigenfaces.

71. The system according to claim 69, characterized in that The steps further include performing the following operations before generating the low-dimensional representation: using different machine learning models to determine the location of one or more features to be excluded; as well as The one or more features to be excluded are removed, wherein the removing comprises one or more of blurring the one or more features to be excluded or placing a solid object over the one or more features to be excluded.

72. The system according to claim 69, characterized in that Generating the low-dimensional representation includes generating a first two-dimensional representation, wherein the step further comprises: generating a second two-dimensional representation for each facial scan of the plurality of facial scans, the second two-dimensional representation being different from the first two-dimensional representation; generating a second low-dimensional representation for each second two-dimensional representation; and After training the machine learning model, the following operations are performed: providing the second low-dimensional representation to a second machine learning model; providing at least one of the one or more weights of the machine learning model to the second machine learning model; and Training the second machine learning model, wherein training the second machine learning model comprises adjusting one or more weights of the second machine learning model.

Images