Systems and methods for alignment and registration

Abstract

Devices, systems, and methods for importing and / or transferring data including dental data from a three-dimensional (3D) dental model to a two-dimensional (2D) dental image may manipulate the 3D dental model based on temporomandibular joints. A 2D projection image is generated from a 3D dental model of a patient. Dental data may be transferred if the 2D projection image matches (within a threshold) a 2D dental image of the patient (from a photograph, etc. If the 2D projection images do not match, the 3D dental model may be modified (based on the temporomandibular joint) and a new 2D projection image generated. In this manner, the position of the 3D dental model may be iterated to generate a plurality of 2D projection images.

Description

Priority Statement

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 585,581, filed September 26, 2023, entitled “SYSTEM AND METHOD FOR ALIGNMENT AND REGISTRATION,” the entire contents of which are incorporated herein by reference. By incorporating via reference

[0002] All publications and patent applications mentioned in this specification are incorporated herein by reference in their entirety, to the extent that each individual publication or patent application is specifically and individually indicated by reference. Technical Field

[0003] The systems and methods described in this article generally involve dental models, and more specifically, alignment and registration between different dental models associated with patients. Background Technology

[0004] Orthodontic procedures typically involve repositioning a patient's teeth into a desired arrangement to correct malocclusion and / or improve aesthetics. To achieve these goals, orthodontic appliances (such as braces, shell appliances, etc.) can be applied to the patient's teeth by an orthodontist and / or by the patient themselves. The appliances can be configured to apply force to one or more teeth to achieve the desired tooth movement according to the treatment plan.

[0005] Generally, treatment planning can be used in any medical procedure to help guide the desired treatment outcome. In some examples, orthodontic treatment planning can be used in orthodontic dental treatments that provide a series of removable appliances (e.g., orthodontic appliances, palatal expanders, etc.) to correct various orthodontic or dental conditions. Therefore, a treatment plan can be used to identify multiple intermediate stages (steps) and the corresponding dental appliances (orthodontic appliances) to be worn sequentially.

[0006] In some cases, a combined simulation of the patient's maxilla and mandible can provide a more comprehensive view of the patient's entire dentition, which can help determine the treatment plan. Summary of the Invention

[0007] The implementation addresses the need for a system for automatically, efficiently, and accurately importing or capturing dental information (tooth numbers and / or other dental features) from three-dimensional (3D) scans, datasets, or models of a patient's dentition into two-dimensional (2D) dental images. This application addresses these and other technical problems by providing technical solutions and / or automated agents for comparing 3D scans, 3D datasets, or other 3D models with 2D images. In some implementations, the 3D model can be projected onto a plane to generate a 2D projection that can be compared with a 2D image. If the 2D projection matches the 2D image (within a threshold), the dental information from the 3D model can be imported (transferred) to the 2D image.

[0008] This article describes devices, systems, and methods for importing or transferring information from a 3D model to a 2D image. The 3D model may include the patient's mandible positioned relative to the patient's maxilla. This positioning may be based on the location of the temporomandibular joint, which hinges the mandible to the maxilla. Generally, the 3D model may be positioned in a virtual 3D space, and a virtual camera may be positioned relative to the 3D model. A 2D projection of the 3D model is determined relative to the virtual camera. The 2D projection is compared to a 2D dental image of the patient. If the 2D projection matches the 2D dental image (within a threshold), data from the 3D model can be imported into the 2D dental image.

[0009] Any method described herein may include: generating a 3D alignment model based on a patient's three-dimensional (3D) dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxillary and mandibular portions; generating a two-dimensional (2D) alignment projection based on the 3D alignment model; determining the 2D difference between the patient's 2D dental image and the 2D alignment projection; and importing dental information from the patient's 3D dental scan into the patient's 2D dental image when the difference is less than a threshold.

[0010] In any method, generating a 3D alignment model may include determining the position of the mandibular portion relative to the maxillary portion. In some examples, the position of the mandibular portion may be constrained by the position of the joint that connects the mandibular portion to the maxillary portion. Furthermore, in some examples, the position of the mandibular portion may be determined at least in part by the temporomandibular joint positioned relative to the maxillary portion.

[0011] In any of the methods described herein, generating a 3D alignment model may include moving the mandibular portion relative to the maxillary portion. In some examples, the movement of the mandibular portion is based on the location of the joint that connects the mandibular portion to the maxillary portion.

[0012] Generally, generating a 2D alignment projection may include determining the position of a projection plane and the position of a 3D alignment model in a shared virtual 3D space. In some examples, generating a 2D alignment projection may include projecting dental elements from the 3D alignment model onto the projection plane, where the 2D alignment projection is based on the projected dental elements. In some other examples, the positions of the 3D alignment model and the projection plane are based at least in part on a viewpoint associated with a virtual camera set in the shared virtual 3D space.

[0013] In some variations, generating a 2D alignment projection may include cropping a portion of the projection plane before determining the 2D difference between the patient's 2D dental image and the 2D alignment.

[0014] In any of the methods described herein, determining the position of the 3D alignment model involves locating the patient's maxilla in 3D space using six degrees of freedom and locating the patient's mandible in 3D space using one degree of freedom.

[0015] In any of the methods described herein, generating a 2D alignment projection may include iteratively determining the position of the projection plane based on the difference between the patient’s 2D dental image and the 2D alignment projection.

[0016] Generally, in any of the methods described herein, 2D dental images can be based on photographs of the patient's dentition. In some variations, 2D dental images may include 2D information of the maxillary and mandibular portions.

[0017] In any of the methods described herein, determining the 2D difference between a patient's 2D dental image and a 2D alignment projection may include determining the difference between corresponding features of the patient's 2D dental image and the 2D alignment projection.

[0018] In some variations, determining the 2D difference between a patient's 2D dental image and a 2D alignment projection may include determining the difference between the contours of the corresponding dental structures in the patient's 2D dental image and the 2D alignment projection.

[0019] In other variations, determining the difference between a patient’s 2D dental image and a 2D alignment projection may include determining the difference between the tooth boundaries determined from the patient’s 2D dental image and the 2D alignment projection.

[0020] In any of the methods described herein, segmentation data may be generated using one or more machine learning engines and one or more 3D models. Furthermore, in any of the methods described herein, dental information may include tooth numbering information. Generally, 2D dental images are based on photographs of the patient's dentition.

[0021] In any method, generating a 3D alignment model may also include iteratively determining the position of the mandibular portion relative to the maxillary portion based on the difference between the patient's 2D dental image and the 2D alignment projection.

[0022] In some examples, generating a 2D alignment projection may include iteratively determining the position of the projection plane based on the difference between the patient's 2D dental image and the 2D alignment projection.

[0023] Generally, any method described herein may also include generating 3D images based on 2D alignment projection. In some variations, in any method described herein, the patient's 2D dental image is a closed-jaw photograph. In some examples, the patient's 3D dental scan may be associated with a previously determined treatment plan.

[0024] For example, any of these methods may include importing dental information from a patient's 3D dental scan and may include determining the occlusal category from the 3D dental scan. For example, determining the occlusal category from a 3D dental scan may include identifying one or more of the following: Class I malocclusion, Class II malocclusion, Class III malocclusion, reverse overbite, deep overbite, and / or open bite. Any of these methods may include measuring the degree of malocclusion from the patient's 3D dental scan.

[0025] Any of these methods can include using a patient's 3D dental scan to determine or correct tooth numbering.

[0026] For example, a method may include: generating or accessing a three-dimensional (3D) alignment model of a digital model of the patient's maxillary portion and a digital model of the patient's mandibular portion, wherein the digital models of the patient's maxillary portion and the patient's mandibular portion are based on one or more intraoral scans, wherein the 3D alignment model includes TMJ parameters; generating a two-dimensional (2D) alignment projection image from the 3D alignment model; determining a difference estimate between a 2D dental image of the patient's teeth and the 2D alignment projection; iteratively adjusting the TMJ parameters based on the difference estimate, and repeating the steps of generating the 2D alignment projection image and determining the difference estimate until the difference is less than a threshold or the number of iterations exceeds a second threshold; and outputting the 3D alignment model including the TMJ parameters.

[0027] This document also describes systems configured to perform any of these methods. For example, one system includes: one or more processors; and a memory configured to store instructions that, when executed by the one or more processors, cause the system to: generate a 3D alignment model based on a patient's three-dimensional (3D) dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxilla and mandible; generate a two-dimensional (2D) alignment projection based on the 3D alignment model; determine a 2D difference between the patient's 2D dental image and the 2D alignment projection; and, when the difference is less than a threshold, import dental information from the patient's 3D dental scan into the patient's 2D dental image.

[0028] For example, this document describes a non-transitory computer-readable storage medium including instructions that, when executed by one or more processors of the device, cause the device to perform operations including: generating a 3D alignment model based on a patient's three-dimensional (3D) dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxilla and mandible; generating a two-dimensional (2D) alignment projection based on the 3D alignment model; determining a 2D difference between the patient's 2D dental image and the 2D alignment projection; and importing dental information from the patient's 3D dental scan into the patient's 2D dental image when the difference is less than a threshold.

[0029] As mentioned, any of these methods can be configured to determine a patient's occlusal category. These methods may include: accessing or receiving one or more 2D images of a patient's dentition; using the one or more 2D images, registering a 3D digital model of the patient's dentition such that a 3D digital model of the patient's maxilla and a 3D digital model of the patient's mandible are mutually registered, such that the 3D digital model can provide relative movement of the mandibular 3D digital model relative to the maxillary 3D digital model; determining the patient's occlusal category from the registered 3D digital model; and outputting the occlusal category.

[0030] Occlusal categories can be one or more of the following: Class I malocclusion, Class II malocclusion, Class III malocclusion, reverse overbite, deep overbite, and / or open bite. For example, determining the occlusal category may include measuring the degree of malocclusion from a 3D digital model. In any of these methods, outputting the occlusal category may include displaying the occlusal category on a user interface.

[0031] Any system described herein may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, cause the system to: generate a 3D alignment model based on a patient's three-dimensional (3D) dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxilla and mandible; generate a two-dimensional (2D) alignment projection based on the 3D alignment model; determine a 2D difference between the patient's 2D dental image and the 2D alignment projection; and, when the difference is less than a threshold, import dental information from the patient's 3D dental scan into the patient's 2D dental image.

[0032] Any non-transitory computer-readable storage medium described herein may include instructions that, when executed by one or more processors of the device, cause the device to perform operations including: generating a 3D alignment model based on a patient's three-dimensional (3D) dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxilla and mandible; generating a two-dimensional (2D) alignment projection based on the 3D alignment model; determining a 2D difference between the patient's 2D dental image and the 2D alignment projection; and importing dental information from the patient's 3D dental scan into the patient's 2D dental image when the difference is less than a threshold.

[0033] All methods and apparatuses described herein, in any combination, are contemplated herein and can be used to achieve the benefits described herein. Attached Figure Description

[0034] A better understanding of the features and advantages of the methods and apparatus described herein will be obtained by referring to the following detailed description and accompanying drawings, which illustrate illustrative embodiments, in which:

[0035] Figure 1A The diagram illustrates an example of a computing environment configured to facilitate the collection and processing of digital scans of dental arches (containing teeth and / or bone).

[0036] Figure 1B This is a diagram illustrating an example of a scan segmentation engine.

[0037] Figure 1C A diagram showing an example including a 3D fusion engine is provided.

[0038] Figure 1D A diagram showing an example including a tooth modeling engine is provided.

[0039] Figure 1E A diagram showing an example including a tooth marking engine is provided.

[0040] Figure 1F A diagram showing an example including a 2D alignment engine is provided.

[0041] Figure 2An example 3D representation of a patient's dentition is shown.

[0042] Figure 3 An example virtual 3D space is shown.

[0043] Figure 4 This is a flowchart illustrating an example method for determining a match between a 2D projection and a 2D dental image.

[0044] Figures 5A to 5C Images related to the open bite described in this article are shown.

[0045] Figures 6A to 6C An image is shown that is associated with the closed bite described in this article.

[0046] Figures 7A to 7C Images associated with tooth numbering based on 2D dental images are shown.

[0047] Figure 8 This is a block diagram of an apparatus, which may be an example of an apparatus configured to perform one or more of the operations described herein.

[0048] Figure 9 This is a diagram illustrating an example computing environment in which the methods and devices described herein can be implemented.

[0049] Figure 10 This is a schematic diagram of an example of the treatment monitoring module described in this article. Detailed Implementation

[0050] Generally, the methods and devices described herein may include simulating and / or modeling a patient's maxilla and mandible together. In some examples, the maxilla and mandible may be integrated into a single model and / or may include a description of the patient's temporomandibular joint (TMJ). The description of the TMJ can provide the relationship between the maxilla and mandible. In some cases, the maxilla and mandible may be modeled separately, but they can be linked together by a description of the patient's TMJ. In some cases, the maxilla, mandible, and TMJ may be part of a single model. The model may be a digital model. Any suitable digital model format may be used, including point clouds, meshes, etc. In practice, these methods and devices may be used for modeling and visualization, including visualizing, displaying, and modifying a patient's jaw, and for treatment planning and / or design (including simulation and testing) of one or more dental appliances. As a non-limiting example, these methods and devices may be used to monitor deep overbite correction of the anterior teeth or any other condition, particularly those involving engagement between the maxilla and mandible. These methods and devices for implementing or using them (where both the maxilla and mandible, as well as the TMJ relationship, can be modeled together) offer significant advancements compared to techniques that perform maxillary and mandibular registration separately, especially in cases where a portion of the maxilla or mandible is occluded, which might hinder accurate modeling of the relationship between the maxilla and mandible.

[0051] The apparatus and / or methods described herein can be used for the planning and manufacture of dental appliances, including resilient polymer positioning devices, which are described in detail in the following documents: U.S. Patent No. 5,975,893 and International Patent Application No. WO 1998 / 058596, both of which are incorporated herein by reference for all purposes. Dental appliance systems employing the technology described in U.S. Patent No. 5,975,893 are commercially available under the trade name Invisalign System from Allian Technologies, Inc., San Jose, California.

[0052] Throughout the description of the embodiments, the terms “orthodontic appliance,” “orthodontic appliance,” or “dental appliance” are used synonymously with the terms “instrument” and “dental appliance” in dental applications. For clarity, the embodiments are described below in the context of the use and application of instruments (more specifically, “dental appliances”).

[0053] As used herein, “patient” can be any subject (e.g., human, non-human, adult, child, etc.) and is alternatively and equivalently referred to as “patient” or “subject” herein. As used herein, “patient” can but not be a medical patient. As used herein, “patient” can include a person receiving orthodontic treatment, which includes orthodontic treatment using a series of orthodontic appliances.

[0054] In any of these methods and devices, the method may include registering (e.g., aligning) a three-dimensional (3D) model with a two-dimensional (2D) image, including projecting the 3D model onto the 2D model. For example, these methods may include registering (3D to 2D) relative to two jaws using one or more camera parameters (e.g., position, rotation, etc.); thus, the methods described herein can coordinate three objects: the maxilla (e.g., a 3D model), the mandible (e.g., a 3D model), and the camera. This coordination can provide the relationship between the maxilla and mandible and allows for accurate and rapid assessment of a patient's occlusion using both jaws. Compared to currently used methods, modeling the maxilla and mandible separately can be significantly less accurate (especially in cases of missing or insufficient mandibular information) and can take significantly longer. Therefore, in general, these methods can use combined jaw pair optimization to determine the TMJ relationship between a patient's maxilla and mandible.

[0055] For example, Figure 1A This is a diagram illustrating an example of a computing environment 100 configured to facilitate the collection and processing of digital scans of dental arches containing teeth and / or bone. Environment 100 includes a computer-readable medium 152, a scanning system 154, a dental arch display system 156, and a segmentation system 158. One or more modules in computing environment 100 may be coupled to each other or to modules not explicitly shown.

[0056] The computer-readable medium 152 and other computer-readable media discussed herein are intended to represent a variety of potentially applicable technologies. For example, computer-readable medium 152 can be used to form a network or part of a network. Where two components are located on a device, computer-readable medium 152 may include a bus or other data conduit or plane. Where the first component is located on a device and the second component is located on a different device, computer-readable medium 152 may include a wireless or wired back-end network or LAN. Computer-readable medium 152 may also include relevant portions of a WAN or other network (if applicable).

[0057] Scanning system 154 may include a computer system configured to scan a patient's dental arch. As used herein, when viewed from an occlusal perspective, a "dental arch" may include at least a portion of the patient's dentition formed by the patient's maxillary and / or mandibular teeth. A dental arch may include one or more of the patient's maxillary or mandibular teeth, such as all teeth in the patient's maxilla or mandible. Scanning system 154 may include memory, one or more processors, and / or sensors to detect contours on the patient's dental arch. Scanning system 154 may be implemented as a camera, intraoral scanner, X-ray device, infrared device, medical scanning device (e.g., CT scanner, CBCT scanner, MRI scanner), etc. In some embodiments, scanning system 154 is configured to generate a three-dimensional (3D) scan of the patient's dentition. In other embodiments, scanning system 154 is configured to generate a two-dimensional (2D) scan or image of the patient's dentition. Scanning system 154 may include a system configured to provide a virtual representation of a physical model of the patient's dental arch. Scanning system 154 may be used as part of an orthodontic treatment plan. In some implementations, the scanning system 154 is configured to capture the patient's dental arch at the beginning, middle, or other stages of orthodontic treatment planning. The scanning system 154 may also be configured to receive 2D or 3D scan data previously acquired or by another system.

[0058] The dental arch display system 156 may include a computer system configured to display at least a portion of a patient's dentition. The dental arch display system 156 may include a memory, one or more processors, and a display device for displaying the patient's dentition. The dental arch display system 156 may be implemented as part of a computer system, a display for a dedicated intraoral scanner, etc. In some embodiments, the dental arch display system 156 facilitates the display of a patient's dentition using scans acquired at an earlier date and / or at a remote location. Note that the dental arch display system 156 may also facilitate the display of scans acquired simultaneously and / or locally. As described herein, the dental arch display system 156 may be configured to display the expected or actual results of an orthodontic treatment plan applied to a dental arch scanned by the scanning system 154. These results may include a 3D virtual representation of the dental arch, a 2D image or reproduction of the dental arch, etc.

[0059] Segmentation system 158 may include a computer system comprising memory and one or more processors configured to process scan data from scanning system 154. In some examples, 2D or 3D scan data may be segmented into individual dental components and processed into a 3D model of the patient's teeth. The 3D segmentation system may be configured to input one or more distinct regions of a 2D scan, 3D scan, or 3D model into a machine learning model to automatically segment the scan or model into individual dental components, including segmenting the scan or model into individual teeth, bone, interdental spaces, and / or gingiva. The segmented 2D / 3D scans or models can be used to create and implement dental treatment plans for the patient. For example, digital treatment planning software may include segmentation system 158 and receive 3D scans of the patient's dentition. Segmentation system 158 may then be configured to automatically segment the 3D scan. The digital treatment planning software may then be configured to automatically generate dental treatment plans for the patient, which may further include generating a 3D model of the patient's dentition including 3D segmentation. The segmentation system 158 may include a scan segmentation engine 160, a 3D fusion engine 162, a tooth modeling engine 164, a tooth marking engine 166, and optionally a treatment modeling engine 168, a 2D alignment engine 169, and a segmentation model data storage 167. One or more of the modules of the segmentation system 158 may be coupled to each other or to other modules not shown.

[0060] The scanning segmentation engine 160 of segmentation system 158 can implement an automated agent to process 2D or 3D scans acquired by scanning system 154. In some embodiments, scanning segmentation engine 160 formats scan data from a dental arch scan into one or more partitions, volumes, clippings, or regions. Scanning segmentation engine 160 can be incorporated into digital therapy planning software. The one or more partitions, volumes, clippings, or regions scanned can be subsets or segments of the original scan. In some embodiments, the one or more partitions, volumes, clippings, or regions scanned can have a resolution different from the resolution of the original 2D or 3D scan. For example, the one or more partitions, volumes, clippings, or regions scanned can have a lower resolution than the original scan. In other embodiments, the one or more partitions, volumes, clippings, or regions scanned can have the same resolution as the original scan. Scanning segmentation engine 160 can also be configured to implement an automated agent to segment 2D or 3D scans. In one implementation, the scan segmentation engine can input one or more scanned partitions, volumes, clippings, or regions into one or more machine learning models for segmentation into individual tooth features, such as maxillary, mandibular, and binary tooth segments. Segments of one or more scanned partitions, volumes, clippings, or regions can be combined to generate a complete semantic segmentation of a 2D or 3D scan.

[0061] The 3D fusion engine 162 of the segmentation system 158 can implement automated agents to align segmented scan data from the scanning segmentation engine 160 with the patient's digital dental 3D treatment plan. The 3D fusion engine 162 can be integrated into digital treatment planning software. In some embodiments, the 3D fusion engine 162 provides a coarse alignment of the segmented scan data and triangulation of each marker volume with the corresponding tooth features in the digital dental 3D treatment plan. The 3D fusion engine 162 can then provide a fine alignment of the segmented scan data with the dental treatment plan. In some embodiments, the 3D fusion engine 162 can preprocess the aligned segmented scan data and digital treatment plan to reduce digital noise and suppress potential segmentation errors. The 3D fusion engine 162 can also be configured to accurately number individual teeth in the digital treatment plan. Additionally, the 3D fusion engine can implement automated agents to stitch scan data representing crowns to digital treatment plan data representing roots, thereby providing the best possible resolution in the final segmented digital treatment plan.

[0062] The tooth modeling engine 164 can implement automated agents to replace or modify low-quality or low-resolution segmentation data from 2D / 3D scans with higher-quality generic tooth models. The tooth modeling engine 164 can be integrated into digital treatment planning software. In one embodiment, the tooth modeling engine 164 can be configured to identify segmented teeth from segmentation scan data and identify generic tooth models corresponding to the segmented teeth. In one embodiment, the tooth modeling engine 164 can implement automated agents to fit the generic tooth model to the segmented teeth. The generic tooth model can be modified / adjusted / rotated to fit precisely within the segmented teeth. The tooth modeling engine 164 can then be configured to implement automated agents to convert the adjusted generic tooth model into a digital treatment plan for the patient. This process can be repeated for all segmented teeth from 2D / 3D scans.

[0063] The tooth labelling engine 166 can automate the process of labeling segmented teeth from segmented 2D / 3D scans received from the segmentation engine 160. The tooth labelling engine can be integrated into digital therapy planning software. In one implementation, the tooth labelling engine 166 receives 2D / 3D segmented scans. The tooth labelling engine 166 can apply a morphological erosion algorithm to the segmented scan to divide it into N voxel volumes, where N is the number of teeth in the segmented scan.

[0064] Optional (e.g.) Figure 1AThe treatment modeling engine 168 (indicated by the dashed line) can be configured to store and / or provide instructions for achieving orthodontic treatment plans and / or the results of orthodontic treatment plans using segmented 3D models, segmented scans, and / or a combination of segmented scans and digital dental models. The treatment modeling engine 168 can be incorporated into digital treatment planning software. Optionally, the treatment modeling engine 168 can provide the results of orthodontic treatment plans on a 3D model. In some embodiments, the 3D model can be rendered into one or more 2D images from multiple perspectives. Optionally, the treatment modeling engine 168 can model the results of applying orthodontic appliances to a patient's dental arch during orthodontic treatment planning. In some embodiments, the treatment modeling engine 168 can be configured to save, transfer, or output digital dental models and / or digital orthodontic treatment plans. In some embodiments, the digital dental model and / or digital orthodontic treatment plan can be displayed on a monitor for a user of the digital treatment planning software, such as a physician. The user can edit or modify the proposed digital dental model and / or digital orthodontic treatment plan, for example, by interacting with a user input device of the digital treatment planning software (e.g., mouse and keyboard, touchscreen, joystick, etc.).

[0065] A 2D alignment engine 169 can determine the alignment between a segmented 3D model and a 2D dental image. The 2D alignment engine 169 can be integrated into digital treatment planning software. The segmented 3D model (which can be determined by one or more modules of the segmentation system 158) can include a maxillary portion and a mandibular portion. In some implementations, the maxillary portion can be independent of the mandibular portion. However, because the mandibular portion is physically constrained to the maxillary portion via the patient's temporomandibular joint, the mandibular portion may be restricted to possible positions relative to a 3D model of the patient's dentition that includes both the maxillary and mandibular portions. The 2D alignment engine 169 can determine the alignment between the patient's 3D model of the patient's dentition and the 2D image. After determining the alignment between the 3D model and the 2D image, information from the patient's 3D model can be used to identify one or more portions or elements included in the 2D image. For example, tooth numbering information from the 3D model can be used to number the teeth in the 2D image. The following will combine... Figures 1F to 8 A more detailed description of the 2D alignment engine 169 is provided.

[0066] The segmentation model data storage 167 stores segmented images, including segmented 3D models of the patient generated by one or more modules within the segmentation system 158. For example, the segmentation model data storage 167 may include a segmented 3D model of the patient, which includes tooth numbering information. The segmentation model data storage 167 may also include 2D images of the patient aligned with one or more 3D models and may include tooth numbering information.

[0067] As used herein, any "engine" may include one or more processors or a portion thereof. A portion of one or more processors may include a portion of hardware less than all the hardware comprising any given one or more processors, such as a subset of registers, a portion of the processor dedicated to one or more threads of a multithreaded processor, a time slice of the processor wholly or partially dedicated to performing a portion of the engine's functionality, etc. Thus, the first engine and the second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on implementation-specific or other considerations, the engine may be centralized or its functionality may be distributed. An engine may include hardware, firmware, or software embodied in a computer-readable medium for execution by a processor. The processor uses implemented data structures and methods to transform data into new data, as described with reference to the accompanying figures. In some examples, the engines discussed herein may be implemented in digital orthodontic treatment planning software.

[0068] The engine described herein, or the engine through which the systems and apparatuses described herein can be implemented, is, for example, a cloud-based engine. As used herein, a cloud-based engine is an engine that can use a cloud-based computing system to run applications and / or functions. All or part of the applications and / or functions can be distributed across multiple computing devices and are not required to be limited to a single computing device. In some embodiments, a cloud-based engine can execute functions and / or modules accessed by an end user through a web browser or container application without requiring the functions and / or modules to be locally installed on the end user's computing device.

[0069] As used herein, "data storage" can include a repository with any applicable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. For example, data storage can be implemented as software embodied in a physical computer-readable medium on a dedicated machine, in firmware, in hardware, a combination thereof, or in an applicable known or convenient apparatus or system. Components associated with data storage (such as database interfaces) can be considered as "part" of data storage, part of some other system components, or a combination thereof, although the physical location and other characteristics of components associated with data storage are not critical to understanding the techniques described herein.

[0070] Data storage can include data structures. As used herein, a data structure is associated with a specific way of storing and organizing data in a computer, making it usable effectively in a given context. Data structures are typically based on a computer's ability to access and store data anywhere in its memory, specified by an address, a bit string that can itself be stored in memory and manipulated by a program. Thus, some data structures are based on using arithmetic operations to compute the address of a data item; others are based on storing the address of the data item within the structure itself. Many data structures use both principles, sometimes combined in a non-simple way. Implementing a data structure typically requires writing an assembly of programs that create and manipulate instances of that structure. The data storage described in this article can be a cloud-based data storage. Cloud-based data storage is data storage that is compatible with cloud-based computing systems and engines.

[0071] Figure 1B This is a diagram illustrating an example of a scan segmentation engine 160. The scan segmentation engine 160 may include an image processing engine 170, a machine learning engine 172, a volume merging engine 174, and a scan data storage engine 176. One or more modules of the scan segmentation engine 160 may be coupled to each other or to modules not shown.

[0072] Image processing engine 170 can implement one or more automated agents configured to format 2D or 3D scan data from dental arch scan data into one or more partitions, volumes, clips, or regions of the scan. For example, the image processing engine can receive or access 3D scans of a patient's dentition, such as CT scans, CBCT scans, or MRI scans, which may include high-resolution imaging data of the patient's dental features, including the patient's teeth and the maxilla and mandible of the patient's jaw. The image processing engine can then process the scan into one or more partitions, volumes, clips, or regions of the scan, which may be a subset of the original scan. For example, one or more partitions, volumes, clips, or regions of the scan may be, for example, clips with data representing only the patient's maxilla, mandible, and / or teeth. In one implementation, the image processing engine can consider specific geometric features of the 2D / 3D scan to determine how / where to clip the scan. For example, the image processing engine can perform tooth region centering calculations to determine where the patient's teeth are located in the 2D / 3D scan.

[0073] Image processing engine 170 can calculate an estimate of the center of a tooth region, which the system can then use to generate one or more partitions, volumes, clippings, or regions of a scan that include scan data of the patient's teeth. Image processing engine 170 can also be configured to implement automated agents to resample or adjust the resolution of a 2D / 3D scan or one or more partitions, volumes, clippings, or regions of the scan. For example, in one embodiment, the entire 2D / 3D scan can be resampled to have a lower resolution than the original scan. In another embodiment, one or more partitions, volumes, clippings, or regions of the scan can be resampled to have a different (e.g., lower) resolution. Image processing engine 170 can provide processed scan data and / or other data to scan data storage 176.

[0074] Machine learning engine 172 can implement one or more automated agents configured to apply one or more machine learning engines to segment processed scan data from an image processing engine. For example, machine learning engine 172 can use one or more partitions, volumes, clippings, or regions of the original 2D / 3D scan (e.g., CT scan, CBCT scan, or MRI scan) and / or scans from an image processing engine as input. As mentioned above, the one or more partitions, volumes, clippings, or regions of the scan can also have various resolutions, as some clippings can have lower resolutions than the original 2D / 3D scan. Multiple of the above inputs can be used to generate segmentation data. For example, a low-resolution version of a 2D / 3D scan can be input into the machine learning engine to generate maxillary / mandibular / binary tooth segmentation. Additionally, one or more partitions, volumes, clippings, or regions of scans at different resolutions can be input into the machine learning engine to generate segmentation data. A higher-resolution clipping of a patient's teeth can be input into the machine learning engine to generate segmentation data for the patient's teeth. Additionally, a lower-resolution clipping of the patient's maxillary / mandibular jaw can be input into the machine learning engine to generate segmentation data. Machine learning engine 172 can provide segmented data and / or other data to scanned data storage 176.

[0075] Examples of machine learning systems that can be used by a machine learning engine include, but are not limited to, convolutional neural networks (CNNs) (such as V-net, U-net, ResNeXt, Xception, RefineNet, Kd-Net, SO Net, Point Net, or Point CNN) and additional machine learning systems (such as decision trees, random forests, logistic regression, support vector machines, AdaBoosT, K-nearest neighbors (KNN), quadratic discriminant analysis, neural networks, etc.). Furthermore, variants of the aforementioned CNNs can be incorporated. For example, a CNN such as U-net can be modified to use alternative convolutional blocks (e.g., ResNeXt or Xception) instead of the default VGG-type blocks.

[0076] The volume merging engine 174 can implement one or more automated agents configured to merge segmentation data from the machine learning engine 172 into a complete semantic segmentation of a 2D or 3D scan, including segmentation of the patient's maxilla / mandible / each tooth. As described above, the machine learning engine can provide segmentation data from a variety of scan data inputs, including segmenting the original 2D / 3D scan, segmenting resampled (e.g., low-resolution) scans, and / or segmenting one or more partitions, volumes, clippings, or regions of the scan. The resulting segmentation data includes multiple segmented volumes, each potentially having varying resolutions and belonging to varying locations within the original 2D / 3D scan. The volume merging engine 174 can be configured to implement automated agents to merge segmentation volumes from the machine learning engine into a single comprehensive segmentation of the original 2D or 3D scan. For example, the volume merging engine 174 can be configured to merge a first volume (e.g., a low-resolution maxillary / mandibular / binary tooth segmentation volume) and a second volume (e.g., a high-resolution, multi-class tooth segmentation volume) using the following steps: 1) removing binary tooth markers from the first volume; 2) adjusting the resolution of the first volume to be the same as the second volume; and 3) replacing voxels in the first volume with voxels from the second volume. The resulting volume contains information about the high-resolution teeth and the low-resolution maxillary and mandibular regions.

[0077] The scan data storage 176 can be configured to store data related to 2D or 3D scans, cropped or resampled scan data, segmented scan data and / or merged volume data from the modules described above.

[0078] Figure 1C A diagram illustrating an example including a 3D fusion engine 162 is shown. The 3D fusion engine 162 may include a feature alignment engine 178, a bone preprocessing engine 180, a tooth numbering engine 182, and a merging dental model data storage 184. One or more modules of the 3D fusion engine 162 may be coupled to each other or to modules not shown.

[0079] The feature alignment engine 178 can implement one or more automated agents configured to align and merge segmented scan data from the scan segmentation engine 160 with a digital 3D dental treatment plan. A digital 3D dental treatment plan can be generated during the patient's dental treatment process. The dental treatment plan may include a 3D model, such as a 3D mesh model or a 3D point cloud, which can be generated from scans of the patient's teeth (such as intraoral scans). The dental treatment plan includes information that can be used to simulate, modify, and / or select among various orthodontic treatment plans. The feature alignment engine 178 is configured to add segmented 3D scan data (such as segmented data from 3D CT scans, CBCT scans, or MRI scans) to be added to the 3D dental treatment plan. It is assumed that the 3D scans are segmented using software different from the dental treatment planning software, and the segmentation results are provided as a 3D array of bone markers and teeth with proportional information. The feature alignment engine automatically aligns the segmented 3D scan data and fills the digital dental plan with realistic tooth roots and bone surfaces.

[0080] The feature alignment engine 178 can first generate a coarse alignment between the segmented 3D scan data and the digital treatment plan. In one implementation, this coarse alignment can be based on a comparison of the crowns from the digital dental model with the volumes of each corresponding segmented tooth from the segmented 3D scan. The feature alignment engine 178 is capable of calculating a vector from the center of the occlusal tooth to the center of the opposite occlusal tooth in the segmented 3D scan. Using these vectors, the system can find the most prominent “tip” point on each tooth in the segmented 3D scan. These “tip” points can be aligned with corresponding points in the digital dental treatment plan. The feature alignment engine 178 can then generate a fine alignment between the segmented 3D scan data and the digital treatment plan. This can be accomplished using, for example, an Iterative Closest Point (ICP) algorithm.

[0081] The bone preprocessing engine 180 is configured to provide preprocessing of the 3D scanned surface to reduce digital noise and suppress potential segmentation errors. In one embodiment, the bone preprocessing engine 180 is configured to repair the alveolar bone in a digital dental model. Once the alveolar bone is repaired, translucent bone can be displayed on the planned tooth movement without the visual interference of moving root contours and immovable alveolar contours. This can be presented to a user, such as a user of digital orthodontic treatment planning software. The alveolar region can be detected as a portion of the bone surface sufficiently close to some of the 3D scanned tooth surface. The alveolar region can be removed from the bone surface by the bone preprocessing engine, and the remaining holes can be filled with a smoothing patch. In one embodiment, the bone preprocessing engine 180 can produce filtering of small connecting components and some general smoothing of the surface.

[0082] The tooth numbering engine 182 can be configured to number and / or renumber individual teeth in a digital dental treatment plan. It should be noted that the tooth numbers in the digital dental treatment plan and the tooth numbers from the segmented 3D scan can differ. A typical reason is missing teeth. For example, if the first premolar is actually missing, the automated 3D scan segmentation may incorrectly guess that the second premolar is missing instead of the first. Therefore, ICP surface matching can be used to ignore the tooth numbering and provide the correct alignment in this case. This process assumes that the tooth numbering in the digital dental treatment plan is correct and updates the tooth numbering in the 3D segmentation of the scan.

[0083] The merged dental model data storage 184 can be configured to store data related to the alignment between the segmented 3D scan and the digital dental model from the aforementioned modules, as well as tooth / alveolar repair and / or tooth numbering data from the bone pretreatment engine.

[0084] Figure 1D A diagram illustrating an example including a tooth modeling engine 164 is shown. The tooth modeling engine 164 may include a general tooth engine 186, a transformation engine 188, and a tooth modeling data storage 190. One or more modules of the tooth modeling engine 164 may be coupled to each other or to modules not shown.

[0085] The general-purpose tooth engine 186 can implement one or more automated agents configured to fit a general-purpose tooth model to segmentation data from segmented 3D scans. As mentioned above, segmentation of 3D scans (such as CT scans, CBCT scans, MRI scans, etc.) can be performed on low-resolution resampled arrays to accommodate the fast but expensive memory available on GPU devices. Therefore, the segmentation details of 3D scans may have noise or low-resolution surfaces. To overcome these drawbacks in segmented 3D scans, the general-purpose tooth engine 186 can be configured to use segmentation data from segmented 3D scans as auxiliary reference data and fit a general-purpose tooth model corresponding to the segmented teeth to that data. For example, a general-purpose tooth model can be constructed according to U.S. Patent No. 7,844,429, which is incorporated herein by reference in its entirety. A universal tooth is a template for a corresponding type of tooth (canine, incisor, premolar, etc.) and can be pre-built using multiple mesh-based models specific to that particular tooth, observed in different patients, and possessing unique landmark points (e.g., a set of 3D points that allow for the reconstruction of a 3D mesh at the desired resolution and properties, such as shape, smoothness, etc.). The universal tooth engine 186 can be configured to match an appropriate universal tooth model to 3D scan segmentation data. In one implementation, the universal tooth engine is capable of selecting a portion of a 3D scan for segmentation, such as a single segmented tooth in the 3D scan. The universal tooth engine is then capable of selecting a universal tooth model corresponding to the selected tooth and fitting the universal tooth model to the segmentation data.

[0086] The transformation engine 188 can implement one or more automated agents configured to adjust the position and orientation of a generic dental model to better match the position and orientation of selected segmented teeth from 3D scan data. The generic dental model can be adjusted by adding or modifying several or all control points. In one implementation, adding control points may include: finding vertex locations from the segmentation data of a specific tooth's 3D scan, overlaying the outline of the generic dental model onto the segmentation data, identifying differences between the segmentation data and the generic dental model, calculating the coordinates of points along the differences, and adding one or more control points to the generic dental model at these calculated coordinates. The control points allow manipulation of the position / orientation of the generic dental model. The adjusted generic dental model can then be transformed into a segmented 3D scan (or into a digital dental treatment plan).

[0087] The tooth modeling data storage 190 can be configured to store data related to the data from the aforementioned modules, including general tooth model data, 3D control point data, and transformation data from the general tooth model to segmented 3D scans or digital dental treatment plans.

[0088] Figure 1EA diagram illustrating an example including a tooth marking engine 166 is shown. The tooth marking engine 166 may include an erosion engine 192 and a tooth marking data storage 194. One or more modules of the tooth marking engine 166 may be coupled to each other or to modules not shown.

[0089] The erosion engine 192 can implement one or more automated agents configured to individually number / label segmented teeth during 3D segmentation scans. In one embodiment, the erosion engine 192 receives a binary volume (label map) of teeth as input after automated segmentation of a 3D scan (such as segmentation of a CT scan, CBCT scan, or MRI scan as described above). The erosion engine 192 can be configured to separate the label map using a watershed algorithm. In one embodiment, a seed for the watershed algorithm is formed by applying erosion iterations to the label map. These seeds take into account the morphological structure of the teeth during seed preparation, thereby improving the quality of volume separation. The seed can be applied to the original binary label map, and the watershed algorithm can be applied again to segment the individual teeth into separate components for more accurate labeling.

[0090] The tooth marker data storage 194 can be configured to store data related to the data from the modules described above, including marker / number data, erosion data, and seed data as described herein.

[0091] Figure 1F A diagram illustrating an example including a 2D alignment engine 169 is shown. The 2D alignment engine 169 may include a 3D model simulation engine 196, a 2D projection engine 197, a registration engine 198, and a registration data storage 199. The 2D alignment engine 169 determines the alignment between a patient's 2D dental image and a patient's 3D alignment model. The 3D alignment model is based on a segmented 3D scan of the patient as determined herein. Therefore, the 3D alignment model may include some or all of the information from the patient's segmented 3D scan. For example, the 3D alignment model may include the maxillary portion and / or the mandibular portion. In some cases, the mandibular portion may be positioned relative to the maxillary portion. The alignment between the 2D dental image and the 3D alignment model can be determined by generating a 2D projection of the patient's 3D alignment model and comparing the 2D projection with the patient's 2D dental image. If the comparison between the 2D projection and the patient's 2D dental image is favorable (i.e., the 2D projection matches the 2D dental image with a threshold), the information from the 3D alignment model (segmented 3D model) can be used with the 2D dental image (imported into the 2D dental image).

[0092] The 2D projection of the patient's 3D alignment model can include projections of one or more teeth. For example, the 2D projection can include incisors numbered 7, 8, 9, and 10. The patient's 2D dental image can also include incisors 7, 8, 9, and 10. The 2D alignment engine 169 can determine the degree of matching between the 2D projection of the incisors and the 2D dental image of the same incisors. That is, the 2D alignment engine 169 (using the 2D projection engine 197 and registration engine 198 described below) can determine the degree of matching between the contours of the incisors in the 2D projection and the contours of the incisors in the 2D dental image. Matching alignment can be based on the contours of the 2D projection and the 2D dental image matching within a threshold. Although incisors are used for illustration here, any element of the 2D projection and the 2D dental image can be compared with each other to determine their degree of matching.

[0093] Once alignment (also known as matching alignment or simply matching) is determined, information from the 3D alignment model can be imported into the 2D image. Example information that can be imported may include tooth numbering information, but any feasible information can be imported and / or transferred. In some variations, information from the 3D alignment model can be used in conjunction with data from the 2D dental image to generate photo-realistic oral images that can be used for patient consultation, education, or diagnosis.

[0094] The 3D alignment model can be determined using a tooth modeling engine 164, a tooth marking engine 166, or any other feasible engine, module, or program. The patient's 3D model alignment can include the maxillary portion with the upper dental arch and the mandibular portion with the lower dental arch. The maxillary portion can be separated from the mandibular portion. Despite separation, the positions of the maxillary and mandibular portions can be linked through the patient's temporomandibular joint. In other words, the mandibular portion can be positioned relative to the maxillary portion based on the position of the temporomandibular joint.

[0095] The location of the maxilla in 3D space can be described using six variables corresponding to the six degrees of freedom. The six example variables can include x, y, z (coordinates in a Cartesian coordinate system) and θ, ψ, and... (Pitch, yaw, and roll). The mandibular portion can be positioned in 3D space relative to the projected positions of the maxillary portion and the temporomandibular joint. Since the temporomandibular joint restricts the position of the mandibular portion, positioning it in 3D space can be simplified to two degrees of freedom (e.g., y and z) relative to the maxillary portion. In some variations, positioning the mandibular portion in 3D space can be simplified to one degree of freedom (the rotation angle around the temporomandibular joint between the maxillary and mandibular portions).

[0096] Figure 2An example 3D representation of a patient's dentition 200 is shown. The dentition 200 includes a maxillary portion 210 and a mandibular portion 220. As described above, the mandibular portion 220 can be located relative to the maxillary portion 210 using the temporomandibular joint 230. In some variations, the maxillary portion 210 can be located in 3D space using six variables (six degrees of freedom), and the mandibular portion 220 can be located by small translational movements relative to the maxillary portion 210 along two variables (e.g., the y-axis and / or the z-axis).

[0097] It is worth noting that this can also be used conversely to locate the patient's dentition 200. That is, the mandibular portion 220 can be located in 3D space using six variables, and then the maxillary portion 210 can be located relative to the mandibular portion 220 using the temporomandibular joint 230.

[0098] Furthermore, the mandibular portion 220, positioned relative to the maxillary portion 210, may also include a jaw angle. The jaw angle describes the amount of jaw opening between the maxillary portion 210 and the mandibular portion 220. The jaw angle (based on the temporomandibular joint 230) can be from a minimum angle (representing a closed mouth) to any maximum angle (representing a fully open mouth).

[0099] return Figure 1F The 3D model simulation engine 196 determines the position of the patient's 3D alignment model in virtual 3D space. Therefore, the 3D model simulation engine 196 positions the patient's segmented 3D scan (which may include the maxillary and mandibular portions) in 3D space. Determining the position of the patient's segmented 3D scan may include: determining the position of the maxillary portion using six variables (six degrees of freedom), and determining the position of the mandibular portion using one or two variables (one or two degrees of freedom). The 3D model simulation engine 196 can also determine the jaw angle between the maxillary and mandibular portions.

[0100] The 2D projection engine 197 can generate 2D projections of the maxilla and mandible portions, which are positioned in 3D space by the 3D model simulation engine 196. The 2D projections are determined from the perspective of a virtual camera positioned in 3D space. The virtual camera's positioning is determined at least partially by the positions of the maxilla and mandible portions, as determined by the 3D model simulation engine 196. The 2D projections are projected onto a reference plane.

[0101] Figure 3An example virtual 3D space 300 is shown. The virtual 3D space 300 can be used to establish a relationship between a 3D alignment model and its associated 2D projection. The 3D alignment model may include an maxillary portion and a mandibular portion, as well as a reference plane. A virtual camera 310, a mandibular portion 320, and a reference plane 330 are shown in the virtual 3D space 300. For simplicity, only the mandibular portion 320 is shown in the virtual 3D space 300. In other variations, the virtual 3D space 300 may include both the maxillary portion (not shown) and the mandibular portion 320. The mandibular portion 320 can be positioned in the virtual 3D space 300 based on data from the 3D model simulation engine 196 and / or the position of the virtual camera 310. The reference plane 330 is an imaginary plane between the virtual camera 310 and the mandibular portion 320. Typically, the reference plane 330 is perpendicular to the virtual camera 310. The virtual 3D space 300 is virtual because the patient's jaw and the virtual camera 310 can exist in a common virtual 3D space. In other words, virtual 3D space 300 can be a structure used to determine or calculate one or more elements without the existence of any physical space.

[0102] In some cases, the 2D projection engine 197 can generate a 2D projection of the 3D alignment model onto the reference plane 330 by determining the visibility of any point on the 3D alignment model at the reference plane 330. In some embodiments, the 2D projection engine 197 can determine from the perspective of the reference plane 330 and / or the virtual camera 310 whether one or more points on the 3D alignment model are partially or completely occluded by other parts of the 3D alignment model.

[0103] The registration engine 198 compares the 2D projection of the 3D alignment model with the patient's 2D dental image. If the 2D projection matches the 2D dental image (within a comparison threshold), the registration engine 198 can import or transfer the corresponding information from the 3D alignment model (and / or the segmented 3D model that provides the basis for the 3D alignment model) to the 2D dental image. As described above, any dental element or feature (such as tooth contours) can be compared to determine if the 2D projection matches the 2D dental image. If a match is determined, information such as tooth numbering can be copied or exported from the 3D alignment model to the 2D dental image.

[0104] On the other hand, if the 2D projection of the 3D alignment model does not match the 2D dental image, the registration engine 198 can adjust the position of the virtual camera (via the 2D projection engine 197), the position of the 3D alignment model in 3D space, and / or the relationship between the upper and lower jaw parts (via the 3D model simulation engine 196). After the position adjustment, the 2D projection engine 197 can generate a new (updated) 2D projection, and the registration engine 198 can compare the new 2D projection with the 2D dental image. The registration engine 198 can iteratively position the 3D alignment model and / or the virtual camera until a match is detected. Once a match is detected, the data can be stored in the registration data storage 199. For example, the registration engine 198 can store the 2D projection data along with additional data (tooth number information, etc.) in the registration data storage 199.

[0105] In some examples, a portion of the projection plane 330 may be cropped before determining whether a match exists between the 2D projection and the 2D dental image. Cropping can help ensure that only relevant portions of each image are compared. Furthermore, cropping can beneficially reduce any additional computations required to determine a match.

[0106] The methods and apparatus described herein for performing these functions can segment and / or number teeth in images using a trained neural network (a trained machine learning agent), and / or compare one or more 3D images with one or more 2D images. For example, one or more original images of a patient's teeth can be segmented and / or renumbered using a trained neural network, and one or more images can be extracted from a 3D model of the patient's jaw and rendered for comparison, in some cases by projecting the images to a segmentation mask. The methods and apparatus can optimize the model by minimizing the difference between the projection and the segmentation mask. Rendering of one or more images from a 3D model can be performed using camera parameters, which can be initially set to a predetermined set of values ​​as an initial starting set (e.g., initial guesses); this initial set can be based on historical data, patient data, etc. Camera values ​​can orient the 3D model in 3D space, thereby allowing virtual selection of one or more images of the 3D model based on the camera parameters. The camera parameters can be fine-tuned by matching virtual (simulated) 2D images with actual images of the patient's teeth. (See above references.) Figure 3The method discussed herein can acquire virtual images of a patient's teeth (e.g., jaw) and can crop simulated images based on camera parameters (e.g., a mesh can be projected onto an image plane indicating a center (e.g., the camera's optical axis), and the image can be cropped based on camera parameters). Therefore, camera information can be used to prepare a 2D projection from a 3D model including the upper and lower teeth; in addition to cropping the image, the camera settings can also provide rotation and perspective transformations, including generating a 2D image of the 3D model that can generate unique images of the upper and lower dental arches that can be compared with actual images of all or part of the upper and lower dental arches. For example, camera parameters can be used to generate virtual images (which can be segmented and / or have numbered teeth) that can be directly compared with images of the patient's teeth (which can also be segmented and / or numbered).

[0107] In general, the methods and devices described herein can introduce constraints that define how the maxilla and mandible move relative to each other. This constraint can be a TMJ relationship, for example, based on a description of the patient's TMJ of the maxilla and mandible, or in some cases, based on a description of the patient's TMJ of a digital model of the maxilla and mandible. This TMJ relationship defines how the maxilla and mandible can move relative to each other. This approach contrasts with earlier studies that allowed unconstrained or minimally constrained movement between models of the maxilla and mandible, including virtual models.

[0108] Constraints on the TMJ can be modeled after manipulation of the patient's anatomical TMJ, and can take into account jaw shape (length), jawbone and / or muscle attachment points, etc.

[0109] In some examples, a standard model of how the TMJ operates for a patient can be used. The mandibular prominence (e.g., typically between 2–6 inches, for example, about 4 inches posterior to the molars) can hinge around the TMJ relative to the maxilla. The mandible is typically hinged at the TMJ, allowing it to move relative to the maxilla and thus rotate around the TMJ. Other models of jaw movement and the TMJ can take into account additional (often limited) degrees of freedom of movement with respect to the TMJ. For example, the maxilla's movement relative to the mandible can be considered as ground, and the mandible can move freely relative to the maxilla in a constrained manner; the mandible can have 6 or 7 degrees of freedom, although the total degrees of freedom may be limited or constrained (e.g., by limiting the amount of anteroposterior, lateral, and pivotal opening / closing relative to the maxilla). The methods for modeling the TMJ described herein allow for the determination of the position of the jaws relative to each other. Although the mandible can move independently of or relative to the maxilla, constraints on the TMJ relationship may impede or limit the relative position of the maxilla and mandible. Therefore, knowing the position of the maxilla allows the position of the mandible to be determined within a constrained range of motion; once the maxilla is used as a reference, only 1-2 degrees of freedom (DOF) can indicate the position of the mandible (specifically, the degree of “opening” of the mouth).

[0110] Therefore, in practice, these methods and devices can create 3D models of the mandible and jaw and render one or more combined 2D images of the mandible and jaw relative to a camera (e.g., a set of camera parameters), where the jaws are open / closed to various degrees that can be derived from the TMJ relation. This allows the device to render the mandible and jaw relative to a camera to determine how to adjust the model and / or camera position. Since the position of the mandible is constrained relative to the maxilla, the position of the mandible can be estimated from the position of the maxilla. Similarly, the TMJ relation can be derived by imposing constraints on the position of the mandible relative to the maxilla and solving for the parameters that constrain these constraints using actual images of the patient's mandible and jaw in various positions (e.g., open, closed, or any intermediate position, including positions associated with speaking, eating, etc.).

[0111] For example, TMJ parameters may be related to constraints on the digital model of the mandible and maxilla (and TMJ). In some examples, at least one parameter of the TMJ parameters may include the hinge position between the mandible and maxilla, which may be a fixed distance from the mandible or vary within a limited range. This parameter can be adjusted or optimized. In some examples, in 3D modeling of a combined model including the mandible and maxilla, the hinge / joint region may be fixed relative to the maxilla.

[0112] Digital 3D models of the patient's maxilla and mandible can be acquired separately or together (e.g., via intraoral scan, via dental impression scan, etc.). The location of the temporomandibular joint (TMJ) (e.g., articulated joint) can then be determined and added to the digital model. The location of the TMJ may initially be unknown, but can be assumed using initial (starting) values. These initial values ​​can be initial presets, selected from a database (e.g., a library) of possible values ​​based on one or more patient characteristics (body type, sex, arch size, etc.), or can be solved using approximations based on one or more patient characteristics. The initial set of TMJ parameters can then be fine-tuned to more accurately model the maxilla, mandible, and TMJ.

[0113] For example, a method or device can determine or fine-tune the position of the TMJ hinge based on image or video data, depending on how the patient moves their jaw (opening / closing their mouth). In some examples, the method and / or device can determine the positions of the maxilla, mandible, and TMJ in (virtual) 3D space; a rendered image can be obtained by projecting a 3D model using camera or image plane data (e.g., using camera parameters). The projection from the 3D model using camera parameters can be compared with one or more actual (e.g., “original”) images (e.g., photographs) or image masks (such as segmented image masks as described herein). The difference between the projection (synthetic image) and the mask can be minimized by adjusting one or more parameters, which may be constrained by constraints on TMJ parameters. For example, the difference can be minimized by adjusting the angle between the maxilla and mandible and / or jaw length, etc. The process can be iterative, such that multiple projections can be acquired using camera parameters after adjusting one or more TMJ parameters to determine the TMJ parameters. For multiple open / closed positions, the process can be performed in parallel or sequentially (or in some cases, simultaneously). The difference between the projected image and the actual image can be minimized for a set of parameters until the difference falls below a threshold level (or until the difference fails to converge after a predefined limit). Once the rendered image matches the input (actual) photograph within the difference threshold, the TMJ parameters can be considered solved. Each set of camera parameters and jaw (TMJ) parameters can be optimized separately or simultaneously. The coordinate system may be irrelevant; polar, spherical, cylindrical, and / or Cartesian coordinates can be used. In some cases, the jaw position can be set or specified at least based on the TMJ's angle and location. The TMJ parameter set can be limited by jaw length and / or angular paths.

[0114] Generally, the methods and apparatus described herein can determine the TMJ parameters by solving for them and / or by measuring them from segmented CBCT scans. For example, the TMJ parameters can be measured directly from images (including but not limited to CBCT images, 2D X-ray images, other tissue penetration scans, etc.). These measurements can be used instead of the parameters determined as described herein, or in addition to the parameters determined as described herein.

[0115] In any of the methods and devices described herein, at least one image of the patient's upper and lower jaws taken in both closed and open jaw positions can be used to determine or configure TMJ parameters; in some cases, two or more images for each jaw position can be used, and two or more jaw positions (e.g., open, closed, first intermediate open position, second intermediate open position, etc.) can be used. Multiple camera positions can be used. Once the relationship between the upper and lower jaws (which may be referred to as the TMJ relationship) is determined, parameterization between the upper and lower jaws can be determined, thereby defining constrained degrees of freedom (e.g., 1 or 2 degrees of freedom for TMJ length and angle). Therefore, understanding or determining the position of the mandible relative to the camera and the degree of the patient's occlusal opening or closing (including identifying deep overbite) can be helpful.

[0116] For example, the method described herein may include receiving or capturing one or more actual images of a patient's teeth (including both the maxilla and mandible, or at least parts of the maxilla and mandible) and segmenting these actual images. Segmentation may include forming a segmentation mask. Any suitable technique for segmentation and / or forming the segmentation mask may be used. One or more projected images may be obtained from a 3D model of the patient's dental arch, including the maxilla and mandible, having TMJ parameters or a set of parameters; the images may be rendered using camera parameters (e.g., position, orientation, etc.) that approximate one or more actual images. The projected (e.g., rendered) images may then be compared with the original images directly or preferably by comparison with the segmentation mask. If the comparison results are sufficiently close (e.g., within a threshold range), the TMJ parameters may be considered accurate. If not, the TMJ parameters may be adjusted, a new projection may be obtained, and the process may be repeated until the error between the actual image and the projected image is within a target range (the threshold range). Once within the threshold range, the final registration result (final TMJ parameters) may be output, and a holistic model of the patient's maxilla and mandible and dentition may be used.

[0117] Figure 4This is a flowchart illustrating an example method 400 for determining a match between a 2D projection and a 2D dental image and importing dental information from a 3D alignment model into the corresponding 2D dental image. Some examples may demonstrate the operations described herein with operations added, operations removed, operations in different orders, operations in parallel, and some operations differing. Method 400 will be described below with respect to the computational environment 100 of Figure 1, but method 400 can be performed by any other suitable system or apparatus.

[0118] In this example, method 400 may first receive or acquire a 3D dental model 402 of the patient. The 3D dental model may include partial or complete data from a 3D scan of the patient (received from an intraoral scanning system such as scanning system 154 in Figure 1). In some cases, the data may include segmentation information (e.g., received from a scan segmentation engine 160). The 3D dental model may be associated with a previously determined dental treatment plan for the patient. In some examples, the 3D dental model may include the location of the maxillary portion, mandibular portion, and temporomandibular joint (e.g., TMJ parameters). The segmentation information may include any feasible dental information, such as tooth number, tooth outline, gingival position, etc.

[0119] Next, in box 404, 2D dental images of the patient can be received or acquired. 2D dental images can be taken or captured at a different time relative to the 3D dental model in box 402. For example, 2D dental images may have been captured to track the progress of a dental treatment procedure. 2D dental images can be captured by scanning system 154, a camera, a smartphone, or any other feasible device.

[0120] Next, in box 406, a 3D alignment model is constructed or generated. The 3D alignment model can be generated by the 3D model simulation engine 196. The 3D alignment model can include one or more features of the 3D dental model received in box 402. Constructing or generating the 3D alignment can include positioning the maxillary and mandibular portions relative to each other based on the location of the temporomandibular joint. In some examples, constructing or generating the 3D alignment model can include determining the relationships between the portions of the 3D dental model. For example, the 3D model simulation engine 196 can determine the relationship between the maxillary and mandibular portions of a patient's 3D dental model. In some cases, the 3D model simulation engine 196 can determine the relationship between the maxillary and mandibular portions based on the degrees of freedom between the 3D dental model and 3D space.

[0121] Next, in box 408, the position of the virtual camera is determined. The virtual camera position establishes the relationship between the 3D alignment model and the 2D image. That is, it can be based on the virtual camera (e.g., ...). Figure 3The viewpoint associated with the position of the virtual camera (310) generates a 2D image or projection from the 3D alignment model. By determining the position of the virtual camera, the position of the projection plane (e.g., projection plane 330) can be determined. Generally, the projection plane is perpendicular to the virtual camera and positioned between the camera and the 3D alignment model.

[0122] Next, in box 410, a 2D projection is generated. The 2D projection (sometimes referred to as a 2D alignment projection) can be generated or determined by a 2D projection engine 197. In some implementations, the 2D projection engine 197 can draw rays from the 3D alignment model to the projection plane. The 2D projection engine 197 can determine which parts of the 3D alignment model are visible, or which parts are occluded by elements including gums or any other feasible items.

[0123] Next, in box 412, the registration difference between the 2D projection and the 2D dental image is determined. In some examples, the registration difference can be determined by the registration engine 198. The registration difference can refer to one or more detectable or measurable differences between the 2D projection and the 2D dental image. For example, the registration engine 198 can determine the difference between the tooth contours of corresponding teeth included in the 2D projection and the 2D dental image. Although tooth contours are used as an example here, the registration engine 198 can determine differences between other corresponding items, such as bone, gingiva, etc.

[0124] Next, in box 414, the determined difference is compared to a threshold. For example, this can be done to determine the difference between the contours of corresponding teeth in the 2D projection and the 2D dental image. If the determined difference (e.g., if the measurement between contours) is less than the threshold, then in box 416, information from the patient's 3D alignment model (which may include segmentation information) can be imported or transferred to the patient's 2D dental image. For example, tooth numbering information can be imported from the patient's 3D alignment model into the patient's 2D dental image. The imported dental information can be stored in the registration data storage 199. In some embodiments, the 2D alignment engine 169 can generate a 3D model or image based on the 3D alignment model and / or the 2D dental image. Alternatively or additionally, a 3D model including TMJ parameters that have been fine-tuned by this method can be output; the final 3D model may include maxillary, mandibular, and TMJ relationships (e.g., position).

[0125] Returning to box 414, if the determined difference is greater than a threshold, a new position for the virtual camera and / or a new position for the 3D alignment model is determined in box 418. Since the determined difference is greater than the threshold, the correspondence between the 3D alignment model and the 2D dental image may be significant. To attempt to reduce the difference, the 3D model simulation engine 196 may determine a new 3D alignment model and / or a new virtual camera position. After determining the new 3D alignment model and / or the new virtual camera position, the method returns to box 406 to iteratively determine whether the 2D projection of the new 3D alignment model is similar to (matches) the 2D dental image. In some examples, the alignment model may reference TMJ parameters, such as TMJ position (jaw length, etc.). In any of these methods and devices, two TMJ joints may be used, or a single TMJ joint may be sufficient.

[0126] Any of these methods can include determining a 3D model comprising the patient's maxilla, mandible, and TMJ parameters. The TMJ parameters can be optimized as described herein, and the TMJ parameters can inform how the mandible can move (e.g., hinged at an angle relative to the maxilla, and in two translational directions (e.g., the y and z axes)). The position or localization of the TMJ relative to the maxilla and / or mandible (TMJ parameters) can be determined iteratively. Therefore, the 3D model comprising the maxilla, mandible, and parameters (TMJ parameters) described herein can be used to predict how the mandible can move for a particular patient based on 2D registration as described herein.

[0127] For example, any of these methods and devices may begin with one or more intraoral scans of the patient's maxilla and mandible (upper and lower dental arches). The relationship between the maxilla and mandible may be defined by one or more parameters, including a modeled temporomandibular joint (TMJ). Generally, these methods and devices can determine the estimated location of the modeled TMJ using 2D-to-3D registration of multiple images of the patient's maxilla and mandible. Once identified, the jaw parameters (especially the modeled TMJ) can be used to move the 3D-modeled mandible relative to the 3D-modeled maxilla with limited, constrained degrees of freedom, such as rotation of the mandibular model around the modeled TMJ, and restricted movement in the x, y, and / or z (e.g., y and z in some cases).

[0128] Therefore, the relationship between modeling the maxilla, mandible, and TMJ can be identified as follows: The mandibular position relative to the 'reference' maxilla and the initial position of the modeled TMJ (e.g., a guess) are set; one or more 2D projections including the maxilla and mandible are generated based on this optimal guess; and these one or more 2D projections are compared with the actual 2D image to determine the degree of positional difference. Parameters (e.g., TMJ position and / or constraints, x, y, z constraints, etc.) can be adjusted, and the mandibular position is iteratively changed to better match the actual 2D image until the difference between the 2D projection generated using the parameters and the actual 2D image (in some examples, segmentation can be used for comparison) is below a threshold difference (e.g., convergence within a threshold) or the number of iterations exceeds a maximum threshold. The threshold can be set by the user or the system and can be adjustable. Thus, these methods can determine parameters that act as constraints, indicating how the maxilla and mandible move relative to each other.

[0129] These methods and devices can serve as a simple model of how the hinge of the TMJ operates between the maxilla and mandible (e.g., the TMJ is expected to be located approximately 4 inches behind the mandible). Any suitable model for this hinge operation of the TMJ can be used. These methods generally use the maxilla as a 'reference' around which the mandible can rotate and move within limited x and y movements. This restricts the mandible's degrees of freedom to 2-3 degrees of freedom, which can indicate the position of the mandible when the mouth is open / closed. Therefore, in practice, the methods and devices described herein can receive 3D models of the maxilla and mandible (in some cases, the 3D model can be segmented) and iteratively solve for parameters (e.g., TMJ parameters, including the position / location of the TMJ relative to the maxilla and mandible) by comparing them with 2D images from multiple different camera positions. The TMJ parameters that indicate the position of the TMJ (e.g., a single TMJ in some examples or left and right TMJs in some examples) can be optimized to determine the position of the "hinge" formed by the TMJ, which can be estimated as optimization parameters. Initially, the position of the TMJ can be estimated to use the initial position of the hinge as one of the parameters of the model. Given the hinge (TMJ) position, the method may include moving a virtual model of the mandible relative to the maxilla. The method may include setting a virtual camera relative to the maxilla and mandible in 3D space. The method may also include projecting the 3D model from the camera onto an image plane to generate a rendered 2D image from the 3D model. This projected image can be compared with an original photograph or image mask taken at approximately the same camera position and / or orientation. The difference between the projected image obtained using initial parameters (e.g., hinge / TMJ position) and the 2D image taken at approximately the same camera angle can be used to adjust parameters, including TMJ parameters. After adjusting the parameters, the process can be repeated to move the mandible relative to the maxilla based on the updated / new parameters. After each set of parameter adjustments, the match between the rendered / projected 2D image and the actual 2D image can be improved until the rendered 2D image matches the actual 2D image sufficiently closely (e.g., within the target range), at which point the parameters can be finally determined. Multiple images may be used in any of these methods. In some cases, each set of camera parameters and jaw parameters (e.g., TMJ parameters) can be represented as angles and / or the x, y, z, and open / closed positions of the jaw. The quality of the fit for the parameters (e.g., TMJ position and / or angular range, range of motion, etc.) can be determined as the fit quality (e.g., 'goodness'). Ideally, these methods and devices can determine the best-fit set of parameters (e.g., having the maximum probability of matching a 2D projection from a 3D model with 2D images captured using one or more cameras). In some cases, once the method or device has determined the relationship between the maxilla and mandible (e.g., observing a first image or set of images), the process can be repeated, for example, to reconstruct for a new parameterization.

[0130] As mentioned earlier, comparisons between 2D and 3D images can be based on segmentation or other landmarks between the 2D projected image and the actual 2D image. Once the relationship between the upper and lower jaws is determined by parameters, a 3D model incorporating these parameters can be used to determine the position of the jaws at any virtual location, and consequently, the position of the teeth and gingiva. This is extremely useful for identifying and / or treating dental conditions, including but not limited to open bite and deep overbite conditions.

[0131] Figures 5A to 5C Images related to the open jaw described herein are shown. Figure 5A A 2D dental image 500 of the patient's dentition is shown. Figure 5B A 2D projection of a jaw pair 510 based on a 2D dental image 500 is shown. The jaw pair 510 may include a maxillary portion and a mandibular portion, and in some cases, may be a projection based on a 3D alignment model positioned in a 3D virtual space as described herein. Figure 5C A 3D model 520 based on the 2D projection of the jaw pair 510 is shown. For example, the comparison result of the 2D projection of the jaw pair 510 with the 2D dental image 500 is good (less than the threshold). The 3D model 520 can be generated based on the 2D projection of the jaw pair 510, the 2D dental image 500, or a segmented 3D dental model (not shown).

[0132] Figures 6A to 6C Images associated with the closed jaw described herein are shown. Figure 6A A 2D dental image 600 of the patient's dentition is shown. Figure 6B A 2D projection of a jaw pair 610 based on a 2D dental image 600 is shown. The jaw pair 610 may include a maxillary portion and a mandibular portion, and in some cases, may be a projection based on an alignment model positioned in 3D virtual space as described herein. Figure 6C A 3D model 620 based on the 2D projection of jaw pair 610 is shown. For example, the 2D projection of jaw pair 610 shows good comparison results with the 2D dental image 600 (below a threshold). The 3D model 620 can be generated based on the 2D projection of jaw pair 610, the 2D dental image 600, or a segmented 3D dental model (not shown). Figures 5A to 5C compared to, Figures 6A to 6C This illustrates jaw closure and how jaw closure can affect one or more images.

[0133] Figures 7A to 7C Images associated with tooth numbering based on 2D dental images are shown. Figure 7A A first 2D dental image 700 is shown, in which the tooth number has been determined by machine language operation or program. In some cases, the tooth number may include errors. For example, tooth 26 is incorrectly shown. Figure 7BA second 2D dental image 710 is shown, which includes one or more tooth numbers, which may be imported from a 3D alignment model and imported according to operations such as those described relative to example method 400. Figure 7C A 2D dental image 720 with corrected tooth numbers is shown.

[0134] Figure 8 This is a block diagram of device 800, which may be an example of a device configured to perform one or more of the operations described herein. Device 800 may include a communication interface 820, a processor 830, and a memory 840.

[0135] A communication interface 820, which can be coupled to a network and processor 830, can transmit data to and receive data from other wired or wireless devices, including remote (e.g., cloud-based) storage devices, cameras, processors, computing nodes, processing nodes, computers, mobile devices (e.g., cellular phones, tablets, etc.) and / or displays. For example, the communication interface 820 may include wired (e.g., serial, Ethernet, etc.) and / or wireless (Bluetooth, Wi-Fi, cellular, etc.) transceivers that can communicate with any other feasible device over any feasible network. In some examples, the communication interface 820 may receive previous dental data (including treatment plans) and / or current dental data.

[0136] The processor 830, which is also coupled to the memory 840, may be any one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the device 800 (such as in the memory 840).

[0137] The memory 840 may include an image data storage 842 that can be used for local storage of patient image data. For example, the image data storage 842 may include... Figure 1A Data storage for the segmentation model 167 Figure 1F Registration data storage or any other feasible information

[0138] The memory 840 may also include a non-transitory computer-readable storage medium (e.g., one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard disk, etc.) which may store the partitioning system software 844. The partitioning system software 844 may include program instructions that, when executed by the processor 830, cause the device 800 to perform corresponding functions. Therefore, the non-transitory computer-readable storage medium of the memory 840 may include instructions for performing all or part of the operations described herein.

[0139] Executing the segmentation system software 844 enables the processor 830 to perform operations on the scan segmentation engine 160, the tooth marking engine 166, and / or the 2D alignment engine 169. For example, the processor 830 can execute the segmentation system software 844 to process, acquire, and / or receive 2D and 3D dental images, and segment the dental images to determine individual portions of the patient's dentition, such as individual teeth, gingiva, etc. In some embodiments, executing the segmentation system software 844 can identify and label the patient's maxillary and mandibular portions.

[0140] The processor 830 can execute segmentation system software 844 to locate a 3D dental model in 3D space. Locating the 3D dental model may include determining the relative positions of the upper and lower jaw portions. Executing the segmentation system software 844 can determine the 2D projection associated with any 3D dental model.

[0141] The processor 830 can execute segmentation system software 844 to identify teeth and label them using corresponding tooth numbers. The processor 830 can also execute segmentation system software 844 to determine the 2D projection of the 3D dental model and whether the 2D projection matches the 2D dental image. In some variations, executing segmentation system software 844 can import or transfer data from the segmented 3D dental model to the 2D projection.

[0142] Generally, these methods and devices can be used in one or more parts of a dental computing environment, including intraoral scanning systems, physician systems, treatment planning systems, patient systems, and / or fabrication systems. In particular, these methods and devices can be used as part of a treatment planning system, for example, to generate an accurate digital model of a patient's dentition, from which treatment plans and / or designs for one or more dental appliances to execute those plans can be generated. For example, Figure 9 This diagram illustrates a variant of a computing environment 900 that can generate one or more orthodontic treatment plans specific to a patient and can manufacture dental appliances that can be used to complete the treatment plan for treating the patient under the guidance of a dental professional. Figure 9The example computing environment 900 shown includes an intraoral scanning system 910, a physician system 920, a treatment planning system 930, a patient system 940, an instrument manufacturing system 950, and a computer-readable medium 960. In some variations, the computing environment (dental computing system) 900 may include only one or a subset of these systems (which may also be referred to as subsystems of the entire system 900). Furthermore, one or more of these systems may be combined with or integrated with one or more other systems (subsystems); for example, systems such as the patient system and physician system may be part of a remote server accessible through a physician and / or patient interface. The computer-readable medium 960 may be divided among all or part of these systems (subsystems); for example, the treatment planning system and the instrument manufacturing system may be part of the same subsystem and may be on the computer-readable medium 960. Furthermore, each of these systems may be further divided into subsystems or components, which may be physically distributed (e.g., between a local processor and a remote processor, etc.) or may be integrated.

[0143] An intraoral scanning system may include an intraoral scanner and one or more processors for processing images. For example, an intraoral scanning system 910 may include a lens 911, a processor 912, a memory 913, a scan capture module 914, and a result simulation module 915. Generally, the intraoral scanning system 910 can capture one or more images of a patient's dentition. The intraoral scanning system 910 can be used in a clinical setting (such as a doctor's office) or in an environment of the patient's choice (e.g., the patient's home). In some cases, the operation of the intraoral scanning system 910 may be performed by an intraoral scanner, a dental camera, a mobile phone, or any other feasible device.

[0144] Lens 911 includes one or more lenses and optical sensors to capture reflected light, particularly reflected light from the patient's dentition. Scan capture module 914 may include instructions (such as non-transitory computer-readable instructions) that may be stored in memory 913 and executed by processor 912 to control the capture of any number of images of the patient's dentition.

[0145] As previously mentioned, in some examples, the methods and apparatus described herein for generating 3D models including the maxilla, mandible, and TMJ may be part of, or accessible by, the intraoral scanning system 910, the computer-readable medium 960, and / or the treatment planning system 930.

[0146] For example, the outcome simulation module 915, which may be part of the intraoral scanning system 910, may include instructions for simulating tooth position based on the treatment plan. In some cases, the outcome simulation module 915 may include simulating tooth position based on the temporomandibular joint position (as described above relative to...). Figure 2 and Figure 3 The instructions (described above) are given. The temporomandibular joint allows for the simulation of the mandibular portion relative to the maxillary portion. The mandibular portion can be positioned relative to the maxillary portion using six degrees of freedom.

[0147] Alternatively or additionally, in some examples, the result simulation module 915 can import tooth numbering information from the 3D model onto the 2D image to aid in determining the result simulation, as described above relative to... Figure 4 As stated above.

[0148] Any component system or subsystem of the dental computing environment 900 can access or use the 3D model of the patient's dentition generated by the methods and apparatus described herein. For example, the physician system 920 may include a treatment management module 921 and an intraoral state capture module 922, which can access or use the 3D model including the maxilla, mandible, and TMJ. The physician system 920 can provide a "physician-oriented" interface for the computing environment 900. The treatment management module 921 can perform any operations that enable a physician or other clinician to manage the treatment of any patient. In some examples, the treatment management module 921 can provide visualization and / or simulation of the patient's dentition relative to a treatment plan. For example, the physician system may include a physician-oriented user interface that allows the physician to manipulate the 3D model including the maxilla, mandible, and TMJ, including moving the maxilla and mandible relative to each other based on the TMJ after accurately configuring the patient's TMJ.

[0149] The intraoral state capture module 922 can provide clinicians with images of the patient's dentition via the physician system 920. These images can be captured by the intraoral scanning system 910 and may also include simulated images of tooth movement based on the treatment plan.

[0150] In some examples, the treatment management module 921 may enable the physician to modify or revise the treatment plan, particularly where images provided by the intraoral status capture module 922 indicate that the movement of the patient's teeth may not be in accordance with the treatment plan. The physician system 920 may include one or more processors configured to execute any feasible non-transitory computer-readable instructions to perform any feasible operations described herein.

[0151] Additionally or alternatively, the treatment planning system 930 may include any of the methods and devices described herein, and / or have access to results (e.g., including 3D models of the maxilla, mandible, and TMJ). The treatment planning system 930 may include a scan processing / detailing module 931, a segmentation module 932, a phased module 933, a treatment monitoring module 934, and a treatment planning database 935. Generally, the treatment planning system 930 can determine a treatment plan for any feasible patient. The scan processing / detailing module 931 can receive or acquire dental scans (such as scans from an intraoral scanning system 910) and can process these scans to "clean up" them by removing scan errors and, in some cases, enhancing the detail of the scan images.

[0152] Treatment planning system 930 may include a segmentation system (such as...) Figure 1A (as shown), or may have access to a segmentation system and / or may include Figures 1B to 1F Any engine described in the description (e.g., scan segmentation engine, 3D fusion engine, tooth modeling engine, tooth marking engine, and / or 2D alignment engine).

[0153] The treatment planning system may include a segmentation module 932, which can segment the dental model into individual parts, including individual teeth, gums, jawbones, etc. In some cases, the dental model may be based on scan data from the scan processing / detailing module 931.

[0154] The phased module 933 can determine the different stages of the treatment plan. Each stage can correspond to a different dental appliance. The phased module 933 can also determine the final position of the patient's teeth based on the treatment plan. Therefore, the phased module 933 can determine part or all of the patient's orthodontic treatment plan. In some examples, the phased module 933 can simulate the movement of the patient's teeth according to the different stages of the patient's treatment plan.

[0155] Treatment monitoring module 934 can monitor the progress of orthodontic treatment planning. In some examples, treatment monitoring module 934 can provide clinicians with an analysis of the progress of treatment planning. Orthodontic treatment plans can be stored in treatment planning database 935. Although not shown here, treatment planning system 930 may include one or more processors configured to execute any feasible non-transitory computer-readable instructions to perform any feasible operations described herein.

[0156] The patient system 940 may include a treatment visualization module 941 and an intraoral state capture module 942. Generally, the patient system 940 can provide a "patient-oriented" interface for the computing environment 900. The treatment visualization module 941 allows the patient to visualize the progress of the orthodontic treatment plan and also to visualize predicted outcomes (e.g., the final position of the teeth). In some examples, the treatment visualization module 941 can use the position of the temporomandibular joint to determine the maxillary and mandibular positions, such as relative to... Figure 2 and Figure 3 As described.

[0157] In some examples, the patient system 940 may capture dentition scans for the treatment visualization module 941 via an intraoral state capture module 942. The intraoral state capture module enables the patient to capture their own dentition via an intraoral scanning system 910. Although not shown here, the patient system 940 may include one or more processors configured to execute any feasible non-transitory computer-readable instructions to perform any feasible operations described herein.

[0158] The appliance manufacturing system 950 may include an appliance manufacturing machine 951, a processor 952, a memory 953, and an appliance generation module 954. Generally, the appliance manufacturing system 950 can directly or indirectly manufacture orthodontic appliances to implement orthodontic treatment plans. In some examples, the orthodontic treatment plan may be stored in a treatment planning database 935.

[0159] Appliance manufacturing machine 951 may include any feasible tools or equipment capable of manufacturing any suitable dental appliance. Appliance generation module 954 may include any non-transitory computer-readable instructions that, when executed by processor 952, instruct appliance manufacturing machine 951 to produce one or more dental appliances. Memory 953 may store data or instructions for use by processor 952. In some examples, memory 953 may temporarily store treatment plans, dental models, or intraoral scans.

[0160] Computer-readable medium 960 may include some or all of the elements described herein with respect to computing environment 900. Computer-readable medium 960 may include non-transitory computer-readable instructions that, when executed by a processor, can provide the functionality of any of the apparatus, machine, or module described herein.

[0161] Figure 10 This is a block diagram of an example of a treatment monitoring module 1000. The treatment monitoring module 1000 can be... Figure 9An example of a treatment monitoring module 934 is provided. Treatment monitoring module 1000 may include a treatment plan collection module 1010, a dentition status capture module 1020, an alignment module 1030, and a treatment suggestion module 1040. Generally, treatment monitoring module 1000 enables clinicians to monitor a patient's progress relative to the treatment plan.

[0162] The treatment plan collection module 1010 can retrieve a patient's treatment plan. In some cases, the patient's treatment plan can be stored in the treatment planning database 935 and can be retrieved from it. The dentition status capture module 1020 can capture or acquire any feasible images of the patient's dentition. In some examples, the dentition status capture module 1020 can perform or conduct any operation described relative to the intraoral scanning system 910.

[0163] Alignment module 1030 can align elements or objects from a treatment plan with elements or objects from an image of the patient's dentition. For example, alignment module 1030 can align teeth (teeth positions) described in the treatment plan with teeth captured through image scanning. In some examples, alignment module 1030 can determine the position of the patient's maxilla and mandible relative to the temporomandibular joint (e.g., relative to...). Figure 2 (as described in Figure 4) to determine tooth alignment.

[0164] The treatment recommendation module 1040 can determine whether any changes to the treatment plan may be necessary to achieve a specific final tooth position. For example, any misalignment between the tooth position in the image scan and the tooth position described by the treatment plan (such as a misalignment greater than a preset amount) can indicate an undesirable outcome. The treatment recommendation module 1040 can recommend changes to the treatment plan to address any potential undesirable outcomes. On the other hand, if a misalignment less than a preset amount exists, the treatment recommendation module 1040 may not recommend any changes to the treatment plan.

[0165] This invention provides a framework for jointly simulating the patient's maxilla and mandible, thereby enabling a more comprehensive view of the patient's entire dentition (compared to a occlusal approach) for virtual oral diagnosis and visualization throughout orthodontic treatment.

[0166] As described herein, a system, apparatus, and / or method are presented for providing a fully parametric jaw alignment modeling framework to simulate the relative pose / movement (including individual tooth movement) of a patient's maxilla and mandible. This allows for flexible rendering of a given jaw: panoramic, orthogonal, perspective, etc. The ability to align 3D dentition with 2D images and infer the pose of each tooth / jaw from multi-view patient photographs is also provided. In some cases, leveraging 3D-to-2D alignment can improve tooth number predictions derived directly from ML models.

[0167] Some advantages include leveraging 3D-to-2D alignment, potentially allowing for improved tooth numbering predictions derived directly from ML models. This can be extended to include deformable 3D gingival / tooth models, rather than rigid 3D gingival / tooth objects from 3D scans. Optimizing 3D deformable gingival / tooth models can be particularly useful in restorative cases. These methods may not require a GPU, but GPU acceleration is supported. In some examples, a 3D-to-2D registration process aligns a patient's 3D dentition with its photographs (open / closed occlusion from different perspectives, e.g., anterior / right / left / occlusal surfaces). In other words, starting with an initial dental scan of the patient showing their actual dentition at the time of photographing (if available, otherwise a generic deformable dentition can be used), one or more orthodontic assessment analyses, including (but not limited to) overbite, overjet, gaps, crowding, etc., can be performed easily and quickly. These methods and devices for performing them can take imperfect tooth segmentation / numbering results and achieve reasonable 3D-to-2D alignment accuracy, thus providing the possibility of fine-tuning tooth segmentation / numbering in post-processing steps. Furthermore, these methods and devices can also allow flexible dental rendering, which is particularly useful for visualization and patient education purposes in dental clinics. They can also allow forward rendering to track the mapping between 3D mesh surfaces and corresponding 2D image pixels, thus enabling 2D-to-3D labeling. In addition, they can also allow the generation of synthetic data from 2D renderings of simulated jaw pairs (and possibly even using GAN models) to allow for more possibilities in the training / improvement of various ML models.

[0168] Generally, these methods and devices allow for relatively quick and easy setup. For example, a jaw-pairing simulation framework can include individual tooth models (in the form of a triangular mesh or some parametric form). Each individual tooth can be rotated and / or translated within the corresponding jaw. Furthermore, the tooth models can have deformable shapes. A jaw-pairing simulation framework (e.g., a 3D model) can include gingival mesh objects (in the form of a triangular mesh, possibly with deformable surfaces). In some examples, the maxilla can consist of an upper gingival object and a group of upper teeth. The maxilla can be fixed in space as if it were fixed to the patient's skull. The mandible can consist of a lower gingival object and a group of lower teeth. Mandibular movement can be parameterized in different ways with all six (or fewer) degrees of freedom. As an example, these methods can simplify mandibular movement to a combination of rotational and / or slight translational movements around the temporomandibular joint.

[0169] These devices can be particularly useful for applications including 3D-to-2D registration of the upper and lower jaws, allowing for oral diagnostic measurements directly from 3D models. By jointly simulating the upper and lower jaws and performing a 3D-to-2D registration process to align the 3D jaw pair model with 2D photographs, a baseline 3D jaw pair model of the patient can be determined, enabling direct oral diagnosis on the obtained 3D model.

[0170] Alternatively or additionally, these methods and devices can improve tooth numbering based on 3D-to-2D alignment. Training purely image-based ML models for tooth numbering tasks can generally be challenging, especially with more complex dentitions (tooth loss, tooth eruption, etc.). Therefore, the methods described herein can improve ML tooth numbering models by utilizing 3D dentition information from treatment plans. Figure 7A An example is shown where a pure ML tooth numbering model incorrectly numbers tooth 25 as tooth 26. To correct this incorrect numbering, a 3D dentition of the patient can be obtained from the treatment plan, and the 3D dentition can be aligned with the 2D image (see, for example...). Figure 7B ).

[0171] exist Figure 7B In this case, limited image quality and limited visibility of multiple teeth may lead to relatively poor alignment; correctly allocating 25 teeth is sufficient (because...). Figure 7B The tooth blob "tooth 26" in the treatment plan largely overlaps with the actual tooth 25 in the treatment plan. Therefore, it can be updated. Figure 7A The original tooth number in the data, thus obtaining, for example Figure 7C The correct number is shown.

[0172] These methods and devices can be useful for data synthesis using the jaw pair models described herein. For example, given the jaw pair and the complete jaw movement mechanism (TMJ property) as described above, complete control over the 3D jaw model can be provided, and it is possible to model different degrees of occlusal opening and closing. Therefore, tooth mask images can be generated from different perspectives at different degrees of jaw opening (similar to those described above). Figure 5B and Figure 6B Furthermore, one or more generative adversarial networks (GANs) can be used to generate photorealistic oral images, which could be useful for training various ML models for oral diagnosis, especially when high-quality real-world images are limited.

[0173] Any of the methods and devices described herein may be particularly useful for assisting photo-based orthodontic assessment and / or diagnosis. For example, these methods and devices can calibrate a 3D model of a patient's dentition against one or more 2D images that may be taken later, relative to the initial scan (e.g., an intraoral scan) used to generate the digital 3D model. The 2D images can be used to calibrate the relationship between the mandible and maxilla in the 3D digital model, and the same or different 2D images (including those taken later) can be compared with the 3D digital model to provide useful diagnostic and / or treatment (including treatment monitoring) information. Tooth number

[0174] For example, the methods and devices described herein can be used to determine and / or confirm tooth numbers. Tooth numbering can be an important step toward successful photo-based orthodontic assessment and diagnosis. While existing AI / ML models can provide acceptable predictions for a considerable number of photographs, tooth numbering presents particular challenges for AI / ML models in both tooth instance segmentation and numbering due to a lack of understanding of the patient's 3D dentition.

[0175] The combined jaw registration framework described in this paper provides an example of a method for registering a patient's upper and lower jaw teeth onto a patient photograph. Starting with an initial tooth segmentation and numbering mask (which may be inaccurate), the registration process described in this paper finds a reasonable estimate of both the camera parameters and the relationship between the patient's upper and lower jaw teeth, such that, under the estimated camera parameters, the projections of the patient's upper and lower jaws will be closely aligned with the content in the patient photograph. The alignment between the projection of the 3D model of the patient's teeth and the photograph of the patient's teeth may have slight deviations depending on the accuracy of the initial tooth segmentation / numbering. Therefore, the initial tooth numbering can be compared with the projected numbering from the 3D teeth. For example, a Bayesian tooth numbering framework can be used, and this framework can be used to correct any numbering errors (if any) in the initial tooth segmentation / numbering mask.

[0176] Compared to single-jaw registration procedures that are primarily limited to analyzing only open-jaw photographs, the combined jaw registration procedure described in this paper can be extended to closed-jaw photographs because it includes a lower jaw articulator model (allowing estimation of the relationship between the patient's maxilla and mandible). For example, the registered 3D model (including parameters such as the TMJ parameters defining the limiting movement and / or degrees of freedom as determined above) can provide combined jaw registration with additional cross-jaw references for tooth numbering, including the relationship between the maxillary and mandibular teeth. In contrast, single-jaw methods may only provide the intramaxillary / mandibular tooth relationships separately. Occlusal categories

[0177] The methods and apparatus described herein may be particularly helpful for examining the occlusal relationship between the maxilla and mandible after registration using these methods and apparatus. For example, an important application of the combined occlusal optimization described herein could be dental occlusal monitoring using photographs taken during, before, or after treatment (e.g., 2D images taken by the user and / or dental professionals). Different types of dental occlusion exist, including Class I, II, and III malocclusions, reverse overbite, deep overbite, open bite, etc. To accurately determine the type of occlusion category and measure the degree of malocclusion from photographs, a 3D model must be accurately estimated from the 2D photographs with high precision; therefore, the relative positions of the maxilla and mandible may be particularly important. With a unified camera system, it is easy to measure the distances to reference points on the registered 3D model (including the maxilla and mandible and their possible relative movements (e.g., based on parameters such as TMJ parameters)), thus determining a consistent estimate of the pixel size of the input photographs when measurements are taken on 2D photographs.

[0178] If separate 3D-to-2D registration of the jaws is used (e.g., without registering the upper and lower jaws of the 3D model and determining parameters such as TMJ parameters as described herein), the accuracy of the results can be significantly reduced. For example, converting a mandibular camera system to an maxillary camera system can result in a unified camera system because there is no unique solution for 3D-to-2D registration. However, since this technique typically ignores the mechanical structure connecting the upper and lower jaws, it has additional degrees of freedom regarding how the upper and lower jaws are positioned relative to each other, resulting in a combined system that often fails to reflect the natural setting of the jaws.

[0179] Occlusal category can be determined from a 3D digital model (which has been registered to a 2D image as described herein) by measuring the distance between one or more teeth in the maxilla and one or more teeth in the mandible when modeling a jaw in a specific position (such as when the jaw is closed). For example, in some cases, occlusal category can be measured by taking the position of the maxillary canines relative to the mandibular teeth. The input can be a closed-jaw photograph, in which most of the mandibular teeth are obscured by the maxilla, making mandibular registration difficult or impossible. However, by using joint pair optimization as described herein, where a digital 3D model of the maxilla and mandible is registered using a 2D image, the digital model can include the accurate relative movement of the mandible relative to the maxilla (e.g., using estimated parameters such as TMJ parameters). In this case, the mandible is part of the optimization, and the visible and covered portions of the maxilla and mandible are naturally derived from the setting of the occlusal pair. This results in higher accuracy in estimating the actual 3D model, and therefore higher accuracy in measuring the distance between reference teeth and determining the occlusal category.

[0180] It should be understood that all combinations of the foregoing concepts and the additional concepts discussed in more detail below (provided that these concepts do not contradict each other) are contemplated as part of the inventive subject matter disclosed herein and can be used to achieve the benefits described herein.

[0181] The process parameters and order of steps described and / or illustrated herein are given by way of example only and may be varied as needed. For example, while the steps shown and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order shown or discussed. The various example methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein, or include additional steps beyond those disclosed.

[0182] Any method described herein (including user interfaces) can be implemented as software, hardware, or firmware, and can be described as a non-transitory computer-readable storage medium storing a set of instructions executable by a processor (e.g., a computer, tablet, smartphone, etc.), which, when executed by the processor, causes the processor to control the execution of any of the following steps, including but not limited to: display, communicating with a user, analysis, modifying parameters (including timing, frequency, intensity, etc.), determination, alarm, etc. For example, any method described herein can be executed at least in part by a device comprising one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the procedures of the method.

[0183] While various embodiments have been described and / or illustrated herein in the context of a full-featured computing system, one or more of these exemplary embodiments may be distributed as a program product in various forms, regardless of the specific type of computer-readable medium on which the distribution is actually executed. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batch files, or other executable files that may be stored on computer-readable storage media or in a computing system. In some embodiments, these software modules may configure the computing system to execute one or more of the exemplary embodiments disclosed herein.

[0184] As described herein, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, each of these computing devices may include at least one memory device and at least one physical processor.

[0185] As used herein, the term "memory" or "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, variations or combinations of one or more of these, or any other suitable storage memory.

[0186] Additionally, as used herein, the term "processor" or "physical processor" generally refers to a processing unit of any type or form of hardware implementation capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the aforementioned memory device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), portions of one or more of these, variations or combinations of one or more of these, or any other suitable physical processor.

[0187] Although shown as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. Additionally, in some embodiments, one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, enable the computing device to perform one or more tasks, such as the method steps.

[0188] Additionally, one or more of the devices described herein can convert data, physical devices, and / or representations of physical devices from one form to another. Alternatively or additionally, one or more of the modules described herein can convert a processor, volatile memory, non-volatile memory, and / or any other part of the physical computing device from one form of computing device to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with a computing device.

[0189] As used herein, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmissive media (such as carrier waves) and nontransitory media such as magnetic storage media (e.g., hard disk drives, magnetic tape drives, and floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs), and Blu-ray discs), electronic storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0190] Those skilled in the art will recognize that any process or method disclosed herein can be modified in many ways. The processing parameters and order of steps described and / or illustrated herein are given by way of example only and can be changed as needed. For example, while steps shown and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order shown or discussed.

[0191] The various exemplary methods described and / or illustrated herein may omit one or more steps described or illustrated herein, or may include additional steps in addition to those disclosed. Furthermore, steps of any method disclosed herein may be combined with any one or more steps of any other method disclosed herein.

[0192] The processor described herein can be configured to perform one or more steps of any of the methods disclosed herein. Alternatively or in combination, the processor can be configured to combine one or more steps of one or more methods disclosed herein.

[0193] When a feature or element is referred to herein as being “on” another feature or element, it may be directly located on that other feature or element, and / or there may be intermediate features and / or elements. Conversely, when a feature or element is referred to as being “directly on” another feature or element, there are no intermediate features or elements. It should also be understood that when a feature or element is referred to as being “connected,” “attached,” or “coupled” to another feature or element, it may be directly connected, attached, or coupled to that other feature or element, or there may be intermediate features or elements. Conversely, when a feature or element is referred to as being “directly connected,” “directly attached,” or “directly coupled” to another feature or element, there are no intermediate features or elements. Although described or illustrated with respect to one embodiment, the features and elements thus described or illustrated may be applied to other embodiments. Those skilled in the art will also recognize that references to structures or features positioned “adjacent” to another feature may have portions overlapping with or below the adjacent feature.

[0194] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, as used herein, the singular forms “a” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. It will also be understood that the terms “comprises” and / or “comprising”, when used in this specification, designate the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the terms “and / or” include any and all combinations of one or more of the associated listed items and may be abbreviated as “ / ”.

[0195] For ease of description, spatially related terms such as “under,” “below,” “lower,” “over,” and “upper” may be used herein to describe the relationship of one element or feature to another (or more) elements or features, as shown in the accompanying figures. It will be understood that spatially related terms are intended to include different orientations of the device in use or operation, in addition to those depicted in the figures. For example, if the device in the figures is inverted, an element described as “under” or “below” other elements or features would be oriented as “over” other elements or features. Thus, the exemplary term “under” can include both above and below orientations. The device may be otherwise oriented (rotated 90 degrees or in other orientations), and the spatially relative descriptive terms used herein are interpreted accordingly. Similarly, unless otherwise specifically stated, the terms “upwardly,” “downwardly,” “vertical,” “horizontal,” etc., are used herein for illustrative purposes only.

[0196] While the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context otherwise requires. These terms may be used to distinguish one feature / element from another. Therefore, without departing from the teachings of the invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0197] In this specification and the following claims, unless the context otherwise requires, the word "comprise" and variations such as "comprises" and "comprising" mean that various components may be used together in methods and articles of manufacture (e.g., compositions and devices that include apparatus and methods). For example, the term "comprising" will be understood to imply the inclusion of any of the stated elements or steps, but does not exclude any other elements or steps.

[0198] Generally, any apparatus and method described herein should be understood as inclusive, but alternatively, all or a subset of components and / or steps may be exclusive and may be represented as “composed of” or alternatively “mainly composed of” individual components, steps, subcomponents or substeps.

[0199] As used herein in the specification and claims, including in the examples, unless otherwise expressly stated, all figures may be understood as if they begin with the words “about” or “approximately”, even if the term is not explicitly stated. When describing size and / or location, the phrases “about” or “approximately” may be used to indicate that the described value and / or location is within a reasonably expected range of value and / or location. For example, numerical values ​​may have values ​​of + / -0.1%, + / -1%, + / -2%, + / -5%, + / -10%, etc., of the stated value (or range of values). Any numerical value given herein should also be understood to include approximately or approximately that value, unless the context otherwise requires. For example, if the value “10” is disclosed, “about 10” is also disclosed. Any numerical ranges described herein are intended to include all subranges contained therein. It should also be understood that, as would be appropriately understood by those skilled in the art, when a value is disclosed, the terms "less than or equal to" that value, "greater than or equal to" that value, and possible ranges between the values ​​are also disclosed. For example, if the value "X" is disclosed, then "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in various different formats, and this data represents end points and start points, as well as ranges for any combination of data points. For example, if specific data point "10" and specific data point "15" are disclosed, it should be understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15, as well as ranges between 10 and 15, are considered disclosed. It should also be understood that each unit between two specific units is also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0200] While various illustrative embodiments have been described above, numerous changes may be made to these embodiments without departing from the scope of the invention as described in the claims. For example, in alternative embodiments, the order in which the various described method steps are performed may typically be altered, and in other alternative embodiments, one or more method steps may be skipped together. Optional features of the various apparatus and system embodiments may be included in some embodiments but not in others. Therefore, the foregoing description is provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention as set forth in the claims.

[0201] The examples and illustrations included herein are shown by way of illustration, not limitation, of specific embodiments in which the subject matter can be practiced. As mentioned, other embodiments can be utilized and derived therefrom, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. The term “invention” may be used individually or collectively herein to refer to these embodiments of the inventive subject matter, merely for convenience and not intended to actively limit the scope of this application to any single invention or inventive concept if more than one invention or inventive concept is actually disclosed. Therefore, while specific embodiments have been illustrated and described herein, it is contemplated that any arrangement for achieving the same purpose may substitute for the specific embodiments shown. This disclosure is intended to cover any and all modifications or variations of the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of the foregoing description.

Claims

1. A method comprising: A 3D alignment model is generated based on the patient's 3D dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxillary and mandibular portions; A two-dimensional 2D alignment projection is generated based on the 3D alignment model; Determine the 2D difference between the patient's 2D dental image and the 2D aligned projection; and When the difference is less than a threshold, dental information is imported from the patient's 3D dental scan into the patient's 2D dental image.

2. The method according to claim 1, wherein, Generating the 3D alignment model includes: determining the position of the mandibular portion relative to the maxillary portion.

3. The method according to claim 2, wherein, The position of the mandibular portion is constrained by the position of the joint that connects the mandibular portion to the maxillary portion.

4. The method according to claim 2, wherein, The position of the mandibular portion is determined at least in part by the temporomandibular joint positioned relative to the maxillary portion.

5. The method according to claim 1, wherein, Generating the 3D alignment model includes moving the mandibular portion relative to the maxillary portion.

6. The method according to claim 5, wherein, The movement of the mandibular portion is based on the position of the joint connecting the mandibular portion to the maxillary portion.

7. The method according to claim 1, wherein, Generating the 2D alignment projection includes: determining the position of the projection plane in the public virtual 3D space and the position of the 3D alignment model.

8. The method according to claim 7, further comprising: Dental elements are projected from the 3D alignment model onto the projection plane, wherein the 2D alignment projection is based on the projected dental elements.

9. The method according to claim 7, wherein, The position of the 3D alignment model and the position of the projection plane are at least partially based on the viewpoint associated with the virtual camera set in the public virtual 3D space.

10. The method according to claim 7, wherein, A portion of the projection plane is cropped before determining the 2D difference between the patient's 2D dental image and the 2D aligned projection.

11. The method according to claim 7, wherein, Determining the position of the 3D alignment model includes: locating the patient's maxilla in 3D space using six degrees of freedom, and locating the patient's mandible in 3D space using one degree of freedom.

12. The method according to claim 7, wherein, Generating the 2D alignment projection includes iteratively determining the position of the projection plane based on the difference between the patient's 2D dental image and the 2D alignment projection.

13. The method according to claim 1, wherein, The 2D dental images are based on photographs of the patient's dentition.

14. The method according to claim 1, wherein, The 2D dental image includes 2D information of the maxillary and mandibular regions.

15. The method according to claim 1, wherein, Determining the 2D difference between the patient's 2D dental image and the 2D alignment projection includes: determining the differences between corresponding features of the patient's 2D dental image and the 2D alignment projection.

16. The method according to claim 1, wherein, Determining the 2D difference between the patient's 2D dental image and the 2D alignment projection includes: determining the difference between the contours of corresponding dental structures in the patient's 2D dental image and the 2D alignment projection.

17. The method according to claim 1, wherein, Determining the difference between the patient's 2D dental image and the 2D alignment projection includes: determining the difference between the tooth boundaries determined from the patient's 2D dental image and the 2D alignment projection.

18. The method according to claim 1, wherein, The segmented data is generated using one or more machine learning engines and one or more 3D models.

19. The method according to claim 1, wherein, The dental information includes tooth number information.

20. The method according to claim 1, wherein, Generating the 3D alignment model further includes iteratively determining the position of the mandibular portion relative to the maxillary portion based on the difference between the patient's 2D dental image and the 2D alignment projection.

21. The method according to claim 1, wherein, Generating the 2D alignment projection includes iteratively determining the position of the projection plane based on the difference between the patient's 2D dental image and the 2D alignment projection.

22. The method according to claim 1, further comprising: 3D images are generated based on the 2D alignment projection.

23. The method according to claim 1, wherein, The patient's 2D dental image is a closed jaw photograph.

24. The method according to claim 1, wherein, The patient's 3D dental scan was associated with a previously determined treatment plan.

25. The method according to claim 1, wherein, Importing dental information from the patient's 3D dental scan includes: determining the bite category from the 3D dental scan.

26. The method of claim 25, wherein, Determining the occlusion category from the 3D dental scan includes identifying one or more of the following: Class I malocclusion, Class II malocclusion, Class III malocclusion, reverse overbite, deep overbite, and / or open bite.

27. The method according to claim 1, further comprising: The degree of malocclusion was measured from the patient's 3D dental scan.

28. The method according to claim 1, further comprising: The patient's 3D dental scan was used to determine or correct tooth numbering.

29. A method comprising: Generate or access a three-dimensional 3D alignment model of a digital model of a patient's maxillary portion and a digital model of a patient's mandibular portion, wherein the digital models of the patient's maxillary portion and the digital models of the patient's mandibular portion are based on one or more intraoral scans, wherein the 3D alignment model includes TMJ parameters; Generate a two-dimensional 2D alignment projection image from the 3D alignment model; Determine the difference estimate between the 2D dental image of the patient's teeth and the 2D aligned projection; The TMJ parameters are iteratively adjusted based on the difference estimate, and the steps of generating the 2D aligned projection image and determining the difference estimate are repeated until the difference is less than a threshold or the number of iterations exceeds a second threshold; and The output includes the 3D alignment model with the TMJ parameters.

30. A system comprising: One or more processors; as well as The memory is configured to store instructions that, when executed by the one or more processors, cause the system to: A 3D alignment model is generated based on the patient's 3D dental scan, wherein the 3D alignment model includes segmentation data of the patient's maxilla and mandible; A two-dimensional 2D alignment projection is generated based on the 3D alignment model; Determine the 2D difference between the patient's 2D dental image and the 2D aligned projection; and When the difference is less than a threshold, dental information is imported from the patient's 3D dental scan into the patient's 2D dental image.

31. A non-transitory computer-readable storage medium comprising instructions, said instructions, when executed by one or more processors of a device, causing the device to perform operations including: A 3D alignment model is generated based on the patient's 3D dental scan. The 3D alignment model includes segmentation data of the patient's maxilla and mandible; A two-dimensional 2D alignment projection is generated based on the 3D alignment model; Determine the 2D difference between the patient's 2D dental image and the 2D aligned projection; as well as When the difference is less than a threshold, dental information is imported from the patient's 3D dental scan into the patient's 2D dental image.

32. A method for determining a patient's bite category, the method comprising: Access or receive one or more 2D images of a patient's dentition; Using the one or more 2D images, a 3D digital model of the patient's dentition is registered such that the 3D digital model of the patient's maxilla and the 3D digital model of the patient's mandible are mutually registered, so that the 3D digital model can provide relative movement of the 3D digital model of the mandible relative to the 3D digital model of the maxilla. Determine the patient's occlusal category from the registered 3D digital model; and Output the bite category.

33. The method according to claim 32, wherein, The occlusion category is one or more of the following: Class I malocclusion, Class II malocclusion, Class III malocclusion, reverse overbite, deep overbite, and / or open bite.

34. The method according to claim 32, wherein, Determining the occlusion category includes measuring the degree of malocclusion from the 3D digital model.

35. The method according to claim 32, wherein, Outputting the bite category includes displaying the bite category on the user interface.

Images