Track progress at home using your phone camera

Abstract

Systems and methods for monitoring a subject's teeth during orthodontic treatment are provided herein. A wide angle (e.g., fisheye) image shows an occlusal view of two dental arches of the subject's dentition and can be input into a monitoring system that is configured to compare the image with a treatment plan to determine how closely the current position of the subject's teeth follows the intended or desired positions according to the orthodontic treatment plan.

Description

[0001] Priority declaration

[0002] This patent application claims priority to U.S. Provisional Patent Application No. 62 / 975,148, filed on February 11, 2020, entitled “AT HOME PROGRESS TRACKING USING WIDE ANGLE CAMERA,” the entire contents of which are incorporated herein by reference.

[0003] Incorporation by reference

[0004] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0005] Orthodontic procedures generally involve repositioning an individual's teeth into a desired arrangement in order to correct malocclusion and / or improve aesthetics. To achieve these goals, orthodontic appliances such as braces, shell appliances, etc. may be applied to the individual's teeth by an orthodontist and / or the individual themselves. The appliance may be configured to apply a force on one or more teeth in order to achieve the desired tooth movement according to a treatment plan.

[0006] An orthodontic appliance may include a removable and / or replaceable device on the teeth. An orthodontic appliance may be provided as part of an orthodontic treatment plan. In some orthodontic treatment plans involving removable and / or replaceable appliances, multiple orthodontic appliances may be provided to an individual during the course of treatment to make incremental positional adjustments to the individual's teeth. An orthodontic appliance may have a polymeric trough with an inner cavity shaped to accommodate the teeth and elastically reposition the teeth from one tooth arrangement to an alternate tooth arrangement. An orthodontic appliance may include an "active" region that applies a repositioning force on the teeth and a "passive" region that maintains the teeth in their current state.

[0007] Treatment planning typically uses a 3D tooth model created from a scan or tooth mold of an individual's teeth. The 3D tooth model may include, for example, a raw tooth point cloud, a tooth mesh, or a simplified parametric representation of the 3D teeth.

[0008] During the course of a treatment plan involving a series of dental appliances configured to move the subject's teeth into a desired configuration, it is beneficial to occasionally check the subject's teeth to confirm that they are moving in the intended and / or desired manner according to the treatment plan. Summary of the invention

[0009] Described herein are methods and apparatus that can provide a low-cost and simplified way to confirm a treatment plan at one or more stages of the treatment plan. In particular, these methods and apparatus can provide a patient or caregiver with techniques (e.g., methods) and apparatus to assist in tracking or monitoring a treatment plan without the use of expensive or complex devices. For example, described herein are methods and apparatus that can be used with a patient's or caregiver's personal telephone. In some examples, accessories or other devices can be coupled to the patient's or caregiver's phone; these accessory devices can be adapted for use with software, hardware, or firmware to assist in capturing images of sufficient quality (or to guide a user subject to capture images) so that the images can accurately track the patient's teeth in the treatment plan.

[0010] Of particular interest herein are methods and apparatus for assisting an object, collectively referred to herein as a "user object" (e.g., a patient or a caregiver, e.g., a parent, guardian, etc.) in collecting one or more images of sufficient size and with sufficient information about distance to teeth, etc., so as to capture sufficient dentition so that the dentition can be easily analyzed by one or more automated agents (including machine learning agents, other software, etc.) and / or human agents (e.g., technicians, dental professionals, etc.). Although a user object may be different from a user expert, a user object may be a professional user. For example, a professional user may use the methods and / or apparatus described herein on a patient, thereby acting as both a user object and a professional user.

[0011] In some examples, a user subject can be instructed to use a phone camera to capture multiple images of a patient's teeth and / or mouth. In some examples, the phone camera can be calibrated or set so that the resulting images can be combined to form one or more images of the entire upper jaw, lower jaw, or both, and the phone camera can process the resulting one or more images to identify, record, and / or track the movement of one or more teeth within the upper jaw and / or lower jaw of the subject. In some examples, one or more adapters (e.g., accessories) can be included, which include optical components, such as one or more lenses, etc. In some examples, one or more lenses can be wide-angle (e.g., "fisheye") lenses.

[0012] For example, any of these methods and devices (e.g., systems, apparatus, etc.) can use a wide-angle (e.g., fisheye) imaging system operated by a subject to capture one or more images (or in some cases, only a single image) of the subject's dentition and determine (and / or indicate to the subject) whether the subject needs to see a dental planning provider to modify or adjust the treatment plan.

[0013] Described herein are methods and systems for monitoring the progress of a dental object during a treatment process. At any particular time point during the treatment process, a 3D model of the expected position of the object's teeth at that time point can be timely projected from a 3D model of the object's teeth prepared before treatment begins. During the treatment process, a camera with a wide-angle (e.g., fisheye) lens can be used to capture a two-dimensional (2D) image (including as a single image) of the object's teeth (typically the occlusal surfaces of one or both jaws). The 2D image can include an occlusal view of two arches or a single (e.g., upper or lower) arch of the object's dentition. The 2D image represents the actual position of the object's teeth at a particular point in orthodontic treatment. The 2D image may be referred to herein as a "raw image" or input 2D image.

[0014] The input 2D image can be provided to a monitoring system that is configured to determine camera parameters (also referred to herein as virtual camera parameters) of a camera with a wide angle lens that was used to capture the input 2D image. The virtual camera parameters include intrinsic parameters of the camera (e.g., optical parameters of the camera) and non-intrinsic parameters of the camera (e.g., the position of the camera in space when the image was captured). The non-intrinsic parameters can be determined relative to the teeth. The monitoring system can use these camera parameters to generate a rendered 2D image from a 3D model of the subject's teeth, wherein the 3D model may correspond to the subject's teeth at a specific time or stage of a treatment plan. Typically, this specific time or stage of the treatment plan can correspond to the current actual time or stage that the subject is experiencing when the input 2D image is captured, thereby allowing the expected position of the teeth (from the 3D model) to be compared with the actual position of the teeth (from the input 2D image) during treatment planning. However, in some examples, it may be desirable to compare the current position of the subject's teeth (e.g., from an input 2D image) to one or more other stages of the treatment plan (including an initial position of the teeth); in such cases, the 3D model used to generate the rendered 2D image may instead correspond to another stage of the treatment plan or to the initial position of the subject's teeth prior to initiating the treatment plan.

[0015] The monitoring system can compare the input 2D image with the rendered 2D image to determine how closely the actual or current position of the subject's teeth follows the expected or desired position according to the orthodontic treatment plan. The methods and systems described herein can perform this comparison in a fast and efficient manner by determining the positions of the centers of the teeth in both the input 2D image and the rendered 2D image, and comparing the silhouettes of the corresponding teeth in the input 2D image with the silhouettes of the corresponding teeth in the rendered 2D image. These steps can be performed after segmenting the teeth in the input 2D image and the rendered 2D image. In some examples, the 3D tooth model can include segmentation information for each tooth, which is used to segment the rendered 2D image. The rendered 2D image can be segmented separately from the input 2D image. In some examples, the segmentation of the rendered 2D image (which can be expected to be roughly similar to the rendered 2D image) can be used to help segment the input 2D image.

[0016] The 2D tooth center can be determined from the input 2D image using any appropriate method. For example, the 2D center of the tooth can be identified manually or automatically or semi-automatically. In some examples, machine learning (e.g., forming a trained network) can be used to find the 2D center of the tooth from the occlusal view provided by the wide-angle (e.g., fisheye) image. Similarly, any appropriate technique can be used to determine the tooth center of the rendered 2D image. In some examples, the tooth center of the rendered 2D image can be identified from a 3D model of the teeth before the rendered 2D image is generated using the virtual camera parameters. Alternatively, in some examples, the tooth center of the rendered 2D image can be determined directly from the rendered 2D image. For example, in some examples, the rendered 2D image can be analyzed using the same technique used to determine the tooth center from the input 2D image (e.g., using a trained neural network).

[0017] In some examples, the tooth center from the input 2D image can be compared with the tooth center from the rendered 2D image to provide an estimate of the difference and / or consistency between the tooth center of the rendered 2D image and the tooth center of the input 2D image. The comparison of the tooth centers can provide an estimate of the translational movement of the teeth (e.g., x, y translation of the teeth) between the treatment plan and the actual teeth of the subject. Alternatively or additionally, the comparison of the contour (e.g., silhouette) of the teeth in the rendered 2D image with the contour of the teeth in the input 2D image can provide an estimate of the rotational movement of the teeth (e.g., as the rotation required to align the teeth of the input 2D image with the teeth of the rendered 2D image). In some examples, only the silhouettes of the segmented teeth can be compared to determine both the translational movement and the rotational movement, however, in some examples, it may be beneficial to use the tooth center to quickly and accurately identify the movement to provide an estimate of the tooth movement. As described herein, estimating tooth movement using the tooth center can be done quickly and may require less calculations. In some examples, more accurate tooth movement estimates (including rotation estimates) may not be necessary.

[0018] Any of the methods and systems described herein can use a comparison between an input 2D image (e.g., a wide-angle image of the occlusal surface of the teeth taken based on the subject) and a rendered 2D image to generate one or more scores (difference scores) that are related to the difference in one or more tooth positions between the input 2D image and the rendered 2D image. The one or more scores can indicate the severity of the difference between the input 2D image and the rendered 2D image. For example, the difference score can be a measure of the total deviation in translation and / or rotation of all segmented teeth. In some examples, separate scores can be used to reflect the differences between different teeth. In some examples, separate scores can be used to reflect the difference between translation and rotation. One or more scores can be weighted and / or scaled. For example, the score may emphasize translation more than rotation; the score may emphasize rotation more than translation; the score may emphasize certain teeth more than other teeth (e.g., molars are more important than premolars, more important than incisors, etc.), etc. The scores can be normalized or averaged.

[0019] In some examples, one or more difference scores may be provided to a subject and / or a dental professional ("dental professional user" or simply "professional user"). For example, the method or apparatus may include alerting a dental professional user (e.g., a doctor, dentist, orthodontist, or other dental professional) when one or more difference scores are above one or more thresholds. In some examples, one or more thresholds may be preset or may be set by a professional user. For example, in some examples, a subject may be using an application software (e.g., "app") of a handheld device (e.g., a smartphone) that includes a wide-angle (e.g., fisheye) lens of a camera. The application software may guide the subject to take a wide-angle image and may process the image (locally using one or more processors in the handheld device, or remotely by transferring the image to a remote server). The processed input 2D image may then be analyzed to determine the difference, as described herein, and if the difference score exceeds a notification threshold, the user subject may be instructed to make an appointment with their dental care provider (e.g., professional user) to adjust the treatment plan. In some examples, the difference score may trigger a notification to the subject to wear the appliance more frequently or otherwise adhere to the treatment plan more diligently.

[0020] A comparison between the subject's dentition (e.g., the subject's tooth positions at a current treatment stage) and the expected or predicted tooth positions of the subject's teeth at a particular treatment stage from a 3D model of the subject's teeth can be output as a difference indicator. The difference indicator can be a difference map, a difference score, and / or a set of difference values. The difference indicator can be output by a method or system.

[0021] For example, in any of the methods and systems described herein, one or more differences (e.g., difference indicators) between the input 2D image and the rendered 2D image can provide difference information in place of or in addition to one or more difference scores. For example, the method or system can generate a difference map that visually shows which teeth deviate from the treatment plan and by how much. In some examples, the difference map can be formed as a composite between the input 2D image and the rendered 2D image, highlighting the differences (through one or more of symbols, text, color, etc.).

[0022] In any of these methods and systems, one or more differences (e.g., difference indicators) between the input 2D image and the rendered 2D image may be provided as a set of difference values, such as a list, spreadsheet, data set, etc., providing alphanumeric data summarizing or listing the differences between the input 2D image and the rendered 2D image.

[0023] The difference indicators (e.g., difference maps, difference scores, and / or difference values) may be stored, displayed, and / or transmitted (for storage and / or display), including to a professional user (e.g., a subject's dental provider) or a third party. In some examples, displayed images (including, but not limited to, input 2D images and rendered 2D images) may be adjusted to reduce or remove image distortion caused by the wide-angle camera assembly.

[0024] For example, described herein are methods, including methods for determining progress of a treatment plan for a subject (e.g., determining deviations from a treatment plan). The methods may also be methods for alerting a dental practitioner (a professional user, e.g., a dentist, an orthodontist, etc.) of a deviation from a dental treatment plan for the subject, and / or methods for correcting or updating a dental treatment plan. For example, a method may include: receiving an input 2D image from a wide-angle (e.g., fisheye) camera, the input 2D image including an occlusal view of a first dental arch and / or a second dental arch of a patient's dentition; identifying dental features of teeth in the first dental arch and / or the second dental arch in the input 2D image; determining virtual camera parameters from the input 2D image; receiving a 3D model of the patient's dentition (e.g., at a target treatment plan stage or time); generating a rendered 2D image from the 3D model using the virtual camera parameters; identifying dental features of teeth in the first dental arch and / or the second dental arch in the rendered 2D image; and comparing the dental features of the rendered 2D image with the dental features of the input 2D image. Any of these methods may further include outputting a difference indicator indicating a difference between the subject's current teeth and a planned treatment stage based on a difference between the teeth from the input 2D image and the rendered 2D image.

[0025] For example, identifying tooth features can include applying a trained machine learning model to a 2D fisheye photograph. In some examples, identifying tooth features includes manually identifying tooth features. The tooth features can include tooth centers. The tooth features can include tooth outlines. In some examples, the tooth features provide the position and orientation of the patient's teeth.

[0026] The input 2D image may represent the actual position of the patient's teeth at that particular point in orthodontic treatment (e.g., at a particular stage in the treatment plan), while the rendered 2D image may represent the expected or desired position of the patient's teeth (at a corresponding stage in the treatment plan) based on a synthetic 3D model of the patient's teeth that predicts the position of the teeth at the corresponding stage.

[0027] In some examples, the comparison indicates how closely the patient's teeth are following the orthodontic treatment plan.

[0028] As mentioned, any of these methods can include updating the orthodontic treatment plan based on the comparison. For example, the updating can include calculating the new tooth movements required to move the patient's teeth from the current position to the desired position. In some examples, the updating can include updating the orthodontic treatment plan using these new tooth movements.

[0029] As described herein, a method (e.g., a method for determining progress in a treatment plan for a subject and / or deviation from a treatment plan, a method for alerting a dental practitioner of a deviation from a dental treatment plan, and / or a method for correcting or updating a dental treatment plan) may include: receiving an input 2D image of the subject's teeth, the input 2D image comprising a wide-angle occlusal view of the subject's upper jaw and / or lower jaw at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's upper jaw and / or lower jaw at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining a difference between teeth from the input 2D image and the rendered 2D image using tooth centers; and based on the difference between teeth from the input 2D image and the rendered 2D image, outputting a difference indicator indicating a difference between the subject's current teeth and the planned treatment stage.

[0030] In some examples, a method as described herein may include: receiving an input 2D image of a subject's teeth, the input 2D image including a wide-angle occlusal view of the subject's upper jaw and / or lower jaw at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's upper jaw and / or lower jaw at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining a translation difference between the teeth from the input 2D image and the rendered 2D image using tooth centers; determining a rotation difference between the teeth from the input 2D image and the rendered 2D image using a silhouette of the teeth; and outputting a difference indicator that indicates a difference between the subject's current teeth and the planned treatment stage.

[0031] Thus, any of these methods may include determining one or both of a translational difference and / or a rotational difference of a tooth. For example, any of these methods may include determining a rotational difference between teeth from an input 2D image and a rendered 2D image using a silhouette of the tooth, wherein a difference indicator is also based on the rotational difference. The rotational difference and / or translational difference between teeth from the 2D image and the rendered 2D image may be determined in whole or in part by applying a trained machine learning model to the input 2D image and the rendered 2D image. Any of these methods may include receiving an input 2D image, including directing a subject to take a wide angle occlusal image. The difference indicator may be one or more of: a difference map, a difference score, and / or a difference value set.

[0032] In general, the input 2D image can be used to determine virtual camera parameters, particularly non-intrinsic camera parameters. In some examples, the input 2D image can be used to determine intrinsic camera parameters. Alternatively, in some examples, some or all of the intrinsic camera parameters can be preset or predetermined (e.g., factory settings, input by camera and / or lens vendors, etc.). In some examples, the virtual camera parameters include parameters that can be iteratively determined from the input 2D image.

[0033] The methods and devices described herein can operate based on a single input 2D image (e.g., a single image captured by a wide-angle (e.g., fisheye) lens). In some examples, receiving an input 2D image of the subject's teeth can include receiving a fisheye view of both the subject's upper jaw and lower jaw. The wide-angle image can be a fisheye image. For example, the wide-angle image can have a field of view covering up to 160 degrees or more (e.g., up to 170 degrees or more, up to 180 degrees or more, up to 190 degrees or more, up to 200 degrees or more, up to 210 degrees or more, up to 220 degrees or more, up to 230 degrees or more, up to 240 degrees or more, up to 250 degrees or more, up to 260 degrees or more, up to 270 degrees or more, etc.). The wide-angle image can include an occlusal (or occlusal perspective) view of all of the subject's teeth for both the upper jaw and lower jaw. This can advantageously allow a single image to be processed as described herein. In some examples, the combination of wide angle images and the use of tooth centers to determine translational differences and tooth silhouettes to determine rotational differences allows for extremely rapid and accurate determination of dental structural differences between actual tooth positions and predicted (e.g., proposed or modeled) tooth positions, thereby requiring less processing power and operating more robustly on input 2D images generated from the subject.

[0034] Any of the methods described herein may include determining a difference score. The difference score may be based on a difference indicator. Any of these methods may also include alerting the subject and / or the subject's dental care provider if the difference score exceeds a threshold. The subject's dental care provider may set or approve the threshold.

[0035] In some examples, the methods described herein may further include updating the orthodontic treatment plan based on the difference indicator.Alternatively or additionally, the methods described herein may include using the difference indicator to calculate new tooth movements required to move the subject's teeth from a current position to a desired position, and / or updating the orthodontic treatment plan using these new tooth movements.

[0036] Also described herein are systems configured to perform any of the methods described herein. For example, a system may include a non-transitory computing device readable medium having stored thereon instructions for tracking patient teeth during orthodontic treatment planning, wherein the instructions are executable by a processor to cause the computing device to: receive an input 2D image from a wide-angle (e.g., fisheye) camera having an occlusal view of a first dental arch and a second dental arch of a patient's dentition; identify dental features of teeth in the first dental arch and the second dental arch in the input 2D image; determine virtual parameters of the wide-angle camera corresponding to the input 2D image; receive a 3D model of the patient's dentition; generate a rendered 2D image from the 3D model using the virtual parameters of the wide-angle camera; identify dental features of teeth in the first dental arch and the second dental arch in the rendered 2D image; compare the dental features of the rendered 2D image with the dental features of the input 2D image.

[0037] Any method and device described herein may include one or more calibration steps. For example, a printed calibration standard (e.g., a calibration standard pattern or target, such as a checkerboard pattern) may be provided to a user object (patient, caregiver, etc.) and / or the user object (patient, caregiver, etc.) may be instructed to print out the calibration standard (using a home printer). The calibration standard may be included in the package. In some examples, the package of a camera or a camera accessory (e.g., an adapter, etc.) for fixing all or part of an optical device (e.g., a wide-angle lens) may be configured as a calibration fixture, which may position the camera in a predetermined position relative to the calibration standard (optionally, using an adapter). The user object may then take several images (and may be instructed by, for example, control software on the user object's phone) to take images of the calibration standard, and then the method or device may use these images to calibrate the user object's camera (including a camera phone). In some examples, the device or method may apply calibration logic to analyze one or more images of the calibration standard and determine the distance to the camera, etc.

[0038] For example, any system described herein may include: one or more processors; a camera; a wide-angle lens; and a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: receiving an input 2D image of teeth of a subject, the input 2D image comprising a wide-angle occlusal view of the subject's maxilla and / or mandible at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining a difference between teeth from the input 2D image and the rendered 2D image using tooth centers; and outputting a difference indicator based on the difference between teeth from the input 2D image and the rendered 2D image, the difference indicator indicating a difference between the subject's current teeth and the planned treatment stage. The systems may also be configured to cause the computer-implemented method to perform any of the steps described above.

[0039] In some examples, the system is configured to include or operate on / with a handheld computing and / or communication device (e.g., a smartphone, tablet computer, etc.). Any of the above steps can be performed locally (e.g., in a handheld computing device), or they can be divided between a local processor and a remote processor. Remote processing can be performed on a remote server (e.g., a cloud-based server), etc.

[0040] In any of the methods and devices described herein, a camera can be registered to the patient's jaw and / or teeth as described herein. After the camera is registered to the entire jaw, the method or device can register each tooth individually, as described above, and by doing so, this information can be used to detect the movement of a single tooth to the entire jaw. For example, the registration can be detected by matching a tooth silhouette projection with the silhouette of the tooth on the image. In some examples, the registration can be detected by, for example, matching a tooth cusp and / or one or more Fisher projections with an image tooth cusp and / or Fisher projection. In some examples, the registration can be detected by training a network that takes as input a tooth image and a tooth depth map from the same camera and produces tooth movement. Any combination of these can be used. In general, the registration from one or more images can be completed from multiple images and / or form a short video shot.

[0041] For example, the method described herein includes: calibrating a camera of a user object of a phone of the user object; receiving multiple input 2D images of the object's upper jaw and / or lower jaw taken using the calibrated camera of the user object; determining virtual camera parameters corresponding to the multiple input 2D images; aligning the camera to the object's upper jaw and / or lower jaw; receiving a 3D model of the object's upper jaw and / or lower jaw at a planned treatment stage; determining translation differences and / or rotation differences between individual teeth from multiple 2D images and 3D models by individually aligning the teeth of the object's upper jaw and / or lower jaw; and outputting a difference indicator, which indicates the difference between the object's current teeth and the planned treatment stage.

[0042] The multiple input 2D images may be combined into a single image (eg, a merged image).Calibrating a camera for a user object may include determining virtual camera parameters.

[0043] Any of these methods may include generating a rendered 2D image of the subject's teeth from the 3D model using virtual camera parameters from the user subject's camera. The rendered 2D image may be used for comparison with the plurality of 2D images (or merged 2D images).

[0044] The methods described herein can determine translational differences and / or rotational differences between teeth from an input 2D image and a 3D model by registering individual teeth of an upper and / or lower jaw of a subject. For example, registering individual teeth can include matching tooth silhouette projections between a rendered 2D image and the input 2D image. In some examples, registering individual teeth includes matching cusps and / or Fisher projections between the rendered 2D image and the input 2D image. In some examples, registering individual teeth includes using a machine learning agent to determine differences between teeth from a 2D image and a rendered 2D image.

[0045] Any of these methods may include instructing the user subject to print a calibration pattern and taking one or more images of the calibration pattern using a camera of the user subject. These images may be used to calibrate the camera and / or the image, including determining virtual camera parameters and / or determining the distance to the camera. The calibration pattern may include a checkerboard pattern or a grid pattern.

[0046] The methods may include calibrating a camera by generating an input 2D image of the subject's teeth.

[0047] Any of the methods described herein may be performed by a system configured to perform the methods. For example, a system may include: one or more processors; and a memory coupled to the one or more processors, the memory being configured to store computer program instructions, the computer program instructions, when executed by the one or more processors, performing a computer-implemented method, the method comprising: calibrating a camera of a user subject of a user subject's phone; receiving a plurality of input 2D images of the subject's upper jaw and / or lower jaw taken using the calibrated camera of the user subject; determining virtual camera parameters corresponding to the plurality of input 2D images; aligning the camera to the subject's upper jaw and / or lower jaw; receiving a 3D model of the subject's upper jaw and / or lower jaw at a planned treatment stage; determining translation differences and / or rotation differences between individual teeth from the plurality of 2D images and the 3D model by individually aligning the teeth of the subject's upper jaw and / or lower jaw; and outputting a difference indicator indicating a difference between the subject's current teeth and the planned treatment stage.

[0048] Other examples of techniques and systems that may benefit from the methods and apparatus described herein may be found, for example, in U.S. Patent Application No. 16 / 370,788, filed on March 29, 2019 (entitled “PHOTOGRAPH-BASED ASSESSMENT OF DENTALTREATMENTS AND PROCEDURES”), which is a continuation of U.S. Patent Application No. 14 / 831,548, filed on August 20, 2015 (entitled “PHOTOGRAPH-BASED ASSESSMENT OF DENTALTREATMENTS AND PROCEDURES”), which was published on April 2, 2019 as US10,248,883. Each of these applications is incorporated herein by reference in its entirety.

[0049] In general, the methods and apparatus described herein can be implemented at very low cost and complexity, allowing subjects to be monitored at home without the need for a dentist or expensive scanning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The novel features of the present invention are particularly set forth in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by referring to the following detailed description and accompanying drawings which set forth exemplary embodiments in which the principles of the present invention are used, and which are as follows:

[0051] Figure 1Ais a diagram illustrating an example of a computing environment configured to monitor teeth of a subject during orthodontic treatment planning.

[0052] Figure 1B is a diagram illustrating an example of one or more 2D graphics engines.

[0053] Figure 1C is a diagram illustrating an example of one or more 3D model engines.

[0054] Figure 1D is a diagram illustrating an example of one or more dental comparison engines.

[0055] FIG. 2A to FIG. 2D Shown is segmented ( Figure 2B ) and tooth features have been extracted ( Figure 2C to Figure 2D ) of the input 2D image ( Figure 2A ) example.

[0056] Figure 3A is an example of a rendered 2D image from a 3D model.

[0057] Figure 3B is an example of a rendered 2D image where tooth features have been extracted.

[0058] Figure 4A is an example of a method for monitoring a subject's teeth during orthodontic treatment planning.

[0059] Figure 4B is another example of a method for monitoring a subject's teeth during orthodontic treatment planning.

[0060] Figure 4C is another example of a method for monitoring a subject's teeth during orthodontic treatment planning.

[0061] Figure 5 is an example of a schematic diagram of a system for determining a difference between a subject's actual tooth position and an expected or predicted position of the subject's teeth during treatment planning.

[0062] Fig. 6A is an example of a subject being guided to take wide angle images of their upper and lower dental arches (input 2D images), showing an occlusal view of the teeth.

[0063] Figure 6B Shown is an example of a wide angle bite perspective view of teeth that may be used as a 2D input image as described herein.

[0064] Figure 7 An example of a wide-angle (eg, fisheye) lens for a smartphone as described herein is shown.

[0065] Fig. 8AAn example of calibration as described herein is shown.

[0066] Figure 8B One example of calibration using a calibration target on packaging that may be included with the product is shown.

[0067] Figure 8C An example of a package that is also configured as a calibration fixture is shown.

[0068] Fig.8D One example of a calibration target including a code or link (eg, a QR code) is shown. DETAILED DESCRIPTION

[0069] Described herein are apparatus (e.g., systems, computing device-readable media, devices, etc.) and methods for monitoring, analyzing, correcting, and / or tracking the progress of orthodontic treatment of a subject. The apparatus and methods described herein may capture or receive a 2D image of a subject's teeth taken at a wide angle (e.g., fisheye) as input (input 2D image), and process the input 2D image to compare it with a model of the subject's dentition (e.g., a 3D digital model) to determine the difference between the actual tooth structure (from the input 2D image) and the predicted or expected tooth structure (from the 3D model) at one or more stages of the treatment plan. The 3D model may represent the structure of the subject's teeth at a particular stage of the treatment plan; in some examples, the stage may correspond to the stage to which the subject's teeth are currently expected to be positioned. The 3D digital model may be generated at the beginning of the treatment plan and may be used to design and manufacture a series of orthodontic appliances (e.g., braces).

[0070] The methods described herein can use the input 2D image and / or predetermined values ​​to determine virtual camera parameters (e.g., intrinsic virtual camera parameters and non-intrinsic virtual camera parameters) of a camera that captured the wide-angle input 2D image. The virtual parameters can then be used to generate a rendered 2D image corresponding to the input 2D image from a 3D tooth model of the subject's teeth for comparison. The rendered 2D image can be compared to the input 2D image to generate a difference indicator such as a difference map, a difference score, and / or a set of difference values. In some examples, the comparison can determine how closely the subject's teeth follow a desired orthodontic treatment plan.

[0071] The apparatus and / or methods described herein can be used for planning and manufacturing dental appliances, including the elastic polymer positioning appliances described in detail in U.S. Patent No. 5,975,893 and published PCT application WO98 / 58596, which are incorporated herein by reference for all purposes. Systems of dental appliances using the technology described in U.S. Patent No. 5,975,893 can be purchased from Align Technology, Inc. of San Jose, California, under the trade name "Invisalign System".

[0072] Throughout the text describing the embodiments, the use of the terms "orthodontic appliance", "appliance" or "dental appliance" is synonymous with the use of the terms "appliance" and "dental appliance" in dental applications. For clarity, the examples are described below in the context of the use and application of an appliance, more specifically a "dental appliance".

[0073] As used herein, a "subject" (or alternatively and equivalently, an "individual") can be any object (e.g., human, non-human, adult, child, etc.), and can alternatively be a patient, a subject receiving treatment, etc. A subject can be a medical patient. An individual or subject can include a person receiving orthodontic treatment (including orthodontic treatment using a series of orthodontic appliances).

[0074] The devices and / or methods (e.g., systems, devices, etc.) described below can be used with and / or integrated into orthodontic treatment plans. The devices and / or methods described herein may include segmenting individual teeth from a three-dimensional model (e.g., a 3D mesh model or a 3D point cloud). The methods and devices described herein can segment 2D images. Segmentation information can be used to measure, compare, and process 2D images. Individual teeth can be segmented automatically (e.g., using a computing device), manually, or semi-automatically. For example, a computing system can automatically perform segmentation by evaluating data (e.g., a three-dimensional scan or a dental impression) of an individual's teeth or dental arches. Segmentation can be performed using a trained network (e.g., as part of a machine learning technique). One or more data structures (e.g., a database) can be used to assist in segmenting a 2D image or 3D model. In some examples, the methods and devices described herein can use segmentation information from a 3D model of a patient's teeth that is input or received. Therefore, segmentation information of teeth from a 3D model can be provided as an input and can be used to analyze an input 2D image, as well as any 2D image generated from a 3D model of the subject's teeth.

[0075] For example, a 3D model of the subject's teeth may be initially generated from an intraoral scanner that scans an individual's dental arch to generate a virtual 3D model of the dental arch. During an intraoral scanning procedure (also referred to as a scanning session), a professional user of the intraoral scanner (e.g., a dentist) may generate multiple different images (also referred to as scans or medical images) of a tooth part, a model of a tooth part, or other object. These images may be discrete images (e.g., point-and-shoot images) or frames from a video (e.g., continuous scanning). The intraoral scanned images may be used to generate a 3D model that may be modified to form a 3D model of all or part of an orthodontic treatment plan ("treatment plan") having multiple stages for moving teeth to a desired configuration (including a final (e.g., aligned) configuration. Although the methods and apparatus described herein may be combined with (and may be received as input) intraoral scan information, the apparatus and methods described herein may be performed without the need for the use of an intraoral scanner. In particular, the methods and apparatus may be operated by the subject in a home environment, thereby allowing the subject to assist in monitoring, tracking, and updating progress in a robust and inexpensive manner.

[0076] Figure 1A 1 is a diagram illustrating an example of a computing environment 100A configured to facilitate the collection and processing of digital scans of a dental arch having teeth therein. The environment 100A includes a computer-readable medium 152 configured to receive a 3D model of a subject's teeth at one or more treatment stages 154, and receives or includes a wide-angle camera assembly 155 and a treatment monitoring system 158, the wide-angle camera assembly 155 including a camera 188 having one or more wide-angle (e.g., fisheye) lenses 157 or coupled to one or more wide-angle (e.g., fisheye) lenses 157. The system can be configured to output 156 images, text, data, and / or alerts. The output 156 can be part of the treatment monitoring system, or it can be separate from the treatment monitoring system. The treatment monitoring system can include one or more modules (e.g., engines, data structures, etc.) for processing and comparing 2D representations and 3D representations of the subject's teeth. One or more modules in the computing environment 100A can be coupled to each other or to modules not explicitly shown.

[0077] The computer-readable medium 152 and other computer-readable media discussed herein are intended to represent various potential applicable technologies. For example, the computer-readable medium 152 can be used to form a network or a portion of a network. When two components are co-located on a device, the computer-readable medium 152 can include a bus or other data pipeline or plane. In the case where a first component is co-located on a device and a second component is located on a different device, the computer-readable medium 152 can include a wireless or wired back-end network or LAN. If applicable, the computer-readable medium 152 can also include a WAN or other network related portion.

[0078] The 3D model of one or more treatment plan stages 154 can be received from a scanning system, a database, a computer system, or any other structure. The 3D model of one or more treatment plan stages can be a 3D model of the entire treatment plan or a subset of the treatment plan. The 3D model can include a treatment plan for an upper dental arch and a lower dental arch. As used herein, a "dental arch" may include all or at least a portion of an individual's dentition formed by an individual's maxillary teeth and / or mandibular teeth. The dental arch may include one or more maxillary teeth or mandibular teeth of an individual, for example, all teeth on the upper or lower jaw of an individual. The 3D model 154 can be stored in a memory, a database, and / or one or more processors, and can be used to generate a rendered 2D image of the object's teeth and provide additional information about the object's teeth (including tooth segmentation, etc.). The 3D model can be associated with a specific object. In some examples, the system may include a verification engine (not shown) that can confirm or verify that the 3D model of one or more treatment plan stages corresponds to a specific object for which an input 2D image was taken. One or more 3D tooth models may include, for example, one or more 3D point clouds or 3D tooth meshes. The treatment monitoring system 158 may be configured to receive 3D model data previously acquired or acquired by another system.

[0079] The wide-angle camera assembly 155 may include a computer system or camera 188 configured to obtain one or more wide-angle 2D images of the subject's dentition, for example, operating with a wide-angle lens 157. The wide-angle lens may be part of the camera or may be added to the camera. The wide-angle camera assembly may be part of the treatment monitoring system 158, or it may be used with the treatment monitoring system 158. As described herein, the wide-angle 2D images may include ultra-wide-angle 2D images produced with a wide-angle (e.g., fisheye) camera or wide-angle (e.g., fisheye) lens, where the subject may have a convex, non-rectilinear appearance. The wide-angle 2D images of the subject's dentition may include, for example, an occlusal view (or a perspective occlusal view) of two dental arches of the subject's teeth in a single 2D image. The wide-angle camera assembly 155 may be, for example, a stand-alone camera or lens, or alternatively a fisheye lens attachment to a smartphone, tablet, or computer. The subject may use the wide-angle camera to capture wide-angle input 2D images of the subject's teeth (e.g., upper and / or lower dental arches) during the orthodontic treatment process.

[0080] Output 156 may include a computer system configured to display at least a portion of the individual's dentition, and / or the difference between an input 2D image (e.g., at a current treatment stage) and one or more treatment plan stages modeled corresponding to the current treatment stage. Output 156 may include a display system, a memory, one or more processors, etc. The output may be part of a display device to display the individual's dentition. Output 156 may be implemented as part of a computer system, a dedicated intraoral scanner, etc. In some implementations, output 156 helps to display the individual's dentition by using scans taken at an earlier date and / or at a remote location. Output 156 may help to display scans taken simultaneously and / or locally. Output 156 may be configured to display the expected or actual results of an orthodontic treatment plan that is applied to a dental arch scanned by scanning system 154. These results may include a 3D virtual representation of a dental arch, a 2D image or rendering of a dental arch, etc. In some examples, the output can be sent to the subject's dental care provider (e.g., an orthodontist, dentist, odontologist, etc.) as one or more images, as text (e.g., a text file listing the differences), as a score (e.g., a numeric or alphanumeric score), as a 3D model or a modified 3D model, etc.

[0081] The treatment monitoring system 158 may include a computer system including a memory and one or more processors, as described herein, configured to monitor the subject's dentition and track the movement of the subject's teeth to the orthodontic treatment plan. For example, the treatment monitoring system may include multiple modules or engines for determining the difference between the patient's actual tooth structure and the planned tooth structure at one or more treatment stages. These modules or engines may correspond to one or more processors. One or more 2D image engines 160 may implement an automated agent to receive an input 2D image of the subject's teeth from a wide-angle camera. The input 2D image may include, for example, an ultra-wide-angle image that includes an occlusal view (or occlusal perspective view) of two dental arches of the subject's dentition. The input 2D image may represent the actual position of the subject's teeth at a specific time or stage of orthodontic treatment. One or more 2D image engines 160 may also be configured to process the input 2D image to extract and identify the tooth features of the subject's teeth. The tooth features may include, for example, tooth centers, tooth silhouettes, tooth landmarks (e.g., pits, cracks and / or peaks) or any other tooth features that provide the position and orientation of the subject's teeth. In some examples, one or more 2D image engines 160 may also be configured to determine virtual camera parameters of the wide-angle camera assembly 155 corresponding to the input 2D image. Alternatively, in some examples, a separate analysis engine (e.g., a processor) for determining virtual camera parameters (e.g., a virtual camera parameter engine) may be included. The virtual camera parameters may include, for example, intrinsic camera parameters and / or non-intrinsic camera parameters of the camera assembly corresponding to the input 2D image.

[0082] One or more 3D model engines 162 can implement one or more automated agents to receive and process scan data or 3D tooth model data from the scanning system 154. In some examples, the treatment monitoring system 158 is configured to receive, store and process the 3D model 154 of the subject's dentition. The 3D model can include, for example, a 3D tooth point cloud or a 3D tooth mesh model, or a series of 3D tooth point clouds or 3D tooth mesh models, each model corresponding to a treatment stage; or, the 3D model can include a segmented 3D tooth point cloud or 3D tooth mesh model of the teeth and a description of the arrangement or structure of the segmented teeth (in some examples, gums, dental arches, etc.) and a guide to the key positions of each stage. The treatment monitoring system 158 is generally configured to generate a rendered 2D image from the 3D tooth model using virtual camera parameters of the camera assembly. Therefore, the rendered 2D image represents the expected position or desired position of the subject's teeth at a specific stage or time of orthodontic treatment; the stage or time can correspond to the actual time or stage of the input 2D image. The one or more 3D model engines 162 can also be configured to process the rendered 2D image to extract and identify the dental features of the subject's teeth. The tooth features may include, for example, tooth centers, tooth silhouettes, or any other tooth features that provide the position and orientation of the subject's teeth, as described above, for comparison with the tooth features from the input 2D image.

[0083] One or more tooth comparison engines 164 may implement one or more automated agents to compare the rendered 2D image from the 3D model with the input 2D image from the wide-angle camera assembly. The comparison may provide an indication of whether the current position of the subject's teeth follows the expected position or the desired position according to the orthodontic treatment plan. In some examples, the tooth comparison engine provides a difference score (also referred to herein as a comparison value), which may indicate how close the current position of the subject's teeth is to the expected position; or, the score may indicate how different the current position is from the model at a particular treatment stage. The tooth comparison engine may generate one or more difference indicators or one or more comparison indicators. The difference indicator may be one or more of the following: a difference map, a difference score, and / or a difference value set. For example, a difference map may include a 2D or 3D representation of the subject's teeth that shows, highlights, or otherwise indicates the difference between the current actual tooth structure and the expected or predicted tooth structure from the treatment plan. The difference map may be one or more color 2D images or labeled 2D images showing position and / or rotation differences, or a 3D model showing position and / or rotation differences. The 2D image or 3D model may be annotated with the differences (including the magnitude of the differences). The markings may include a heat map representation of the differences. In some examples, the set of difference values ​​may be a textual representation of the differences (and / or similarities) between the current tooth configuration and the treatment plan. The textual representation may be a database (e.g., a table, a list, or any other data structure).

[0084] In some examples, the one or more tooth comparison engines 164 determine and calculate new tooth movements that can be applied to the subject's teeth to achieve the desired position. The one or more tooth comparison engines 164 can also output a difference indicator to the subject (e.g., via a message, an alarm, etc.) and / or the subject's dental care provider and / or a third party. In some examples, the one or more tooth comparison engines 164 are configured to trigger one or more alarms when the magnitude of the difference between the actual configuration and the treatment plan exceeds a threshold.

[0085] One or more optional treatment modeling engines 166 can be configured to store and / or provide instructions to implement the orthodontic treatment plan and / or the results of the orthodontic treatment plan using the new tooth movement. One or more optional treatment modeling engines (e.g., machine learning engines) 166 can provide the results of the orthodontic treatment plan on the updated 3D tooth model. In some examples, the updated 3D tooth model can include the new tooth movement. One or more optional treatment modeling engines 166 can model the results of applying orthodontic braces to an individual's dental arch during the orthodontic treatment plan.

[0086] As used herein, any "engine" may include one or more processors or a portion thereof. A portion of one or more processors may include some parts of hardware that are less than all of the hardware that includes any given one or more processors, such as a subset of registers, a portion of a processor dedicated to one or more threads of a multithreaded processor, a time slice (during which the processor is dedicated in whole or in part to executing some functions of the engine), etc. Therefore, the first engine and the second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on the specific implementation or other considerations, the engine may be centralized, or its functions may be distributed. The engine may include hardware, firmware, or software contained in a computer-readable medium for execution by a processor. The processor converts data into new data using the implemented data structures and methods, as described with reference to the accompanying drawings herein.

[0087] The engine described herein, or the engine by which the systems and devices described herein can be implemented, can be a cloud-based engine. As used herein, a cloud-based engine is an engine that can run applications and / or functions by using a cloud-based computing system. All or part of the application and / or function can be distributed across multiple computing devices and need not be limited to only one computing device. In some examples, a cloud-based engine can execute functions and / or modules that an end user accesses through a web browser or container application without having to install these functions and / or modules locally on the end user's computing device.

[0088] As used herein, "datastores" may include repositories having any applicable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. For example, a data store may be implemented as software contained in a physical computer-readable medium on a specific-purpose machine, in firmware, in hardware, a combination thereof, or in an applicable known or convenient device or system. Components associated with a data store (such as a database interface) may be considered "part" of the data store, part of some other system component, or a combination thereof, although the physical location and other characteristics of the components associated with the data store are not important to understanding the techniques described herein.

[0089] Data storage may include data structures. As used herein, a data structure is associated with a specific way of storing and organizing data in a computer so that the data structure can be used effectively in a given context. Data structures are generally based on the ability of a computer to obtain and store data at any location in its memory, which is specified by an address, a bit string that can be stored in the memory and manipulated by a program. Therefore, some data structures are based on using arithmetic operations to calculate the address of a data item; while other data structures are based on storing the address of a data item in its own structure. Many data structures use these two principles, and sometimes these two principles are combined in a non-trivial way. The implementation of a data structure generally requires writing a set of programs to generate and manipulate instances of the structure. The data storage described herein may be a cloud-based data storage. A cloud-based data storage is a data storage compatible with a cloud-based computing system and engine.

[0090] Figure 1B 1 is a diagram showing an illustrative example of one or more 2D image engines 160a. The one or more 2D image engines 160a may include a segmentation engine 168, a feature extraction engine 170, a virtual parameter engine 172, and a 2D image data storage 174. One or more modules of the one or more 2D image engines 162a may be coupled to each other or to modules not shown. The one or more 2D image engines 160a are configured to receive and process an input 2D image of a subject's teeth from a wide-angle camera assembly.

[0091] The one or more segmentation engines 168 may implement one or more automated agents configured to process the input 2D images from the wide angle camera assembly 155. The one or more segmentation engines 168 may include a graphics engine for processing the input 2D images. The one or more segmentation engines 168 may be configured to segment the input 2D images into individual dental components, including segmenting the input 2D images into individual teeth. Figure 2A is an example of an input 2D image from a wide-angle camera assembly. Figure 2A As shown, the input 2D image is a single image that includes an occlusal view of both dental arches of the subject's dentition. The wide angle nature of the camera assembly 155 enables such a view of both dental arches in a single 2D image. Figure 2B 1 is an example of an input 2D image that has been segmented (e.g., by a segmentation engine). As shown, the input 2D image has been segmented into individual dental components (e.g., teeth) with lower value image data (e.g., gums, lips, palate, etc.) removed from the image. The 2D image data store 174 can store and provide the input 2D image, the segmented 2D images, and / or other data to other modules of the treatment monitoring system 158.

[0092] The methods and devices described herein can benefit from the use of a wide-angle camera assembly and corresponding wide-angle input 2D images. These images can be processed as described herein and can be captured without the need to retract the cheeks. Thus, the subject can quickly and easily capture the image themselves (or can have a family member, friend, or caregiver capture it). Although the methods and devices can be used with a single wide-angle image, in some examples, more than one wide-angle image can be captured and used; in some examples, the images can be combined.

[0093] In some examples, the systems and methods described herein may include guidance for acquiring one or more input 2D images (e.g., a treatment monitoring system may include an image acquisition guidance engine). For example, the system may provide instructions for capturing images (on a subject's handheld device), including text, audio, images, video, etc., for positioning and capturing input 2D images. The system may check the quality of the captured input 2D image to ensure that it is focused, properly positioned (e.g., to provide an occlusal view), and / or includes sufficient features (e.g., includes all teeth).

[0094] The one or more feature extraction engines 170 may implement one or more automated agents configured to extract tooth features from an input 2D image or a segmented input 2D image. As used herein, a "tooth feature" may include data points from one or more input 2D images that are associated with a center, silhouette, edge, contour, vertex, vector, or surface of a subject's teeth. "Tooth features" that provide information or details about the orientation of a subject's teeth are particularly useful. For example, a tooth feature may include a tooth center that provides data about the position of a subject's teeth. Similarly, a tooth silhouette may provide data about both the position and orientation of a subject's teeth. Figure 2C An example of an input 2D image of a subject's teeth with tooth features including tooth silhouettes extracted and shown on the 2D image is shown. Figure 2D Only tooth features of the tooth centers extracted from the 2D image are shown. Although the tooth centers provide data about the location of individual teeth, the tooth centers may not easily provide sufficient data about the orientation of each tooth. The 2D image data store 174 can store the extracted features and / or other data and provide it to other modules of the treatment monitoring system 158.

[0095] One or more virtual camera parameter engines 172 can implement one or more automated agents that are configured to determine virtual camera parameters of a wide-angle camera assembly corresponding to an input 2D image. The virtual camera parameters of the wide-angle camera assembly can include non-intrinsic parameters and intrinsic parameters, wherein the non-intrinsic parameters define the position and orientation of the camera assembly relative to a world coordinate system, and the intrinsic parameters allow a mapping between camera coordinates and pixel coordinates in an image coordinate system. The virtual parameters of a wide-angle (e.g., fisheye) camera assembly can be iteratively calculated using an optimizer function. In some examples, the 2D image data store 174 can store virtual camera parameters and / or other data and provide it to other modules of the treatment monitoring system 158.

[0096] Figure 1C 1 is a diagram showing an example of one or more 3D model engines 162a. The one or more 3D model engines 162a may include a 2D rendering engine 176, a segmentation engine 178, a feature extraction engine 180, and a 3D model data storage 182. One or more modules of the one or more 3D model engines 162a may be coupled to each other or to modules not shown.

[0097] The 2D rendering engine 176 may implement one or more automated agents configured to render 2D images of the 3D tooth model received from the scanning system 154. The 2D image may be rendered from the 3D model using virtual camera parameters of the wide angle camera assembly from the virtual parameter engine 172. The rendered 2D image represents the expected or desired position of the subject's teeth at a particular time or stage of orthodontic treatment. The 3D model data store 182 may store and provide rendered 2D images and / or other data to other modules of the treatment monitoring system 158.

[0098] The one or more segmentation engines 178 may implement one or more automated agents that are configured to process the rendered 2D image from the 2D rendering engine 176. The one or more segmentation engines 178 may include a graphics engine for processing the rendered 2D image. The one or more segmentation engines 178 may be configured to segment the rendered 2D image into individual dental components, including segmenting the rendered 2D image into individual teeth. As described above, in some examples, the segmentation may be provided or assisted by a 3D model used to generate the rendered 2D image (e.g., the 3D model may be segmented or may include segmentation information). Figure 3A is an example of a rendered 2D image from a 2D rendering engine. Figure 3A As shown in , the rendered 2D image is a single image that includes an occlusal view of the subject's dental arch. Although only a single dental arch is shown for simplicity, it should be understood that the rendered 2D image can include an occlusal view of two dental arches, similar to the input 2D image. The virtual camera parameters can be used to generate the dental arch view from the 3D model as a 2D projection of the 3D model. The 3D model data store 182 can store the input 2D image, the segmented 2D image, and / or other data and provide it to other modules of the treatment monitoring system 158.

[0099] The one or more feature extraction engines 180 may implement one or more automated agents configured to extract tooth features from a rendered 2D image or a segmented rendered 2D image. The tooth features may include data points from one or more input 2D images that are associated with the center, silhouette, edge, contour, vertex, vector, or surface of the subject's teeth. Tooth features that provide information or details about the orientation of the subject's teeth are particularly useful. For example, the tooth center may provide data about the position of the subject's teeth (translation information). The tooth silhouette may provide data about the position and orientation (e.g., rotation) of the subject's teeth. In some examples, as described herein, it may be beneficial (and computationally simple) to use the tooth center to determine the translation difference between the input 2D image and the treatment plan from the rendered 2D image, while the rotation difference may be determined by comparing the tooth silhouettes from the input 2D image and the rendered 2D image.

[0100] Figure 3B An example of a rendered 2D image of a subject's teeth with tooth features including tooth silhouettes extracted and shown on the rendered 2D image is shown. The 3D model data store 182 may store and provide the extracted features and / or other data to other modules of the treatment monitoring system 158 .

[0101] Figure 1D is a diagram illustrating an example of one or more teeth comparison engines 164a. The one or more teeth comparison engines 164a may include a teeth movement engine 184 and a teeth movement data store 186. One or more modules of the one or more teeth comparison engines 164a may be coupled to each other or to modules not shown.

[0102] The tooth movement engine 184 can implement one or more automated agents to compare the rendered 2D image from the 3D model with the input 2D image from the wide-angle camera assembly. The comparison of the 2D images can provide an indication of whether the current position of the subject's teeth follows the expected position or desired position of the subject's teeth according to the orthodontic treatment plan. In some examples, the comparison provides a discrete value (e.g., between 0 and 1) that indicates how closely the subject's teeth follow the orthodontic treatment plan (e.g., a value of 1 would indicate that the subject's teeth follow the treatment plan perfectly, while a value of 0 indicates that the subject's teeth do not follow the treatment plan at all). The comparison between the input 2D image and the rendered 2D image can be used to modify the orthodontic treatment plan. For example, the comparison can be used to calculate new tooth movements required to move the teeth from the current position to the desired position. These new tooth movements can be used to update the orthodontic treatment plan.

[0103] For example, for each corresponding tooth in the input 2D image and the rendered 2D image (individual teeth may be identified by segmentation), the tooth movement engine may use the tooth center to estimate the position difference of the two types of teeth. For example, the relative translation of the tooth center may provide an indication of the translation difference of the tooth position between the actual planned position and the treatment planned position. Similarly, the tooth rotation between the actual (input 2D image) and the treatment plan (rendered 2D image) may be compared by comparing the tooth silhouettes. Segmentation provides the outline (silhouette) of the tooth as well as an indication of the identity of the tooth (e.g., tooth number). The tooth silhouette of the input 2D image may be compared by rotating the silhouette to substantially match the silhouette of the corresponding tooth from the rendered 2D image and / or the silhouette of the corresponding tooth in the 3D model.

[0104] In use, the systems and methods described herein can guide a subject to capture a current image (input 2D image) occlusal view of teeth, compare the current view with a treatment plan for a corresponding (or future) treatment planning stage, and provide a difference indication and / or provide guidance to a professional user (e.g., a dentist, orthodontist, or other dental professional) when adjusting the treatment plan.

[0105] For example, Figure 4A A method is schematically shown. Figure 4A In the embodiment, the subject (or the subject's agent, such as a family member, friend, caregiver, etc.) can optionally be guided through the steps of taking a wide angle 2D image of an occlusal view (or perspective occlusal view) of the teeth 452. In some examples, the initial steps can include attaching or connecting a wide angle lens or adapter to an existing camera, such as a smartphone camera. Fig. 6A An example of this is shown in . Figure 7 An example of a wide angle (e.g., fisheye) lens 703 coupled to a smartphone 701 is shown. For example, a subject may be guided by a system to capture an image, the system may provide visual guidance to capture a selfie while looking in a mirror, and / or may provide voice guidance to capture a selfie without a mirror. The system may indicate (e.g., via an alert) that the quality of the captured image is not suitable, or that the quality of the captured image may be improved. For example, the system may alert the user subject if lighting conditions are insufficient. As mentioned, a single image may be used with a wide angle camera assembly, or multiple images may be captured in succession with slight changes in angle to assist in achieving better accuracy.

[0106] Figure 6B An example of a wide angle image showing an occlusal (bite perspective) view of a subject's teeth taken as described herein is shown. These images can be taken without a retractor and can be initially analyzed by the device to determine if the image quality is adequate. In general, Figure 6B The images shown in the figure are occlusal perspective views of the upper and lower dental arches; the wide-angle camera assembly allows for simultaneous imaging of both the upper and lower jaws in the same view. In general, the wide-angle 2D images described herein may be referred to as occlusal views because they primarily (but not exclusively) show the occlusal surfaces of the subject's teeth. These views may also be referred to as occlusal perspective views because they show wide-angle perspective views of the maxillary occlusal surface and / or the mandibular occlusal surface.

[0107] Once sufficient input 2D images are received, the method or system for performing the method can determine virtual camera parameters 454 for the 2D images, as described above. For example, an optimizer can be used to find intrinsic and / or extrinsic parameters of the camera by iteratively calculating and recalculating the steps of tooth centers corresponding to adjusting virtual camera parameters. The optimization is such that the projected tooth centers can be projected to image pixels as close as possible to the tooth centers in one or more images. In some examples, the camera assembly can be pre-calibrated and some or all of the intrinsic parameters can be provided.

[0108] These virtual camera parameters can then be used to generate a rendered 2D image 456 from a 3D model of the patient's teeth (at a corresponding or future treatment stage). The rendered 2D image can then be compared to the input 2D image by comparing features between the two images (and / or 3D models). For example, a translation difference between the teeth of the input 2D image and the rendered 2D image can be determined 458 by identifying and comparing tooth centers of corresponding teeth between the input 2D image and the rendered 2D image. The rotation difference between the teeth of the input 2D image and the rendered 2D image can be determined 460 by using tooth silhouettes (e.g., tooth contours) from the segmented teeth in each image. Alternatively, in some examples, the translation difference can also or alternatively be determined from tooth silhouettes instead of or in addition to tooth centers.

[0109] Based on these comparisons, difference indicators (e.g., difference scores, difference maps, and / or difference values) between the input 2D image and the rendered 2D image may be determined, and the difference indicators may be sent, stored, and / or displayed 462. Optionally, if one or more differences exceed a threshold, one or more alerts may be sent to the subject and / or the subject's dental care provider (e.g., a professional user).

[0110] Figure 4B is another example of a flow chart describing a method for monitoring teeth of a subject during orthodontic treatment planning. Figure 4A As shown, the method can be performed by a system (e.g., Figure 1A One or more systems in the computing environment 100A shown in FIG. 1 are automatically implemented.

[0111] At operation 402, the method may include receiving an input 2D image from a wide-angle camera assembly, the input 2D image having an occlusal view of a first dental arch and a second dental arch of a subject's dentition. The input 2D image may be captured by the subject using a wide-angle camera or a wide-angle (e.g., fisheye) lens attachment of a smartphone, tablet, or PC, etc. In some implementations, the input 2D image includes an occlusal view of both dental arches of the subject's teeth.

[0112] At operation 404, the method may also include identifying dental features of teeth in the first dental arch and the second dental arch in the input 2D image. In some implementations, the dental features may be identified or extracted using a trained machine learning model. In other implementations, the dental features may be manually identified or extracted. The dental features may include tooth centers, tooth silhouettes, or other dental features that provide a location and / or orientation of the subject's teeth.

[0113] Next, at operation 406, the method may further include determining virtual parameters of a wide-angle camera corresponding to the input 2D image. The virtual parameters of the wide-angle camera may include non-intrinsic parameters and intrinsic parameters, wherein the non-intrinsic parameters define the position and orientation of the camera relative to the world coordinate system, and the intrinsic parameters allow mapping between camera coordinates and pixel coordinates in the image coordinate system. The virtual parameters of the fisheye lens may be iteratively calculated, for example, using an optimizer function.

[0114] Next, at operation 408, the method may include receiving a 3D model of the subject's dentition. The 3D model may be generated and received from a scanning system such as a 3D dental scanning system. Next, at operation 410, the method may also include generating a rendered 2D image from the 3D model using the virtual parameters of the wide-angle camera (the virtual parameters calculated at operation 406). The rendered 2D image represents the expected or desired positions of the subject's teeth.

[0115] At operation 411, the method may also include identifying dental features of teeth in the first dental arch and the second dental arch in the rendered 2D image. In some implementations, the dental features may be identified or extracted using a trained machine learning model. In other implementations, the dental features may be manually identified or extracted. The dental features may include tooth centers, tooth silhouettes, or other dental features that provide a location and / or orientation of the subject's teeth.

[0116] Next, at operation 412, the method may also include comparing the tooth features of the rendered 2D image with the tooth features of the input 2D image. As described above, the comparison of the 2D images may provide an indication of whether the current position of the subject's teeth follows the expected or desired position of the subject's teeth according to the orthodontic treatment plan. In some examples, the comparison provides a discrete value between 0 and 1 that indicates how closely the subject's teeth follow the orthodontic treatment plan (e.g., a value of 1 would indicate that the subject's teeth follow the treatment plan perfectly, while a value of 0 would indicate that the subject's teeth do not follow the treatment plan at all).

[0117] Optionally, the orthodontic treatment plan may be modified using a comparison between the input 2D image and the rendered 2D image at operation 414. For example, the comparison may be used to calculate new tooth movements required to move the teeth from current positions to desired positions. These new tooth movements may be utilized to update the orthodontic treatment plan.

[0118] In some examples of the methods described herein, the comparison between the input 2D image and the reconstructed 2D image can be limited to the tooth centers (eg, the centroid of each tooth). The translation difference can provide sufficient coarse tracking information for home monitoring.

[0119] In some examples, the methods and apparatus described herein may be performed without using a wide-angle lens. For example, the methods and apparatus may be performed using multiple cameras (e.g., phone cameras) to generate multiple images. These images may be combined into a single (pseudo-wide-angle) image and processed as described above, or processed using one or more simplified procedures. For example, Figure 4C One such method is schematically shown. Figure 4C In the embodiment of the present invention, the subject (or the subject's agent, such as a family member, friend, caregiver, etc.) can be guided through the steps of taking multiple images of one or two dental arches including teeth 482. The camera can be an existing camera of the subject or caregiver (user subject), such as a smartphone camera. For example, the user subject can be guided by the system to take images, and the system can provide visual guidance to take self-portraits while looking in a mirror, and / or can provide voice guidance to take self-portraits without a mirror. The system can indicate (e.g., through an alarm) that the quality of the image taken is not suitable, or that the quality of the image taken can be improved. For example, the system can issue an alarm to the user subject when the lighting conditions are insufficient. Multiple images can be taken continuously with slight changes in angle to assist in achieving better accuracy.

[0120] Before or after capturing an image, the user subject can calibrate the camera 484. In some examples, the user subject can be instructed to download and / or print an image of a calibration target (e.g., a grid, a chessboard, etc.). The calibration can be performed to determine virtual camera parameters. In some examples, the calibration can help determine the spacing between the camera and the image. Intrinsic parameters and / or non-intrinsic parameters of the camera can be identified. In some examples, the multiple images captured can be combined into a single (merged) image of each (or both) dental arches.

[0121] These virtual camera parameters can then be used to generate one or more rendered 2D images from a 3D model of the patient's teeth (at a corresponding or future treatment stage). The one or more rendered 2D images can then be compared to the input 2D image by comparing features between the two images (and / or 3D models). For example, translation differences and / or rotation differences between the teeth of the input 2D image and the rendered 2D image can be determined.

[0122] In some examples, for example, as part of calibration, the camera can be registered to the entire jaw. Each tooth can be individual and thus movement of a single tooth to the entire jaw detected. This registration can be performed by matching tooth silhouette projections to tooth silhouettes on the image; and / or matching tooth cusps and / or Fisher projections to image tooth cusps and / or Fisher projections; and / or using a trained network that takes as input a tooth image and a tooth depth map from the same camera and produces tooth movement 488. Alternatively or additionally, the rotational difference between the teeth of the input 2D image and the rendered 2D image can be determined by using tooth silhouettes (e.g., tooth contours) from the segmented teeth in each image. Alternatively, in some examples, the translational difference can also or alternatively be determined from the tooth silhouettes and / or by using tooth centers.

[0123] Based on these comparisons, difference indicators (e.g., difference scores, difference maps, and / or difference values) between the input 2D image and the rendered 2D image may be determined, and the difference indicators may be transmitted, stored, and / or displayed 490. Optionally, if one or more differences exceed a threshold, one or more alerts may be sent to the subject and / or the subject's dental care provider (e.g., a professional user).

[0124] The methods described herein may be performed by a device such as a data processing system, which may include hardware, software, and / or firmware for performing many of the steps described above. For example, Figure 5 is a simplified block diagram of a system 500 as described herein.

[0125] Any of these systems 500 may include or be configured to operate on a processor 554 and may include or be configured to operate with a camera assembly including a camera 556 and an integrated or separate wide angle (e.g., fisheye) lens 558. The system may generally include a controller 552 that includes any or all of the above modules / engines (e.g., Figures 1A to 1DModules / engines in ). For example, the controller may include an input 2D image generator 560 and an input 2D image analyzer 562 (which may collectively be a 2D input engine). In some examples, the controller may include or operate a virtual camera parameter identifier 564 (which may also be part of the 2D input engine), which may identify intrinsic parameters and / or non-intrinsic parameters of the camera component. The controller may also include a rendered 2D image generator 561 and a rendered 2D image analyzer 563 (which may be included as part of the 3D model engine). A difference determination engine 565 (e.g., a tooth comparison engine) may also be included. Finally, the controller may also include and / or control an output engine 566, which may trigger and present an alarm, store, send and / or display one or more comparison (e.g., difference) indicators.

[0126] Generally speaking, any of these processing systems can include at least one processor, which communicates with multiple peripheral devices (which can include all or some of the above components) through a bus subsystem. These peripheral devices can usually include a storage subsystem (e.g., a memory subsystem and a file storage subsystem), a grouped user interface input and output device, and an interface to an external network (including a public switched telephone network). The interface can be a modem and a network interface, and can be coupled to a corresponding interface device in other data processing systems through a communication network interface. These systems can include a terminal or a low-end personal computer or a high-end personal computer, a workstation or a mainframe.

[0127] These systems may include user interface input devices, which may include keyboards, and may also include pointing devices and scanners. The pointing device may be an indirect pointing device, such as a mouse, trackball, touch pad, or graphics tablet, or a direct pointing device, such as a touch screen incorporated into a display. Other types of user interface input devices may be used, such as voice identification systems.

[0128] The user interface output device may include a printer and a display subsystem including a display controller and a display device coupled to the controller. The display device may be a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), or a projection device. The display subsystem may also provide non-visual displays, such as audio output.

[0129] The storage subsystem may maintain the basic programming and data structures that provide the functionality of the present invention. The software modules discussed above may be stored in the storage subsystem. The storage subsystem may include a memory subsystem and a file storage subsystem. The memory subsystem may include a plurality of memories including a main random access memory (RAM) for storing instructions and data during program execution and a read-only memory (ROM) in which fixed instructions may be stored.

[0130] The file storage subsystem can provide persistent (non-volatile) storage for program and data files, and typically includes, for example, at least one hard disk drive and at least one floppy disk drive (with associated removable media). Other devices may also be present, such as a CD-ROM drive and an optical disk drive (both with their associated removable media). In addition, the system may include a drive of the type with a removable media cartridge. One or more drives may be located at a remote location, such as in a server on a local area network, or at a site on the World Wide Web of the Internet.

[0131] A "bus subsystem" may generally include any mechanism that enables various components and subsystems to communicate with each other as expected. Other components do not have to be located in the same physical location. Thus, for example, some parts of the file storage system may be connected via various local area network media or wide area network media (including telephone lines). Similarly, input devices and displays do not have to be located in the same location as the processor, although it is expected that the present invention will most often be implemented in the environment of PCS and workstations. The bus subsystem may be a single bus, or may include multiple buses, such as a local bus and one or more expansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus or PCI), as well as serial ports and parallel ports. Network connections are typically established by devices such as a network adapter located on one of these expansion buses or a modem located on a serial port. The client computer may be a desktop system or a portable system.

[0132] Various alternatives, modifications and equivalents may be used in place of the components described above.While computer-assisted techniques may be used to determine the final position of the teeth, an expert user may move the teeth to their final position by independently manipulating one or more teeth while satisfying prescription constraints.

[0133] Additionally, the techniques described herein may be implemented in hardware or software or a combination of the two. These techniques may be implemented in a computer program executed on a programmable computer, each programmable computer including a processor, a storage medium (including volatile and non-volatile memory and / or storage elements) that can be read by the processor, and suitable input and output devices. Program code is applied to data input by using an input device to perform the functions and generate output information. The output information is applied to one or more output devices.

[0134] Each program can be implemented in a high-level procedural or object-oriented programming language to operate in conjunction with a computer system. However, if desired, these programs can be implemented in assembly language or machine language. In any case, the language can be a compiled language or an interpreted language.

[0135] Each such computer program may be stored on a storage medium or device (e.g., a CD-ROM, hard disk, or magnetic disk) that can be read by a general or special programmable computer, for configuring and operating the computer when the storage medium or device is read by the computer to perform the process. The system may also be implemented as a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner.

[0136] Therefore, any method described herein (including user interface) can be implemented as software, hardware or firmware, and can be described as a non-transitory computer-readable storage medium that stores a set of instructions that can be executed by a processor (e.g., a computer, tablet, smartphone, etc.), which, when executed by the processor, causes the processor control to perform any steps including but not limited to the following: display, communication with the user, analysis, modification of parameters (including timing, frequency and intensity, etc.), determination, issuing an alarm, etc.

[0137] As mentioned above, any of the devices and methods may include calibration using a calibration standard. For example, a patient or caregiver (e.g., a user subject) can use a printed calibration standard (e.g., a calibration standard pattern or target, such as a checkerboard pattern) to calibrate the user subject's camera (e.g., a mobile phone camera, a stand-alone camera, a tablet camera, etc.). In some examples, a printed calibration standard can be provided to the user subject and / or the user subject can be instructed to print out the calibration standard (using a home printer). The calibration standard can be included in the packaging. In some examples, the packaging of a camera or a camera accessory (e.g., an adapter, etc.) for fixing all or part of an optical device (e.g., a wide-angle lens) can be configured as a calibration fixture that can position the camera in a predetermined position relative to the calibration standard (optionally, using an adapter).

[0138] For example, Fig. 8A An example of the use of a calibration standard with a user subject's telephone is schematically shown. Fig. 8A In , phone 801 includes an attachment 803 on the phone to interface with the internal phone camera. The attachment may include one or more lenses (e.g., a wide angle lens). The camera may be calibrated by taking a picture of a calibration target 805 (shown as a checkerboard pattern in this example). In some examples, the checkerboard pattern may be part of a box or packaging 810 that may be provided with the device (e.g., with an attachment that may be coupled to a camera of a user object). Fig. 8A and Figure 8B , the calibration pattern is shown as printed or attached to the package cover. The package may also include a code or link 807 (e.g., a quick response ("QR") code, a website address, etc.) that may provide instructions or a link to an application that may be downloaded to the user subject's camera device (e.g., a cell phone, tablet, etc.). The camera may be linked or registered to the user subject (e.g., a patient). The product (device) may be registered, and the application may assist in capturing and / or storing and / or processing images captured by the user subject's phone. The code or link (e.g., a QR code, a bar code, an alphanumeric code, etc.) may be a calibration target or may be part of a calibration target, such as Fig.8D shown.

[0139] In some examples, a calibration frame or fixture may be used to hold the user subject's phone at a fixed predetermined distance from the calibration target. For example, the device's packaging (e.g., box) 810 may include a calibration target and a location where the phone may be placed, held, or secured. Figure 8C An example of a box is shown that is configured to include a calibration target 805 (e.g., a calibration pattern) printed on the box and a support or mount 812 on the box. In some examples, the box includes a hole or opening on one side against which a camera can be supported to image the calibration pattern printed or attached within the box. Thus, the camera can image the calibration target at a predetermined (known) fixed distance 814, and the optical properties (focus, etc.) can therefore be calibrated based on the distance. In some examples, multiple calibration targets can be included (at the same or different distances). For example, a package can include multiple calibration targets. In some cases, multiple targets can be visible from the same location where the camera is placed (or held). The targets can be compact (e.g., Fig. 8A ), or can be larger (as shown in the example Figure 8C shown).

[0140] Although preferred embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided as examples only. Without departing from the present invention, those skilled in the art will now appreciate many examples, changes and alternatives. It should be understood that in practicing the present invention, various alternatives to the embodiments of the present invention described herein may be adopted. A variety of different combinations of the embodiments described herein are possible, and these combinations are considered to be a part of the present disclosure. In addition, all features discussed in conjunction with any one of the embodiments herein may be easily applied to other embodiments herein. The appended claims are intended to define the scope of the present invention, and therefore will cover methods and structures and their equivalents within the scope of these claims.

[0141] When a feature or element is referred to as "on" another feature or element in this article, it can be directly located on the other feature or element, or there can also be intermediate features and / or elements. On the contrary, when a feature or element is referred to as "directly on" another feature or element, there are no intermediate features or elements. It will also be understood that when a feature or element is referred to as "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element, or there can be intermediate features or elements. On the contrary, when a feature or element is referred to as "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or shown with respect to an example, the features and elements described or shown in this way can be applied to other examples. Those skilled in the art will also recognize that the reference to a structure or feature set to be "adjacent" to another feature can have a portion overlapping or located under the adjacent feature.

[0142] The terms used herein are used only to describe specific examples and are not intended to limit the present invention. For example, as used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any one and all combinations of one or more associated listed items, and may be abbreviated as " / ".

[0143] Spatially relative terms such as "under", "below", "lower", "over", "upper", etc. may be used herein to facilitate describing the relationship of one element or feature relative to another or multiple elements or features as shown in the accompanying drawings. It should be understood that, in addition to the orientations depicted in the accompanying drawings, spatially relative terms are intended to include different orientations of the device in use or operation. For example, if the device in the figure is inverted, the element described as "under" or "beneath" other elements or features will be oriented to be "over" the other elements or features. Therefore, the exemplary term "under" can include both above and below orientations. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein can be interpreted accordingly. Similarly, unless otherwise specifically indicated, the terms "upwardly", "downwardly", "vertical", "horizontal", etc. used herein are for explanation purposes only.

[0144] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Therefore, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0145] Unless the context requires otherwise, throughout the specification and the appended claims, the word "comprise" and examples such as "comprises" and "comprising" mean that various components can be used together in methods and articles (e.g., compositions and devices including apparatus and methods). For example, the term "comprising" will be understood to mean including any stated elements or steps without excluding any other elements or steps.

[0146] In general, any apparatus and method described herein should be understood to be inclusive, but all or a subset of components and / or steps may alternatively be exclusive and may be expressed as "consisting of" or alternatively "consisting essentially of" the various components, steps, subcomponents or sub-steps.

[0147] As used in this specification and claims, including as used in the examples, unless otherwise expressly specified, all numbers can be interpreted as beginning with the word "about" or "approximately", even if the word does not appear explicitly. When describing size and / or position, the phrase "about" or "approximately" can be used to indicate that the value and / or position described is within the reasonable expected range of the value and / or position. For example, the numerical value can be + / -0.1% of the specified value (or value range), + / -1% of the specified value (or value range), + / -2% of the specified value (or value range), + / -5% of the specified value (or value range), + / -10% of the specified value (or value range), etc. Unless the context indicates otherwise, any numerical value given herein should also be understood to include the approximate or approximate value of the value. For example, if the value "10" is disclosed, "approximately 10" is also disclosed. Any numerical range described herein is intended to include all subranges contained therein. It should also be understood that when a value is disclosed, values ​​that are "less than or equal to" that value, values ​​that are "greater than or equal to that value," and possible ranges between values ​​are also disclosed, as appropriately understood by those skilled in the art. For example, if a value "X" is disclosed, "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents endpoints and starting points, as well as ranges for any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it can be understood that greater than, greater than or equal to, less than, less than or equal to, equal to 10 and 15, and between 10 and 15 are all considered disclosed. It should also be understood that every single number between two specific single numbers is also disclosed. For example, if 10 and 15 are disclosed, 11, 12, 13, and 14 are also disclosed.

[0148] Although various illustrative examples are described above, any of a number of changes may be made to the various examples without departing from the scope of the invention as described in the claims. For example, in alternative examples, the order of execution of the various method steps described may be frequently changed, and in other alternative examples, one or more method steps may be skipped entirely. Optional features of the various device and system examples may be included in some examples and not included in other examples. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as described in the claims.

[0149] The examples and illustrations included herein illustrate specific examples by way of illustration and not limitation, in which separate themes can be practiced. As previously mentioned, other examples can be utilized and derived therefrom, so that structural and logical replacements and changes can be made without departing from the scope of the present disclosure. These examples of the subject matter of the present invention may be referred to herein individually or collectively by the term "invention", for the purpose of convenience only, and if more than one invention or inventive concept is in fact disclosed, it is not intended to voluntarily limit the scope of the present application to any single invention or inventive concept. Therefore, although specific examples have been illustrated and described herein, any arrangement calculated to achieve the same purpose may replace the specific examples shown. The present disclosure is intended to cover any and all modifications or variations of various examples. Combinations of the above examples and other examples not specifically described herein will be apparent to those skilled in the art after reading the above description.

Claims

1. A method, comprising: receiving an input 2D image of teeth of a subject, the input 2D image comprising a wide-angle occlusal view of an upper jaw and / or a lower jaw of the subject at a current treatment stage; Determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining a difference between teeth of a subject from the input 2D image and the rendered 2D image; as well as Based on a difference between teeth of the subject from the input 2D image and the rendered 2D image, a difference indicator is output, the difference indicator indicating a difference between current teeth of the subject and a planned treatment stage.

2. The method according to claim 1, wherein: Determining differences between teeth of the subject from the input 2D image and the rendered 2D image includes registering individual teeth of the subject's upper and / or lower jaw.

3. The method according to claim 2, wherein: Registering each tooth includes matching tooth silhouette projections between the rendered 2D image and the input 2D image.

4. The method according to claim 2, wherein: Registering each tooth includes matching tooth cusps and / or Fisher projections between the rendered 2D image and the input 2D image.

5. The method according to claim 2, wherein: Registering the individual teeth includes using a machine learning agent to determine differences between the subject's teeth from the input 2D image and the rendered 2D image.

6. The method according to claim 1, further comprising: The user subject is instructed to print a calibration pattern, and one or more images of the calibration pattern are captured using a camera of the user subject.

7. The method according to claim 1, wherein: Receiving the input 2D image includes: guiding the subject to capture the wide-angle occlusal view.

8. The method according to claim 1, wherein: The difference indicator is one or more of: a difference map, a difference score, and / or a set of difference values.

9. The method according to claim 1, wherein: Determining the virtual camera parameters includes: iteratively determining the virtual camera parameters from the input 2D image.

10. The method according to claim 1, wherein: Receiving an input 2D image of the subject's teeth includes receiving a fisheye view of both the subject's upper and lower jaws.

11. The method according to claim 1, wherein: Determining a difference between teeth of a subject from the input 2D image and the rendered 2D image includes determining a translation difference using tooth centers, including applying a trained machine learning model to the input 2D image and the rendered 2D image.

12. A method comprising: receiving an input 2D image of teeth of a subject, the input 2D image comprising a wide-angle occlusal view of an upper jaw and / or a lower jaw of the subject at a current treatment stage; Determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining a translation difference between teeth of the subject from the input 2D image and the rendered 2D image using tooth centers; determining a rotational difference between the subject's teeth from the input 2D image and the rendered 2D image using a silhouette of the subject's teeth; as well as A difference indicator is output, the difference indicator indicating a difference between the subject's current teeth and the planned treatment stage.

13. A system, comprising: one or more processors; Camera; Wide-angle lens; as well as a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: receiving an input 2D image of teeth of a subject, the input 2D image comprising a wide-angle occlusal view of an upper jaw and / or a lower jaw of the subject at a current treatment stage; Determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the subject's teeth from the 3D model using the virtual camera parameters; determining differences between teeth of the subject from the input 2D image and the rendered 2D image using tooth centers; and Based on a difference between teeth of the subject from the input 2D image and the rendered 2D image, a difference indicator is output, the difference indicator indicating a difference between current teeth of the subject and a planned treatment stage.

14. The system according to claim 13, wherein: The computer-implemented method also includes determining a rotational difference between the subject's teeth from the input 2D image and the rendered 2D image using a silhouette of the subject's teeth, wherein the difference indicator is further based on the rotational difference.

15. The system of claim 14, wherein: The computer-implemented method also includes applying a trained machine learning model to the input 2D image and the rendered 2D image to determine differences between teeth of the subject from the input 2D image and the rendered 2D image using tooth centers.

16. The system of claim 14, wherein: The difference indicator is one or more of: a difference map, a difference score, and / or a set of difference values.

17. The system of claim 14, wherein: Determining the virtual camera parameters includes: determining non-intrinsic camera parameters.

18. The system of claim 14, wherein: Determining the virtual camera parameters includes: iteratively determining the virtual camera parameters from the input 2D image.

19. The system of claim 14, wherein: Determining a difference between teeth of a subject from the input 2D image and the rendered 2D image includes determining a translation difference using tooth centers, including applying a trained machine learning model to the input 2D image and the rendered 2D image.

20. The system of claim 14, wherein: The computer-implemented method also includes determining a discrepancy score based on the discrepancy indicator and alerting a dental care provider of the subject if the discrepancy score exceeds a threshold.

21. The system of claim 14, wherein: The computer-implemented method also includes updating an orthodontic treatment plan based on the discrepancy indicator.

22. The system of claim 14, wherein: The computer-implemented method also includes using the difference indicator to calculate a new tooth movement required to move the subject's teeth from a current position to a desired position.

23. The system of claim 22, wherein: The computer-implemented method also includes updating an orthodontic treatment plan using the new tooth movement.

24. A method comprising: Calibrate the camera of the user subject's mobile phone; receiving a plurality of input 2D images of the upper jaw and / or lower jaw of a subject captured using a calibrated camera; Determining virtual camera parameters corresponding to the plurality of input 2D images; registering the camera with the upper jaw and / or lower jaw of the subject; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; determining translational differences and / or rotational differences between individual teeth from the plurality of 2D images and the 3D model by individually registering teeth of the upper jaw and / or lower jaw of the subject; as well as outputting a difference indicator indicating a difference between the subject's current teeth and the planned treatment stage, The method further comprises: generating a rendered 2D image of the subject's teeth from the 3D model using virtual camera parameters from the user subject's camera.

25. The method according to claim 24, wherein: Determining translational differences and / or rotational differences between teeth of the subject from the input 2D image and the 3D model includes registering individual teeth of the subject's upper and / or lower jaw.

26. The method according to claim 25, wherein: Registering each tooth includes matching tooth silhouette projections between a 2D image rendered from the 3D model and the input 2D image.

27. The method according to claim 25, wherein: Registering each tooth includes matching tooth cusps and / or Fisher projections between the rendered 2D image and the input 2D image.

28. The method according to claim 25, wherein: Registering the individual teeth includes using a machine learning agent to determine differences between the subject's teeth from the input 2D image and the 2D image rendered from the 3D model.

29. The method of claim 24, further comprising: The user subject is instructed to print a calibration pattern, and one or more images of the calibration pattern are captured using a camera of the user subject.

30. The method of claim 29, wherein: The calibration pattern includes a checkerboard pattern or a grid pattern.

31. The method of claim 29, wherein: The camera is calibrated, including generating an input 2D image of the subject's teeth.

32. A system, comprising: one or more processors; as well as a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: Calibrate the camera of the user subject's mobile phone; receiving a plurality of input 2D images of the upper jaw and / or lower jaw of a subject captured using a calibrated camera; Determining virtual camera parameters corresponding to the plurality of input 2D images; registering the camera with the upper jaw and / or lower jaw of the subject; receiving a 3D model of the subject's maxilla and / or mandible at a planned treatment stage; determining translational differences and / or rotational differences between individual teeth from the plurality of 2D images and the 3D model by individually registering teeth of the subject's upper and / or lower jaws; and outputting a difference indicator indicating a difference between the subject's current teeth and the planned treatment stage, The method further comprises: generating a rendered 2D image of the subject's teeth from the 3D model using virtual camera parameters from the user subject's camera.

Images