Tracking progress at home using mobile phone camera

Abstract

Systems and methods for monitoring teeth of a subject during orthodontic treatment are provided herein. Wide-angle (e.g., fisheye) images show bite views of two arches of an ivory column and may be input into a monitoring system configured to compare the images to a treatment plan to determine how closely a current position of a subject's tooth follows an expected position or desired position according to an orthodontic treatment plan.

Description

[0001] This application is a divisional application of the application with the filing date of February 11, 2021, application number 2021800142034, and invention title "Tracking Progress at Home Using a Mobile Phone Camera".

[0002] Priority Claim

[0003] This patent application claims the priority of U.S. Provisional Patent Application No. 62 / 975,148, titled "AT HOME PROGRESS TRACKING USING WIDE ANGLE CAMERA", filed on February 11, 2020, the entire content of which is incorporated herein by reference.

[0004] Incorporation by Reference

[0005] All publications and patent applications mentioned in this specification are incorporated herein by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0006] Orthodontic procedures generally involve repositioning an individual's teeth into a desired alignment in order to correct malocclusions and / or improve aesthetics. To achieve these goals, orthodontic appliances such as braces, aligner trays, etc. can be applied to the individual's teeth by an orthodontist and / or the individual themselves. The appliance can be configured to apply a force on one or more teeth in order to achieve the desired tooth movement according to a treatment plan.

[0007] Orthodontic appliances can include devices that are removable and / or replaceable on the teeth. Orthodontic appliances can be provided as part of an orthodontic treatment plan. In some orthodontic treatment plans involving removable and / or replaceable appliances, multiple orthodontic appliances can be provided to an individual during the course of treatment to effect incremental positional adjustments to the individual's teeth. Orthodontic appliances can have a polymeric tray with a lumen that is shaped to receive the teeth and elastically reposition the teeth from one tooth alignment to a successive tooth alignment. Orthodontic appliances can include an "active" region that applies a repositioning force on the teeth and a "passive" region that holds the teeth in their current state.

[0008] Treatment plans typically use 3D tooth models created from scans of an individual's teeth or dental casts. The 3D tooth model can include, for example, the original tooth point cloud of the 3D teeth, a tooth mesh, or a simplified parametric representation.

[0009] During the course of a treatment plan involving a series of dental appliances, it is beneficial to periodically examine the teeth of the subject to confirm that they are moving in the expected and / or desired manner according to the treatment plan, where the dental appliances are configured to move the teeth of the subject into a desired configuration. SUMMARY OF THE INVENTION

[0010] Described herein are methods and devices that can provide a low-cost and simplified way to confirm a treatment plan during one or more stages of the treatment plan. In particular, these methods and devices can provide a patient or caregiver with techniques (e.g., methods) and devices to assist in tracking or monitoring the treatment plan without the use of expensive or complex devices. For example, described herein are methods and devices that can be used with a personal telephone of a patient or caregiver. In some examples, an attachment or other device can be coupled to the telephone of the patient or caregiver; these attachment devices can be adapted to be used with software, hardware, or firmware to assist in taking an image of high enough quality (or guiding a user subject in taking an image) such that the image can accurately track the teeth of the patient in the treatment plan.

[0011] Particular attention is paid herein to methods and devices for assisting an object (e.g., a patient or caregiver, e.g., a parent, guardian, etc.), collectively referred to herein as a "user subject", in collecting one or more images having a sufficient size and having sufficient information regarding distance from the teeth, etc., to capture a sufficient dentition such that the dentition can be readily analyzed by one or more automated agents (including machine learning agents, other software, etc.) and / or human agents (e.g., technicians, dental professionals, etc.). Although the user subject may be different from a user expert, the user subject can be a professional user. For example, a professional user can use the methods and / or devices described herein on a patient and thus act as both the user subject and the professional user.

[0012] In some examples, a user subject can be directed to take multiple images of a patient's teeth and / or oral cavity using a telephone camera. In some examples, the telephone camera can be calibrated or set such that the resulting images can be combined to form one or more images of the entire maxilla, mandible, or both the maxilla and mandible, and the telephone camera can process the resulting one or more images to identify, record, and / or track the movement of one or more teeth within the maxilla and / or mandible of the subject. In some examples, one or more adapters (e.g., attachments) can be included, the one or more adapters including optical components, such as one or more lenses, etc. In some examples, one or more of the lenses can be a wide-angle (e.g., "fish-eye") lens.

[0013] For example, any one of these methods and devices (e.g., systems, apparatuses, etc.) can use a wide-angle (e.g., fish-eye) imaging system for object manipulation to capture one or more images (or in some cases, only a single image) of an object's dentition and determine (and / or indicate to the object) whether the object needs to see a dental plan provider to modify or adjust the treatment plan.

[0014] Described herein are methods and systems for monitoring the progress of a dental object during a treatment process. At any particular point in time during the treatment process, a three-dimensional (3D) model of the expected position of the object's teeth at that point in time can be projected in a timely manner from a 3D model of the object's teeth prepared prior to the start of the treatment. During the treatment process, a camera with a wide-angle (e.g., fish-eye) lens can be used to take two-dimensional (2D) images (including as a single image) of the object's teeth (typically the occlusal surfaces of one or both jaws). The 2D images can include occlusal views of both dental arches or a single (e.g., upper or lower) dental arch of the object's dentition. The 2D images represent the actual positions of the object's teeth at a specific point in the orthodontic treatment. In this document, the 2D images can be referred to as "original images" or input 2D images.

[0015] The input 2D images can be provided to a monitoring system that is configured to determine the camera parameters (also referred to herein as virtual camera parameters) of the camera with a wide-angle lens that was used to take the input 2D images. The virtual camera parameters include the intrinsic parameters of the camera (e.g., the optical parameters of the camera) and the non-intrinsic parameters of the camera (e.g., the position of the camera in space when the image was taken). The non-intrinsic parameters can be determined relative to the teeth. The monitoring system can use these camera parameters to generate a rendered 2D image from the 3D model of the object's teeth, where the 3D model may correspond to the object's teeth at a specific time or stage of the treatment plan. Typically, this specific time or stage of the treatment plan can correspond to the current actual time or stage that the object is undergoing when the input 2D image was taken, thereby allowing a comparison between the expected position of the teeth (from the 3D model) and the actual position of the teeth (from the input 2D image) during the treatment plan. However, in some examples, it may be desirable to compare the current position of the object's teeth (e.g., from the input 2D image) with one or more other stages of the treatment plan (including the initial position of the teeth); in such cases, the 3D model used to generate the rendered 2D image may alternatively correspond to another stage of the treatment plan or the initial position of the object's teeth prior to the start of the treatment plan.

[0016] The monitoring system can compare the input 2D image with the rendered 2D image to determine how closely the actual or current position of the object teeth follows the expected or desired position according to the orthodontic treatment plan. The methods and systems described herein can perform this comparison in a fast and efficient manner by determining the positions of the centers of the teeth in both the input 2D image and the rendered 2D image, and comparing the profiles of the corresponding teeth in the input 2D image with the profiles of the corresponding teeth in the rendered 2D image. These steps can be performed after segmenting the teeth in the input 2D image and the rendered 2D image. In some examples, the 3D tooth model can include segmentation information for each tooth, which is used to segment the rendered 2D image. The rendered 2D image can be segmented separately from the input 2D image. In some examples, the segmentation of the rendered 2D image (which can be expected to be substantially similar to the rendered 2D image) can be used to assist in segmenting the input 2D image.

[0017] Any suitable method can be used to determine the 2D tooth center from the input 2D image. For example, the 2D center of the tooth can be identified manually, automatically, or semi-automatically. In some examples, machine learning (e.g., forming a trained network) can be used to find the 2D center of the tooth from the occlusal view provided by a wide-angle (e.g., fish-eye) image. Similarly, any suitable technique can be used to determine the tooth center of the rendered 2D image. In some examples, the tooth center of the rendered 2D image can be identified from the 3D model of the tooth before generating the rendered 2D image using the virtual camera parameters. Alternatively, in some examples, the tooth center of the rendered 2D image can be determined directly from the rendered 2D image. For example, in some examples, the same technique used to determine the tooth center from the input 2D image (e.g., using a trained neural network) can be used to analyze the rendered 2D image.

[0018] In some examples, the tooth centers from the input 2D image can be compared to the tooth centers from the rendered 2D image to provide an estimate of the differences and / or consistencies between the tooth centers of the rendered 2D image and the input 2D image. The comparison of tooth centers can provide an estimate of the translational movement (e.g., x, y translation of the tooth) of the teeth between the treatment plan and the actual teeth of the subject. Alternatively or additionally, the comparison of the tooth profiles (e.g., outlines) in the rendered 2D image to the tooth profiles in the input 2D image can provide an estimate of the rotational movement of the teeth (e.g., as the rotation required to align the teeth of the input 2D image with the teeth of the rendered 2D image). In some examples, only the outlines of the segmented teeth can be compared to determine both translational and rotational movement. However, in some examples, it may be beneficial to use the tooth centers to quickly and accurately identify the movement to provide an estimate of tooth movement. As described herein, estimating tooth movement using tooth centers can be done quickly and may require less computation. In some examples, a more accurate estimate of tooth movement (including rotational estimates) may not be necessary.

[0019] Any method and system described herein can use the comparison between an input 2D image (e.g., a wide-angle image of the occlusal surface of the teeth taken of the subject) and a rendered 2D image to generate one or more scores (difference scores) that are related to the differences in the positions of one or more teeth between the input 2D image and the rendered 2D image. The one or more scores can indicate the severity of the differences between the input 2D image and the rendered 2D image. For example, the difference score can be a measure of the total deviation of all segmented teeth in translation and / or rotation. In some examples, separate scores can be used to reflect the differences of different teeth. In some examples, separate scores can be used to reflect the differences between translation and rotation. The one or more scores can be weighted and / or scaled. For example, the score may place more emphasis on translation than rotation; the score may place more emphasis on rotation than translation; the score may place more emphasis on certain teeth than other teeth (e.g., molars are more important than premolars, more important than incisors, etc.) and so on. The scores can be normalized or averaged.

[0020] In some examples, one or more difference scores can be provided to an object and / or a dental professional (“dental professional user” or simply “professional user”). For example, the method or device can include alerting a dental professional user (e.g., a doctor, dentist, orthodontist, or other dental professional) when one or more difference scores are higher than one or more thresholds. In some examples, one or more thresholds can be preset or can be set by the professional user. For example, in some examples, the object may be using an application software (e.g., “app”) of a handheld device (e.g., a smartphone) that includes a wide-angle (e.g., fish-eye) lens with a camera. The application software can guide the object to take a wide-angle image and can process the image (locally using one or more processors in the handheld device or remotely by sending the image to a remote server). The processed input 2D image can then be analyzed to determine differences as described herein, and if the difference score exceeds a notification threshold, the user object can be instructed to schedule an appointment with their dental care provider (e.g., the professional user) to adjust the treatment plan. In some examples, the difference score can trigger a notification to the object to wear the appliance more frequently or otherwise comply more diligently with the treatment plan.

[0021] A comparison between the dentition of the object (e.g., the position of the object's teeth at the current treatment stage) and the expected or predicted tooth positions of the object's teeth at a particular treatment stage from a 3D model of the object's teeth can be output as a difference indicator. The difference indicator can be a difference map, a difference score, and / or a set of difference values. The difference indicator can be output by the method or system.

[0022] For example, in any of the methods and systems described herein, one or more differences (e.g., difference indicators) between the input 2D image and the rendered 2D image can provide difference information to replace or supplement one or more difference scores. For example, the method or system can generate a difference map that visually shows which teeth deviate from the treatment plan and by how much. In some examples, the difference map can be formed as a composite of the input 2D image and the rendered 2D image to highlight the differences (by one or more of symbols, text, colors, etc.).

[0023] In any of these methods and systems, one or more differences (e.g., difference indicators) between the input 2D image and the rendered 2D image can be provided as a set of difference values, e.g., a list, spreadsheet, data set, etc., providing alphanumeric data that summarizes or lists the differences between the input 2D image and the rendered 2D image.

[0024] Difference indicators (e.g., difference maps, difference scores, and / or difference values) can be stored, displayed, and / or sent (for storage and / or display), including being transmitted to a professional user (e.g., the dental provider of the subject) or a third party. In some examples, the displayed images (including but not limited to input 2D images and rendered 2D images) can be adjusted to reduce or remove image distortion caused by the wide-angle camera component.

[0025] For example, described herein are methods, including methods for determining the progress of a treatment plan for a subject (e.g., determining a deviation from a treatment plan). These methods can also be methods for alerting a dental practitioner of the subject (a professional user, e.g., a dentist, an orthodontist, etc.) of a deviation from a dental treatment plan, and / or methods for correcting or updating a dental treatment plan. For example, a method can include: receiving an input 2D image from a wide-angle (e.g., fisheye) camera, the input 2D image including an occlusal view of a first dental arch and / or a second dental arch of a patient's dentition; identifying dental features of teeth in the first dental arch and / or the second dental arch in the input 2D image; determining virtual camera parameters from the input 2D image; receiving a 3D model of the patient's dentition (e.g., at a target treatment plan stage or time); generating a rendered 2D image from the 3D model using the virtual camera parameters; identifying dental features of teeth in the first dental arch and / or the second dental arch in the rendered 2D image; and comparing the dental features of the rendered 2D image with the dental features of the input 2D image. Any of these methods can also include outputting a difference indicator based on the difference between the teeth from the input 2D image and the rendered 2D image, the difference indicator indicating the difference between the subject's current teeth and the planned treatment stage.

[0026] For example, identifying dental features can include applying a trained machine learning model to a 2D fisheye photo. In some examples, identifying dental features includes manually identifying dental features. Dental features can include the centers of teeth. Dental features can include tooth profiles. In some examples, the dental features provide the position and orientation of the patient's teeth.

[0027] The input 2D image can represent the actual position of the patient's teeth at a particular point during orthodontic treatment (e.g., at a particular stage of a treatment plan), while the rendered 2D image can represent the expected or desired position of the patient's teeth (at the corresponding stage of a treatment plan) of a synthetic 3D model of the patient's teeth based on predicting the tooth positions at the corresponding stage.

[0028] In some examples, the comparison indicates how closely the patient's teeth are following an orthodontic treatment plan.

[0029] As mentioned, any of these methods may include updating an orthodontic treatment plan based on a comparison. For example, the update may include calculating new tooth movements required to move the patient's teeth from their current position to a desired position. In some examples, the update may include using these new tooth movements to update the orthodontic treatment plan.

[0030] As described herein, a method (e.g., a method of determining treatment plan progress and / or deviation from a treatment plan for an object, a method of alerting a dentist of the object to a deviation from a dental treatment plan, and / or a method of correcting or updating a dental treatment plan) may include: receiving an input 2D image of the object's teeth, the input 2D image including a wide-angle occlusal view of the object's maxilla and / or mandible at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the object's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the object's teeth from the 3D model using the virtual camera parameters; determining a difference between the teeth from the input 2D image and the rendered 2D image using tooth centers; and outputting a difference indicator that indicates a difference between the object's current teeth and the planned treatment stage based on the difference between the teeth from the input 2D image and the rendered 2D image.

[0031] In some examples, a method as described herein may include: receiving an input 2D image of the object's teeth, the input 2D image including a wide-angle occlusal view of the object's maxilla and / or mandible at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the object's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the object's teeth from the 3D model using the virtual camera parameters; determining a translational difference between the teeth from the input 2D image and the rendered 2D image using tooth centers; determining a rotational difference between the teeth from the input 2D image and the rendered 2D image using tooth profiles; and outputting a difference indicator that indicates a difference between the object's current teeth and the planned treatment stage.

[0032] Thus, any of these methods may include determining one or both of a translational difference and / or a rotational difference of the teeth. For example, any of these methods may include determining a rotational difference between the teeth from the input 2D image and the rendered 2D image using tooth profiles, wherein the difference indicator is also based on the rotational difference. The rotational difference and / or translational difference between the teeth from the 2D image and the rendered 2D image may be determined in whole or in part by applying a trained machine learning model to the input 2D image and the rendered 2D image. Any of these methods may include receiving the input 2D image, including guiding the object to take a wide-angle occlusal image. The difference indicator may be one or more of the following: a difference map, a difference score, and / or a set of difference values.

[0033] Generally, the input 2D image can be used to determine virtual camera parameters, particularly non-intrinsic camera parameters. In some examples, the input 2D image can be used to determine intrinsic camera parameters. Alternatively, in some examples, some or all of the intrinsic camera parameters can be preset or predetermined (e.g., factory settings, input by the camera and / or lens vendor, etc.). In some examples, the virtual camera parameters include parameters that can be determined iteratively from the input 2D image.

[0034] The methods and devices described herein can operate based on a single input 2D image (e.g., a single image captured by a wide-angle (e.g., fish-eye) lens). In some examples, receiving an input 2D image of an object's teeth can include receiving a fish-eye view of both the object's maxilla and mandible. The wide-angle image can be a fish-eye image. For example, the wide-angle image can have a field of view covering up to 160 degrees or more (e.g., up to 170 degrees or more, up to 180 degrees or more, up to 190 degrees or more, up to 200 degrees or more, up to 210 degrees or more, up to 220 degrees or more, up to 230 degrees or more, up to 240 degrees or more, up to 250 degrees or more, up to 260 degrees or more, up to 270 degrees or more, etc.). The wide-angle image can include an occlusal (or bite perspective) view of all of the object's teeth in both the maxilla and mandible. This can advantageously allow a single image to be processed as described herein. In some examples, the combination of the wide-angle image and using the tooth centers to determine translation differences and using the tooth profiles to determine rotation differences allows for an extremely fast and accurate determination of the dental configuration differences between the actual tooth positions and the predicted (e.g., planned or modeled) tooth positions, thus requiring less processing power and operating more robustly on the input 2D images generated by the object.

[0035] Any method described herein can include determining a difference score. The difference score can be based on a difference indicator. Any of these methods can also include, if the difference score exceeds a threshold, issuing an alert to the object and / or the object's dental care provider. The object's dental care provider can set or approve the threshold.

[0036] In some examples, the methods described herein can also include updating an orthodontic treatment plan based on the difference indicator. Alternatively or additionally, the methods described herein can include using the difference indicator to calculate new tooth movements required to move the object's teeth from the current position to the desired position, and / or using these new tooth movements to update the orthodontic treatment plan.

[0037] Also described herein is a system configured to perform any of the methods described herein. For example, a system can include a non-transitory computer-readable medium having instructions stored thereon for tracking a patient's teeth during an orthodontic treatment plan, wherein the instructions are executable by a processor to cause the computer device to: receive an input 2D image from a wide-angle (e.g., fisheye) camera, the input 2D image having an occlusal view of a first dental arch and a second dental arch of a patient's dentition; identify tooth features of teeth in the first dental arch and the second dental arch in the input 2D image; determine virtual parameters of the wide-angle camera corresponding to the input 2D image; receive a 3D model of the patient's dentition; generate a rendered 2D image from the 3D model using the virtual parameters of the wide-angle camera; identify tooth features of teeth in the first dental arch and the second dental arch in the rendered 2D image; and compare the tooth features of the rendered 2D image with the tooth features of the input 2D image.

[0038] Any of the methods and devices described herein can include one or more calibration steps. For example, a printed calibration standard (e.g., a calibration standard pattern or target such as a checkerboard pattern) can be provided to a user object (patient, caregiver, etc.) and / or the user object (patient, caregiver, etc.) can be instructed to print out the calibration standard (using a home printer). The calibration standard can be included in the packaging. In some examples, the packaging of the camera or a camera accessory (e.g., an adapter, etc.) for securing all or part of the optics (e.g., a wide-angle lens) can be configured as a calibration jig that can position the camera in a predetermined position relative to the calibration standard (optionally, using an adapter). The user object can then take a number of images (and possibly be instructed by control software on, e.g., the user object's phone) of the calibration standard, and then the method or device can use these images to calibrate the user object's camera (including a camera phone). In some examples, the device or method can apply calibration logic to analyze one or more images of the calibration standard and determine distances to the camera, etc.

[0039] For example, any system described herein may include: one or more processors; a camera; a wide-angle lens; and a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method that includes: receiving an input 2D image of an object's teeth, the input 2D image including a wide-angle occlusal view of the object's maxilla and / or mandible at a current treatment stage; determining virtual camera parameters corresponding to the input 2D image; receiving a 3D model of the object's maxilla and / or mandible at a planned treatment stage; generating a rendered 2D image of the object's teeth from the 3D model using the virtual camera parameters; determining a difference between the teeth from the input 2D image and the rendered 2D image using a tooth center; and outputting a difference indicator based on the difference between the teeth from the input 2D image and the rendered 2D image, the difference indicator indicating a difference between the object's current teeth and the planned treatment stage. These systems may also be configured such that the computer-implemented method performs any of the above steps.

[0040] In some examples, the system is configured to include a handheld computing and / or communication device (e.g., a smartphone, a tablet, etc.) or operate on / with a handheld computing and / or communication device. Any of the above steps may be performed locally (e.g., in a handheld computing device), or they may be divided between a local processor and a remote processor. Remote processing may be performed on a remote server (e.g., a cloud-based server), etc.

[0041] In any method and device described herein, a camera may be registered to a patient's jaw and / or teeth as described herein. After registering the camera to the entire jaw, the method or device may register each tooth individually, as described above, and by doing so, this information may be used to detect movement of an individual tooth relative to the entire jaw. For example, registration may be detected by matching a tooth profile projection to the profile of the tooth on the image. In some examples, registration may be detected by, for example, matching a cusp and / or one or more Fisher projections to the image cusp and / or Fisher projections. In some examples, registration may be detected by training a network that takes as input a tooth image and a tooth depth map from the same camera and produces tooth movement. Any combination of these may be used. Generally, registration from one or more images may be done from multiple images and / or form a short video taken.

[0042] For example, the methods described herein include: calibrating a camera of a user object for a telephone of the user object; receiving a plurality of input 2D images of an object's maxilla and / or mandible captured using the calibrated camera of the user object; determining virtual camera parameters corresponding to the plurality of input 2D images; registering the camera to the object's maxilla and / or mandible; receiving a 3D model of the object's maxilla and / or mandible at a planned treatment stage; determining translational differences and / or rotational differences between individual teeth from the plurality of 2D images and the 3D model by separately registering the teeth of the object's maxilla and / or mandible; and outputting a difference indicator that indicates a difference between the object's current teeth and the planned treatment stage.

[0043] The plurality of input 2D images may be combined into a single image (e.g., a composite image). Calibrating the camera of the user object may include determining virtual camera parameters.

[0044] Any of these methods may include generating a rendered 2D image of the object's teeth from the 3D model using the virtual camera parameters from the user object's camera. The rendered 2D image may be used for comparison with the plurality of 2D images (or composite 2D image).

[0045] The methods described herein may determine translational differences and / or rotational differences between teeth from the input 2D images and the 3D model by registering the individual teeth of the object's maxilla and / or mandible. For example, registering the individual teeth may include matching tooth profile projections between the rendered 2D image and the input 2D image. In some examples, registering the individual teeth includes matching cusp and / or Fisher projections between the rendered 2D image and the input 2D image. In some examples, registering the individual teeth includes using a machine learning agent to determine differences between teeth from the 2D image and the rendered 2D image.

[0046] Any of these methods may include instructing the user object to print a calibration pattern and capturing one or more images of the calibration pattern using the user object's camera. These images may be used to calibrate the camera and / or the image, including determining virtual camera parameters and / or determining the distance to the camera. The calibration pattern may include a checkerboard pattern or a grid pattern.

[0047] These methods may include calibrating the camera by generating input 2D images of the object's teeth.

[0048] Any method described herein may be performed by a system configured to perform these methods. For example, a system may include: one or more processors; and a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method that includes: calibrating a camera of a user object's phone for the user object; receiving a plurality of input 2D images of an object's upper jaw and / or lower jaw captured using the calibrated camera of the user object; determining virtual camera parameters corresponding to the plurality of input 2D images; registering the camera to the object's upper jaw and / or lower jaw; receiving a 3D model of the object's upper jaw and / or lower jaw at a planned treatment stage; determining translational differences and / or rotational differences between respective teeth from the plurality of 2D images and the 3D model by separately registering the teeth of the object's upper jaw and / or lower jaw; and outputting a difference indicator that indicates a difference between the object's current teeth and the planned treatment stage.

[0049] Other examples of technologies and systems that may benefit from the methods and devices described herein may be found, for example, in U.S. Patent Application No. 16 / 370,788, filed on March 29, 2019 (entitled "PHOTOGRAPH-BASED ASSESSMENT OF DENTAL TREATMENTS AND PROCEDURES"), which is a continuation of U.S. Patent Application No. 14 / 831,548, filed on August 20, 2015 (entitled "PHOTOGRAPH-BASED ASSESSMENT OF DENTAL TREATMENTS AND PROCEDURES"), which was published as U.S. 10,248,883 on April 2, 2019. Each of these applications is incorporated herein by reference in its entirety.

[0050] Generally, the methods and devices described herein may be performed at very low cost and complexity to enable an object to be monitored at home without the need for a dentist or expensive scanning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the invention will be obtained by reference to the following detailed description that sets forth exemplary embodiments, in which the principles of the invention are utilized, and the accompanying drawings, of which:

[0052] Figure 1AFIG. is an example of a computing environment configured to monitor an object's teeth during orthodontic treatment planning.

[0053] Figure 1B FIG. is an example of one or more 2D image engines.

[0054] Figure 1C FIG. is an example of one or more 3D model engines.

[0055] Figure 1D FIG. is an example of one or more tooth comparison engines.

[0056] Figures 2A to 2D Shows an input 2D image ( Figure 2B ) that has been segmented ( Figures 2C to 2D ) and from which tooth features have been extracted ( Figure 2A ) as an example.

[0057] Figure 3A FIG. is an example of a rendered 2D image from a 3D model.

[0058] Figure 3B FIG. is an example of a rendered 2D image from which tooth features have been extracted.

[0059] Figure 4A FIG. is an example of a method for monitoring an object's teeth during orthodontic treatment planning.

[0060] Figure 4B FIG. is another example of a method for monitoring an object's teeth during orthodontic treatment planning.

[0061] Figure 4C FIG. is another example of a method for monitoring an object's teeth during orthodontic treatment planning.

[0062] Figure 5 FIG. is an example of a schematic diagram of a system for determining the difference between the actual tooth position of an object and the expected or predicted position of the object's teeth during treatment planning.

[0063] Figure 6A FIG. is an example in which an object is guided to take a wide-angle image (input 2D image) of its upper and lower dental arches, thereby showing an occlusal view of the teeth.

[0064] Figure 6B FIG. shows an example of a wide-angle occlusal perspective view of teeth that can be used as a 2D input image as described herein.

[0065] Figure 7 FIG. shows an example of a wide-angle (e.g., fish-eye) lens of a smartphone as described herein.

[0066] Figure 8AShows an example of calibration as described herein.

[0067] Figure 8B Shows an example of calibration using a calibration target on packaging that can be included with a product.

[0068] Figure 8C Shows an example of a package that is also configured as a calibration jig.

[0069] Figure 8D Shows an example of a calibration target that includes a code or link (e.g., QR code). Detailed Description

[0070] Described herein are devices (e.g., systems, computer-readable media, apparatuses, etc.) and methods for monitoring, analyzing, correcting, and / or tracking the orthodontic treatment progress of an object. The devices and methods described herein can capture or receive as input a 2D image of an object's teeth taken at a wide angle (e.g., fisheye) (input 2D image), and process the input 2D image to compare it with a model of the object's dentition (e.g., 3D digital model) to determine the difference between the actual tooth configuration (from the input 2D image) and the predicted or desired tooth configuration (from the 3D model) at one or more stages of a treatment plan. The 3D model can represent the configuration of the object's teeth at a particular stage of a treatment plan; in some examples, the stage can correspond to the stage to which the object's teeth are expected to be currently positioned. The 3D digital model can be generated at the start of a treatment plan and can be used to design and manufacture a series of orthodontic appliances (e.g., aligners).

[0071] The methods described herein can use the input 2D image and / or predetermined values to determine virtual camera parameters (e.g., intrinsic virtual camera parameters and non-intrinsic virtual camera parameters) of a camera that captured the wide-angle input 2D image. The virtual parameters can then be used to generate a rendered 2D image corresponding to the input 2D image from a 3D tooth model of the object's teeth for comparison. The rendered 2D image can be compared with the input 2D image to generate difference indicators such as a difference map, difference score, and / or set of difference values. In some examples, the comparison can determine how closely the object's teeth are following the desired orthodontic treatment plan.

[0072] The devices and / or methods described herein can be used for the planning and fabrication of dental appliances, which include the resilient polymeric positioning appliances described in detail in U.S. Patent No. 5,975,893 and published PCT application WO98 / 58596, which are incorporated herein by reference for all purposes. A system for dental appliances employing the technology described in U.S. Patent No. 5,975,893 can be purchased from Align Technology, Inc., San Jose, California, under the trade name "Invisalign System".

[0073] Throughout the description of the embodiments, the use of the terms "orthodontic appliance", "appliance", or "tooth appliance" is synonymous with the use of the terms "apparatus" and "dental appliance" in the context of dental applications. For clarity, the examples are described hereinafter in the context of the use and application of an apparatus, and more specifically, a "dental appliance".

[0074] As used herein, an "object" (or alternatively and equivalently, an "individual") can be any object (e.g., human, non-human, adult, child, etc.) and can alternatively be a patient, a subject being treated, etc. The object can be a medical patient. An individual or object can include a person undergoing orthodontic treatment, including orthodontic treatment utilizing a series of orthodontic appliances.

[0075] The devices and / or methods described below (e.g., systems, devices, etc.) can be used in conjunction with and / or integrated into an orthodontic treatment plan. The devices and / or methods described herein can include segmenting an individual's teeth from a three-dimensional model (e.g., a 3D mesh model or a 3D point cloud). The methods and devices described herein can segment 2D images. The segmentation information can be used to measure, compare, and process 2D images. The teeth of an individual can be segmented automatically (e.g., using a computing device), manually, or semi-automatically. For example, a computing system can perform the segmentation automatically by evaluating data of the individual teeth or dental arches (e.g., a three-dimensional scan or a dental impression). The segmentation can be accomplished using a trained network (e.g., as part of machine learning techniques). One or more data structures (e.g., databases) can be used to assist in segmenting 2D images or 3D models. In some examples, the methods and devices described herein can use segmentation information from a 3D model of the patient's teeth that is input or received. Thus, the segmentation information of the teeth from the 3D model can be provided as an input and can be used to analyze the input 2D images, as well as any 2D images generated from the 3D model of the object's teeth.

[0076] For example, a 3D model of an object's teeth can be initially generated from an intraoral scanner that scans an individual dental arch to generate a virtual 3D model of the dental arch. During an intraoral scanning procedure (also referred to as a scanning session), a professional user of the intraoral scanner (e.g., a dentist) can generate multiple different images (also referred to as scans or medical images) of tooth regions, models of tooth regions, or other objects. These images can be discrete images (e.g., point-and-shoot images) or frames from a video (e.g., continuous scans). Intraoral scan images can be used to generate a 3D model that can be modified to form all or part of an orthodontic treatment plan ("treatment plan") having multiple stages for moving the teeth to a desired configuration (including a final (e.g., aligned) configuration). Although the methods and devices described herein can incorporate (and can receive as input) intraoral scan information, these devices and methods described herein can be performed without the use of an intraoral scanner. In particular, these methods and devices can be operated by an object in a home environment, thereby allowing the object to assist in monitoring, tracking, and updating progress in a robust and inexpensive manner.

[0077] Figure 1A FIG. is an example of a computing environment 100A configured to facilitate the collection and processing of a digital scan of a dental arch having teeth therein. Environment 100A includes a computer-readable medium 152 configured to receive a 3D model of an object's teeth at one or more treatment stages 154, and to receive or include a wide-angle camera assembly 155 and a treatment monitoring system 158, the wide-angle camera assembly 155 including a camera 188 having one or more wide-angle (e.g., fish-eye) lenses 157 or coupled to one or more wide-angle (e.g., fish-eye) lenses 157. The system can be configured to output 156 images, text, data, and / or alerts. The output 156 can be part of the treatment monitoring system, or it can be separate from the treatment monitoring system. The treatment monitoring system can include one or more modules (e.g., engines, data structures, etc.) for processing and comparing 2D and 3D representations of an object's teeth. One or more modules in computing environment 100A can be coupled to each other, or to modules not explicitly shown.

[0078] The computer-readable medium 152 and other computer-readable media discussed herein are intended to represent various potentially applicable technologies. For example, the computer-readable medium 152 can be used to form a network or part of a network. When two components are co-located on a device, the computer-readable medium 152 can include a bus or other data conduit or plane. In the case where a first component is co-located on one device and a second component is located on a different device, the computer-readable medium 152 can include a wireless or wired backend network or LAN. If applicable, the computer-readable medium 152 can also include the relevant portion of a WAN or other network.

[0079] 3D models of one or more treatment plan phases 154 can be received from a scanning system, a database, a computer system, or any other structure. The 3D models of one or more treatment plan phases can be 3D models of an entire treatment plan or a subset of a treatment plan. The 3D model can include treatment plans for the upper dental arch and the lower dental arch. As used herein, a "dental arch" can include all or at least a portion of an individual's dentition formed by the individual's maxillary teeth and / or mandibular teeth. A dental arch can include one or more maxillary teeth or mandibular teeth of an individual, e.g., all of the teeth on an individual's maxilla or mandible. The 3D model 154 can be stored in a memory, a database, and / or one or more processors, and can be used to generate a rendered 2D image of the object teeth and to provide additional information about the object teeth (including tooth segmentation, etc.). The 3D model can be associated with a particular object. In some examples, the system can include a verification engine (not shown) that can confirm or verify that the 3D model of one or more treatment plan phases corresponds to the particular object for which the input 2D image was taken. One or more 3D tooth models can include, for example, one or more 3D point clouds or 3D tooth meshes. The treatment monitoring system 158 can be configured to receive 3D model data that was previously acquired or acquired by another system.

[0080] The wide-angle camera assembly 155 can include a computer system or a camera 188 that is configured to obtain one or more wide-angle 2D images of an object dentition, for example, operating with a wide-angle lens 157. The wide-angle lens can be part of the camera or can be added to the camera. The wide-angle camera assembly can be part of a treatment monitoring system 158 or it can be used in conjunction with the treatment monitoring system 158. As described herein, the wide-angle 2D images can include ultra-wide-angle 2D images produced with a wide-angle (e.g., fish-eye) camera or a wide-angle (e.g., fish-eye) lens, where the object may have a bulging non-linear appearance. The wide-angle 2D image of the object dentition can include, for example, an occlusal view (or perspective occlusal view) of the two dental arches of the object's teeth in a single 2D image. The wide-angle camera assembly 155 can be, for example, a stand-alone camera or lens, or alternatively a fish-eye lens attachment for a smartphone, tablet, or computer. The object can use the wide-angle camera to take wide-angle input 2D images of the object's teeth (e.g., the upper dental arch and / or the lower dental arch) during orthodontic treatment.

[0081] The output 156 can include a computer system that is configured to display at least a portion of an individual's dentition and / or the difference between the input 2D image (e.g., at the current treatment stage) and one or more treatment plan stages of a model corresponding to the current treatment stage. The output 156 can include a display system, a memory, one or more processors, etc. The output can be part of a display device to display an individual's dentition. The output 156 can be implemented as part of a computer system, a dedicated intraoral scanner, etc. In some implementations, the output 156 facilitates displaying an individual's dentition by using scans taken at an earlier date and / or a remote location. The output 156 can facilitate displaying scans taken simultaneously and / or locally. The output 156 can be configured to display the expected or actual results of an orthodontic treatment plan that is applied to the dental arch scanned by the scanning system 154. These results can include a 3D virtual representation of the dental arch, a 2D image of the dental arch, or a rendering, etc. In some examples, the output can be sent to the object's dental care provider (e.g., orthodontist, dentist, dental hygienist, etc.) as one or more images, as text (e.g., a text file listing the differences), as a score (e.g., a numerical or alphanumeric score), as a 3D model, or a modified 3D model, etc.

[0082] The treatment monitoring system 158 can include a computer system that includes a memory and one or more processors, which, as described herein, are configured to monitor an object's dentition and track the movement of the object's teeth to an orthodontic treatment plan. For example, the treatment monitoring system can include multiple modules or engines for determining differences between a patient's actual tooth configuration and a planned tooth configuration during one or more treatment phases. These modules or engines can correspond to one or more processors. One or more 2D image engines 160 can implement an automated agent to receive input 2D images of an object's teeth from a wide-angle camera. The input 2D images can include, for example, ultra-wide-angle images that include an occlusal view (or occlusal perspective) of both dental arches of the object's dentition. The input 2D images can represent the actual positions of the object's teeth at a particular time or stage of orthodontic treatment. One or more 2D image engines 160 can also be configured to process the input 2D images to extract and identify tooth features of the object's teeth. Tooth features can include, for example, tooth centers, tooth profiles, tooth landmarks (e.g., pits, fissures, and / or cusps), or any other tooth features that provide the position and orientation of the object's teeth. In some examples, one or more 2D image engines 160 can also be configured to determine virtual camera parameters of the wide-angle camera assembly 155 corresponding to the input 2D images. Alternatively, in some examples, a separate analysis engine (e.g., a processor) (e.g., a virtual camera parameter engine) for determining virtual camera parameters can be included. Virtual camera parameters can include, for example, intrinsic camera parameters and / or non-intrinsic camera parameters of the camera assembly corresponding to the input 2D images.

[0083] One or more 3D model engines 162 may implement one or more automated agents to receive and process scan data or 3D tooth model data from the scanning system 154. In some examples, the treatment monitoring system 158 is configured to receive, store, and process 3D models 154 of an object dentition. The 3D models may include, for example, 3D tooth point clouds or 3D tooth mesh models, or a series of 3D tooth point clouds or 3D tooth mesh models, each corresponding to a treatment stage; alternatively, the 3D models may include segmented 3D tooth point clouds or 3D tooth mesh models of the teeth and guidance on the arrangement or structure of the segmented teeth (in some examples, the gums, dental arches, etc.) and key positions at each stage. The treatment monitoring system 158 is generally configured to generate a rendered 2D image from the 3D tooth model using the virtual camera parameters of the camera component. Thus, the rendered 2D image represents the expected or desired position of the object teeth at a specific stage or time of orthodontic treatment; this stage or time may correspond to the actual time or stage of the input 2D image. One or more 3D model engines 162 may also be configured to process the rendered 2D image to extract and identify tooth features of the object teeth. Tooth features may include, for example, tooth centers, tooth profiles, or any other tooth features that provide the position and orientation of the object teeth, as described above, for comparison with tooth features from the input 2D image.

[0084] One or more tooth comparison engines 164 may implement one or more automated agents to compare the rendered 2D image from the 3D model with the input 2D image from the wide-angle camera component. This comparison may provide an indication of whether the current position of the object teeth follows the expected or desired position according to the orthodontic treatment plan. In some examples, the tooth comparison engine provides a difference score (also referred to herein as a comparison value), which may indicate how close the current position of the object teeth is to the desired position; alternatively, the score may indicate how different the current position is from the model at a specific treatment stage. The tooth comparison engine may generate one or more difference indicators or one or more comparison indicators. The difference indicator may be one or more of the following: a difference map, a difference score, and / or a set of difference values. For example, the difference map may include a 2D or 3D representation of the object teeth that shows, highlights, or otherwise indicates the difference between the current actual tooth structure and the desired or predicted tooth structure from the treatment plan. The difference map may be one or more colored 2D images or marked 2D images showing position and / or rotation differences, or a 3D model showing position and / or rotation differences. The 2D image or 3D model may be annotated with the differences (including the magnitude of the differences). The markings may include a heat map representation of the differences. In some examples, the set of difference values may be a textual representation of the difference (and / or similarity) between the current tooth structure and the treatment plan. The textual representation may be a database (e.g., a table, a list, or any other data structure).

[0085] In some examples, one or more tooth comparison engines 164 determine and calculate new tooth movements that can be applied to an object's teeth to achieve a desired position. One or more tooth comparison engines 164 may also output a difference indicator to the object (e.g., via a message, alert, etc.) and / or the object's dental care provider and / or a third party. In some examples, one or more tooth comparison engines 164 are configured to trigger one or more alerts when the magnitude of the difference between the actual construction and the treatment plan exceeds a threshold.

[0086] One or more optional treatment modeling engines 166 may be configured to use the new tooth movements to store and / or provide instructions to achieve an orthodontic treatment plan and / or the result of an orthodontic treatment plan. One or more optional treatment modeling engines (e.g., machine learning engines) 166 may provide the result of an orthodontic treatment plan on an updated 3D tooth model. In some examples, the updated 3D tooth model may include the new tooth movements. One or more optional treatment modeling engines 166 may model the result of applying an orthodontic appliance to an individual dental arch during the orthodontic treatment plan.

[0087] As used herein, any "engine" may include one or more processors or a portion thereof. A portion of one or more processors may include some part of the hardware that is less than the entire hardware including any given one or more processors, such as a subset of registers, a portion of a processor dedicated to one or more threads of a multi-threaded processor, a time slice (during which the processor is dedicated, in whole or in part, to performing a portion of the functions of the engine), etc. Thus, the first engine and the second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on the particular implementation or other considerations, the engine may be centralized, or its functions may be distributed. The engine may include hardware, firmware, or software included in a computer-readable medium for execution by a processor. The processor converts data into new data using the implemented data structures and methods, as described with reference to the figures herein.

[0088] The engines described herein, or the engines through which the systems and devices described herein may be implemented, may be cloud-based engines. As used herein, a cloud-based engine is an engine that can run applications and / or functions by using a cloud-based computing system. All or part of the applications and / or functions may be distributed across multiple computing devices and need not be limited to one computing device. In some examples, a cloud-based engine may execute functions and / or modules accessed by an end user through a web browser or a container application without locally installing these functions and / or modules on the end user's computing device.

[0089] As used herein, "datastores" can include repositories having any suitable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other suitable known or convenient organizational formats. For example, a datastore can be implemented as software embodied in a physical computer-readable medium on a special-purpose machine, embodied in firmware, in hardware, in a combination thereof, or in a suitable known or convenient device or system. Components associated with a datastore, such as a database interface, can be considered "part" of the datastore, part of some other system component, or a combination thereof, although the physical location and other characteristics of components associated with a datastore are not important for understanding the techniques described herein.

[0090] A datastore can include data structures. As used herein, a data structure is associated with a particular way of storing and organizing data in a computer such that the data structure can be used effectively in a given context. Data structures are generally based on the ability of a computer to obtain and store data at any location in its memory, the location being specified by an address, a bit string that can itself be stored in memory and manipulated by a program. Thus, some data structures are based on using arithmetic operations to calculate the address of a data item; while other data structures are based on storing the address of a data item in their own structure. Many data structures use both principles, sometimes in a non-trivial way. Implementing a data structure generally requires writing a set of programs to generate and manipulate instances of the structure. The datastores described herein can be cloud-based datastores. A cloud-based datastore is a datastore that is compatible with cloud-based computing systems and engines.

[0091] Figure 1B is a diagram showing a schematic example of one or more 2D image engines 160a. One or more 2D image engines 160a can include a segmentation engine 168, a feature extraction engine 170, a virtual parameter engine 172, and a 2D image datastore 174. One or more modules of one or more 2D image engines 162a can be coupled to each other or to modules not shown. One or more 2D image engines 160a are configured to receive and process input 2D images of an object tooth from a wide-angle camera assembly.

[0092] One or more segmentation engines 168 may implement one or more automated agents that are configured to process input 2D images from the wide-angle camera assembly 155. One or more segmentation engines 168 may include a graphics engine for processing the input 2D images. One or more segmentation engines 168 may be configured to segment the input 2D image into individual tooth components, including segmenting the input 2D image into individual teeth. Figure 2A is an example of an input 2D image from the wide-angle camera assembly. As Figure 2A shown, the input 2D image is a single image of an occlusal view of two dental arches including the subject dentition. The wide-angle nature of the camera assembly 155 enables such a view of the two dental arches in a single 2D image. Figure 2B is an example of the input 2D image after segmentation (e.g., by the segmentation engine). As shown, the input 2D image has been segmented into individual tooth components (e.g., teeth) with lower-value image data (e.g., gums, lips, palate, etc.) removed from the image. The 2D image data store 174 may store the input 2D image, the segmented 2D image, and / or other data and provide it to other modules of the treatment monitoring system 158.

[0093] The methods and devices described herein may benefit from the use of a wide-angle camera assembly and the corresponding wide-angle input 2D images. These images may be processed as described herein and may be taken without the need to retract the cheeks. Thus, the subject may quickly and easily take the images themselves (or may have a family member, friend, or caregiver take them). Although these methods and devices may be used with a single wide-angle image, in some examples, more than one wide-angle image may be taken and used; in some examples, these images may be combined.

[0094] In some examples, the systems and methods described herein may include guidance for obtaining one or more input 2D images (e.g., the treatment monitoring system may include an image acquisition guidance engine). For example, the system may provide instructions (on the subject's handheld device) for taking an image, including text, audio, images, video, etc., for positioning and taking the input 2D image. The system may check the quality of the input 2D image taken to ensure that it is in focus, properly positioned (e.g., to provide an occlusal view), and / or includes sufficient features (e.g., includes all teeth).

[0095] One or more feature extraction engines 170 may implement one or more automated agents configured to extract dental features from an input 2D image or a segmented input 2D image. As used herein, "dental features" may include data points from one or more input 2D images that are related to the center, profile, edge, contour, vertex, vector, or surface of an object tooth. "Dental features" that provide information or details about the orientation of an object tooth are particularly useful. For example, a dental feature may include a tooth center that provides data about the position of an object tooth. Similarly, a tooth profile may provide data about both the position and orientation of an object tooth. Figure 2C An example of an input 2D image of an object tooth with dental features is shown, where the dental features include a tooth profile that is extracted and shown on the 2D image. In contrast, Figure 2D only the dental feature of the tooth center extracted from the 2D image is shown. While the tooth center provides data about the position of individual teeth, the tooth center may not readily provide sufficient data about the orientation of each tooth. The 2D image data store 174 may store the extracted features and / or other data and provide them to other modules of the treatment monitoring system 158.

[0096] One or more virtual camera parameter engines 172 may implement one or more automated agents configured to determine virtual camera parameters of a wide-angle camera assembly corresponding to an input 2D image. The virtual camera parameters of the wide-angle camera assembly may include extrinsic parameters and intrinsic parameters, where the extrinsic parameters define the position and orientation of the camera assembly relative to a world coordinate system, and the intrinsic parameters allow a mapping between camera coordinates and pixel coordinates in an image coordinate system. The virtual parameters of a wide-angle (e.g., fisheye) camera assembly may be calculated iteratively using an optimizer function. In some examples, the 2D image data store 174 may store the virtual camera parameters and / or other data and provide them to other modules of the treatment monitoring system 158.

[0097] Figure 1C is a diagram showing an example of one or more 3D model engines 162a. One or more 3D model engines 162a may include a 2D rendering engine 176, a segmentation engine 178, a feature extraction engine 180, and a 3D model data store 182. One or more modules of one or more 3D model engines 162a may be coupled to each other or to modules not shown.

[0098] The 2D rendering engine 176 may implement one or more automated agents configured to render 2D images of a 3D tooth model received from the scanning system 154. Virtual camera parameters from the wide-angle camera component of the virtual parameter engine 172 may be used to render the 2D images from the 3D model. The rendered 2D images represent the expected or desired position of the object teeth at a particular time or stage of orthodontic treatment. The 3D model data store 182 may store the rendered 2D images and / or other data and provide them to other modules of the treatment monitoring system 158.

[0099] One or more segmentation engines 178 may implement one or more automated agents configured to process the rendered 2D images from the 2D rendering engine 176. One or more segmentation engines 178 may include a graphics engine for processing the rendered 2D images. One or more segmentation engines 178 may be configured to segment the rendered 2D images into individual tooth components, including segmenting the rendered 2D images into individual teeth. As described above, in some examples, segmentation may be provided or assisted by the 3D model used to generate the rendered 2D images (e.g., the 3D model may be segmented or may include segmentation information). Figure 3A is an example of a rendered 2D image from the 2D rendering engine. As Figure 3A shown, the rendered 2D image is a single image including an occlusal view of the object dental arch. Although only a single dental arch is shown for simplicity, it should be understood that the rendered 2D image may include an occlusal view of both dental arches, similar to the input 2D image. Virtual camera parameters may be used to generate the dental arch view from the 3D model as a 2D projection of the 3D model. The 3D model data store 182 may store the input 2D images, segmented 2D images, and / or other data and provide them to other modules of the treatment monitoring system 158.

[0100] One or more feature extraction engines 180 may implement one or more automated agents configured to extract tooth features from the rendered 2D images or the segmented rendered 2D images. Tooth features may include data points from one or more input 2D images that are related to the center, profile, edge, contour, vertex, vector, or surface of the object teeth. Tooth features that provide information or details about the orientation of the object teeth are particularly useful. For example, the tooth center may provide data (translation information) about the position of the object teeth. The tooth profile may provide data about the position and orientation (e.g., rotation) of the object teeth. In some examples, as described herein, it may be beneficial (and computationally simple) to use the tooth center to determine the translational difference between the input 2D image and the treatment plan from the rendered 2D image, while the rotational difference may be determined by comparing the tooth profiles from the input 2D image and the rendered 2D image.

[0101] Figure 3B An example of a rendered 2D image of an object's teeth with tooth features is shown, the tooth features including a tooth profile that is extracted and shown on the rendered 2D image. The 3D model data store 182 can store the extracted features and / or other data and provide them to other modules of the treatment monitoring system 158.

[0102] Figure 1D FIG. is an example of a diagram showing one or more tooth comparison engines 164a. One or more tooth comparison engines 164a can include a tooth movement engine 184 and a tooth movement data store 186. One or more modules of one or more tooth comparison engines 164a can be coupled to each other or to modules not shown.

[0103] The tooth movement engine 184 can implement one or more automated agents to compare a rendered 2D image from a 3D model with an input 2D image from a wide-angle camera assembly. The comparison of the 2D images can provide an indication of whether the current position of the object's teeth follows the expected or desired position of the object's teeth according to an orthodontic treatment plan. In some examples, the comparison provides a discrete value (e.g., between 0 and 1) that indicates how closely the object's teeth follow the orthodontic treatment plan (e.g., a value of 1 would indicate that the object's teeth perfectly follow the treatment plan, while a value of 0 indicates that the object's teeth do not follow the treatment plan at all). The comparison between the input 2D image and the rendered 2D image can be used to modify the orthodontic treatment plan. For example, the comparison can be used to calculate new tooth movements required to move the teeth from the current position to the desired position. These new tooth movements can be utilized to update the orthodontic treatment plan.

[0104] For example, for each corresponding tooth in the input 2D image and the rendered 2D image (the individual teeth can be identified by segmentation), the tooth movement engine can use the tooth centers to estimate the position differences between the two types of teeth. For example, the relative translation of the tooth centers can provide an indication of the translational difference in the tooth positions between the actual planned position and the treatment plan position. Similarly, the tooth rotation between the actual (input 2D image) and the treatment plan (rendered 2D image) can be compared by comparing the tooth profiles. Segmentation provides the outline (profile) of the tooth as well as an indication of the tooth identity (e.g., tooth number). The tooth profile of the input 2D image can be compared by rotating the profile to adequately match the profile of the corresponding tooth from the rendered 2D image and / or the corresponding tooth in the 3D model.

[0105] In use, the systems and methods described herein can guide an object to take a current image (input 2D image) of the teeth in an occlusal view, compare the current view to a treatment plan for a corresponding (or future) treatment plan phase, and provide an indication of differences and / or provide guidance to a professional user (e.g., dentist, orthodontist, or other dental professional) when adjusting the treatment plan.

[0106] For example, Figure 4A A method is schematically illustrated. In Figure 4A , optionally, an object (or an agent of the object, e.g., family member, friend, caregiver, etc.) 452 can be guided by taking a wide-angle 2D image of an occlusal view (or perspective occlusal view) of the teeth. In some examples, the initial step can include attaching or connecting a wide-angle lens or adapter to an existing camera, e.g., a smartphone camera. Figure 6A An example of this is shown in Figure 7 An example of a wide-angle (e.g., fisheye) lens 703 coupled to a smartphone 701 is shown. For example, an object can be guided by the system to take an image, and the system can provide visual guidance to take a selfie while looking in a mirror, and / or can provide voice guidance to take a selfie without a mirror. The system can indicate (e.g., via an alert) that the quality of the captured image is not suitable, or can improve the quality of the captured image. For example, the system can alert the user object in the case of insufficient lighting conditions. As mentioned, a single image can be used with the wide-angle camera assembly, or multiple images can be taken continuously with slight changes in angle to assist in achieving better precision.

[0107] Figure 6B Examples of wide-angle images are shown that depict an occlusal (perspective occlusal) view of an object's teeth as described herein. These images can be taken without a retractor, and can be initially analyzed by the device to determine if the image quality is sufficient. Generally, Figure 6B the images shown in

[0108] Once sufficient input 2D images are received, a method or a system for performing the method can determine virtual camera parameters 454 for the 2D images, as described above. For example, an optimizer can be used to find the intrinsic and / or extrinsic parameters of the camera by iteratively calculating and re-calculating steps corresponding to adjusting the virtual camera parameters for the center of the teeth. The optimization enables the projected center of the teeth to be projected onto image pixels as close as possible to the center of the teeth in one or more images. In some examples, the camera assembly can be pre-calibrated and some or all of the intrinsic parameters can be provided.

[0109] These virtual camera parameters can then be used to generate a rendered 2D image 456 from a 3D model of the patient's teeth (at a corresponding or future treatment stage). The rendered 2D image can then be compared to the input 2D image by comparing features between the two images (and / or the 3D model). For example, a translational difference 458 between the teeth of the input 2D image and the rendered 2D image can be determined by identifying and comparing the centers of the teeth of the corresponding teeth in the input 2D image and the rendered 2D image. A rotational difference between the teeth of the input 2D image and the rendered 2D image can be determined by using the tooth profiles (e.g., tooth contours) of the segmented teeth in each image 460. Alternatively, in some examples, the translational difference can also or alternatively be determined from the tooth profiles instead of the centers of the teeth or in addition to the centers of the teeth, from the tooth profiles.

[0110] Based on these comparisons, a difference indicator (e.g., a difference score, a difference map, and / or a difference value) between the input 2D image and the rendered 2D image can be determined and the difference indicator can be sent, stored, and / or displayed 462. Optionally, if one or more differences exceed a threshold, one or more alerts can be sent to the subject and / or the subject's dental care provider (e.g., a professional user).

[0111] Figure 4B is another example of a flowchart depicting a method for monitoring a subject's teeth during an orthodontic treatment plan. As Figure 4A shown, the method can be automatically implemented by a system (e.g., Figure 1A one or more systems in the computing environment 100A shown in

[0112] At operation 402, the method can include receiving an input 2D image from a wide-angle camera assembly, the input 2D image having an occlusal view of a first dental arch and a second dental arch of the subject's dentition. The input 2D image can be taken by the subject using a wide-angle camera or a wide-angle (e.g., fish-eye) lens attachment of a smartphone, a tablet, or a PC. In some implementations, the input 2D image includes an occlusal view of two dental arches of the subject's teeth.

[0113] At operation 404, the method may further include identifying dental features of teeth in the first and second dental arches in the input 2D image. In some implementations, a trained machine learning model may be used to identify or extract the dental features. In other implementations, the dental features may be identified or extracted manually. The dental features may include a tooth center, a tooth profile, or other dental features that provide the position and / or orientation of the subject's teeth.

[0114] Next, at operation 406, the method may further include determining virtual parameters of the wide-angle camera corresponding to the input 2D image. The virtual parameters of the wide-angle camera may include extrinsic parameters and intrinsic parameters, where the extrinsic parameters define the position and orientation of the camera relative to the world coordinate system, and the intrinsic parameters allow a mapping between the camera coordinates and the pixel coordinates in the image coordinate system. The virtual parameters of the fisheye lens may be iteratively calculated, for example, using an optimizer function.

[0115] Next, at operation 408, the method may include receiving a 3D model of the subject's dentition. The 3D model may be generated and received from a scanning system such as a 3D dental scanning system. Next, at operation 410, the method may further include generating a rendered 2D image from the 3D model using the virtual parameters of the wide-angle camera (the virtual parameters calculated at operation 406). The rendered 2D image represents the expected or desired position of the subject's teeth.

[0116] At operation 411, the method may further include identifying dental features of teeth in the first and second dental arches in the rendered 2D image. In some implementations, a trained machine learning model may be used to identify or extract the dental features. In other implementations, the dental features may be identified or extracted manually. The dental features may include a tooth center, a tooth profile, or other dental features that provide the position and / or orientation of the subject's teeth.

[0117] Next, at operation 412, the method may further include comparing the dental features of the rendered 2D image with the dental features of the input 2D image. As described above, the comparison of the 2D images may provide an indication of whether the current position of the subject's teeth follows the expected or desired position of the subject's teeth according to the orthodontic treatment plan. In some examples, the comparison provides a discrete value between 0 and 1 that indicates how closely the subject's teeth follow the orthodontic treatment plan (e.g., a value of 1 would indicate that the subject's teeth perfectly follow the treatment plan, while a value of 0 indicates that the subject's teeth do not follow the treatment plan at all).

[0118] Optionally, at operation 414, the comparison between the input 2D image and the rendered 2D image can be used to modify the orthodontic treatment plan. For example, this comparison can be used to calculate the new tooth movements required to move the teeth from their current positions to the desired positions. These new tooth movements can be utilized to update the orthodontic treatment plan.

[0119] In some examples of the methods described herein, the comparison between the input 2D image and the reconstructed 2D image can be limited to the centers of the teeth (e.g., the centroids of each tooth). The translational differences can provide sufficient rough tracking information for home monitoring.

[0120] In some examples, the methods and devices described herein can be performed without using a wide-angle lens. For example, these methods and devices can use multiple cameras (e.g., phone cameras) to perform to generate multiple images. These images can be combined into a single (pseudo-wide-angle) image and processed as described above, or processed using one or more simplified procedures. For example, Figure 4C A schematic illustration of one such method is shown. In Figure 4C optionally, an object (or an agent of the object, e.g., a family member, a friend, a caregiver, etc.) 482 can be guided by taking a plurality of images including one or two dental arches of the teeth. The camera can be an existing camera of the object or the caregiver (user object), e.g., a smartphone camera. For example, the user object can be guided by a system to take an image, and the system can provide visual guidance to take a self-portrait photo while looking at a mirror, and / or can provide voice guidance to take a self-portrait photo without a mirror. The system can indicate (e.g., via an alert) that the quality of the taken image is inappropriate, or can improve the quality of the taken image. For example, the system can alert the user object in case of insufficient lighting conditions. A plurality of images can be continuously taken with slight changes in the angle to assist in achieving better accuracy.

[0121] Before or after taking the images, the user object can calibrate the camera 484. In some examples, the user object can be instructed to download and / or print an image of a calibration target (e.g., a grid, a checkerboard, etc.). Calibration can be performed to determine the virtual camera parameters. In some examples, calibration can help determine the spacing between the camera and the image. The intrinsic parameters and / or non-intrinsic parameters of the camera can be identified. In some examples, the plurality of taken images can be combined into a single (merged) image of each (or both) dental arch.

[0122] These virtual camera parameters can then be used to generate one or more rendered 2D images from a 3D model of the patient's teeth (at a corresponding or future treatment stage). The one or more rendered 2D images can then be compared to the input 2D image by comparing features between the two images (and / or 3D models). For example, translational differences and / or rotational differences between the teeth of the input 2D image and the rendered 2D image can be determined.

[0123] In some examples, for instance, as part of calibration, the camera can be registered to the entire jaw. Each tooth can be separate, and movement of an individual tooth relative to the entire jaw can thus be detected. This registration can be performed by: matching a tooth profile projection to the tooth profile on the image; and / or matching a cusp and / or Fisher projection to the image cusp and / or Fisher projection; and / or using a trained network that takes as input a tooth image and a tooth depth map from the same camera and produces tooth movement 488. Alternatively or additionally, the rotational difference between the teeth of the input 2D image and the rendered 2D image can be determined by using the tooth profile (e.g., tooth contour) of the segmented teeth from each image. Alternatively, in some examples, the translational difference can also or alternatively be determined from the tooth profile and / or by using the tooth center.

[0124] Based on these comparisons, a difference indicator (e.g., a difference score, a difference map, and / or a difference value) between the input 2D image and the rendered 2D image can be determined and the difference indicator 490 can be sent, stored, and / or displayed. Optionally, if one or more differences exceed a threshold, one or more alerts can be sent to the subject and / or the subject's dental care provider (e.g., a professional user).

[0125] The methods described herein can be performed by a device such as a data processing system that can include hardware, software, and / or firmware for performing many of the above steps. For example, Figure 5 is a simplified block diagram of a system 500 as described herein.

[0126] Any of these systems 500 can include or be configured to operate on a processor 554 and can include or be configured to operate with a camera assembly that includes a camera 556 and an integrated or separate wide-angle (e.g., fisheye) lens 558. The system can generally include a controller 552 that includes any or all of the above modules / engines (e.g., Figures 1A to 1Din the module / engine). For example, the controller may include an input 2D image generator 560 and an input 2D image analyzer 562 (which may together be a 2D input engine). In some examples, the controller may include or operate a virtual camera parameter identifier 564 (which may also be part of the 2D input engine), and the virtual camera parameter identifier 564 may identify the intrinsic and / or non-intrinsic parameters of the camera component. The controller may also include a rendered 2D image generator 561 and a rendered 2D image analyzer 563 (which may be included as part of the 3D model engine). A difference determination engine 565 (e.g., a tooth comparison engine) may also be included. Finally, the controller may also include and / or control an output engine 566, which may trigger and present an alert, store, send, and / or display one or more comparison (e.g., difference) indicators.

[0127] Generally, any of these processing systems may include at least one processor that communicates via a bus subsystem with a plurality of peripheral devices (which may include all or some of the above components). These peripheral devices typically may include a storage subsystem (e.g., a memory subsystem and a file storage subsystem), a set of user interface input and output devices, and an interface to an external network (including the public switched telephone network). The interface may be a modem and a network interface and may be coupled via a communication network interface to a corresponding interface device in other data processing systems. These systems may include terminals or low-end personal computers or high-end personal computers, workstations, or mainframes.

[0128] These systems may include a user interface input device, which may include a keyboard and may also include a pointing device and a scanner. The pointing device may be an indirect pointing device, such as a mouse, a trackball, a touchpad, or a graphics tablet, or a direct pointing device, such as a touch screen incorporated into a display. Other types of user interface input devices, such as a voice identification system, may be used.

[0129] The user interface output device may include a printer and a display subsystem that includes a display controller and a display device coupled to the controller. The display device may be a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), or a projection device. The display subsystem may also provide a non-visual display, such as an audio output.

[0130] The storage subsystem can maintain the basic programming and data structures that provide the functions of the present invention. The software modules discussed above can be stored in the storage subsystem. The storage subsystem can include a memory subsystem and a file storage subsystem. The memory subsystem can include a plurality of memories, including a main random access memory (RAM) for storing instructions and data during program execution and a read-only memory (ROM) in which fixed instructions can be stored.

[0131] The file storage subsystem can provide persistent (non-volatile) storage for program and data files and typically includes, for example, at least one hard disk drive and at least one floppy disk drive (with associated removable media). Other devices may also be present, such as CD-ROM drives and optical disk drives (both with their associated removable media). In addition, the system can include a drive of the removable media cartridge type. One or more drives can be located at a remote location, such as in a server on a local area network or at a site on the World Wide Web of the Internet.

[0132] The "bus subsystem" can generally include any mechanism that enables the various components and subsystems to communicate with each other as expected. The other components do not have to be located in the same physical location. Thus, for example, some parts of the file storage system can be connected via various local area network media or wide area network media (including telephone lines). Similarly, the input device and the display do not have to be located in the same location as the processor, although it is expected that the present invention will most often be implemented in the environment of PCs and workstations. The bus subsystem can be a single bus or can include multiple buses, such as a local bus and one or more expansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus, or PCI), as well as serial ports and parallel ports. Network connections are typically established through devices such as network adapters located on one of these expansion buses or modems located on serial ports. The client computer can be a desktop system or a portable system.

[0133] Various alternatives, modifications, and equivalents can be used in place of the above components. Although computer-aided techniques can be used to determine the final positions of the teeth, a professional user can move the teeth to their final positions by independently manipulating one or more teeth while satisfying the prescription constraints.

[0134] Additionally, the techniques described herein can be implemented in hardware, software, or a combination of both. These techniques can be implemented in a computer program executed on a programmable computer, each programmable computer including a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), and appropriate input and output devices. The program code is applied to data input by using the input device to perform the functions and generate output information. The output information is applied to one or more output devices.

[0135] Each program can be implemented in a high-level procedural or object-oriented programming language to operate in conjunction with a computer system. However, if desired, these programs can be implemented in assembly language or machine language. In any case, the language can be a compiled or interpreted language.

[0136] Each such computer program can be stored on a storage medium or device (e.g., a CD-ROM, hard disk, or disk) readable by a general or special purpose programmable computer for configuring and operating the computer to execute the processes when the storage medium or device is read by the computer. The system can also be implemented as a computer-readable storage medium configured with a computer program, where the storage medium so configured causes the computer to operate in a specific and predefined manner.

[0137] Thus, any method described herein (including a user interface) can be implemented as software, hardware, or firmware and can be described as a non-transitory computer-readable storage medium that stores a set of instructions executable by a processor (e.g., a computer, tablet, smartphone, etc.), which when executed by the processor, causes the processor to control to perform any steps including but not limited to the following: display, communicate with a user, analyze, modify parameters (including timing, frequency, and intensity, etc.), determine, issue an alert, etc.

[0138] As mentioned above, any device and method can include calibration using a calibration standard. For example, a patient or caregiver (e.g., a user object) can use a printed calibration standard (e.g., a calibration standard pattern or target, e.g., a checkerboard pattern) to calibrate the user object's camera (e.g., a cell phone camera, a stand-alone camera, a tablet camera, etc.). In some examples, a printed calibration standard can be provided to the user object and / or the user object can be instructed to print out the calibration standard (using a home printer). The calibration standard can be included in the packaging. In some examples, the packaging of the camera or a camera accessory (e.g., an adapter, etc.) for securing all or part of the optics (e.g., a wide-angle lens) can be configured as a calibration jig that can position the camera in a predetermined position relative to the calibration standard (optionally, using an adapter).

[0139] For example, Figure 8A schematically shows an example of a calibration standard being used with a user's phone. In Figure 8A , phone 801 includes an attachment 803 on the phone to interface with the internal phone camera. The attachment may include one or more lenses (e.g., a wide-angle lens). The camera can be calibrated by taking a photo of a calibration target 805 (shown as a checkerboard pattern in this example). In some examples, the checkerboard pattern can be part of a box or package 810 that can be provided with the device (e.g., an attachment that can be coupled to the user's camera). In Figure 8A and Figure 8B , the calibration pattern is shown printed or attached to the package lid. The package may also include a code or link 807 (e.g., a Quick Response (“QR”) code, a web address, etc.), which can provide a link to instructions or application software that can be downloaded to the user's camera device (e.g., a mobile phone, a tablet, etc.). The camera can be linked or registered to the user (e.g., a patient). The product (device) can be registered, and the application software can assist in taking and / or storing and / or processing the images captured by the user's phone. The code or link (e.g., a QR code, a barcode, an alphanumeric code, etc.) can be the calibration target or can be part of the calibration target, as Figure 8D shown.

[0140] In some examples, a calibration frame or jig can be used to hold the user's phone at a fixed predetermined distance from the calibration target. For example, the packaging of the device (e.g., a box) 810 can include the calibration target and a positioning location where the phone can be placed, held, or fixed. Figure 8C shows an example of a box configured to include a calibration target 805 (e.g., a calibration pattern) printed on the box and a bracket or fixture 812 on the box. In some examples, the box includes a hole or opening on one side against which the camera can be supported to image the calibration pattern printed or attached inside the box. Thus, the camera can image the calibration target at a predetermined (known) fixed distance 814, and thus can calibrate the optical characteristics (focus, etc.) based on the distance. In some examples, multiple calibration targets (at the same or different distances) can be included. For example, the packaging can include multiple calibration targets. In some cases, multiple targets can be seen from the same location where the camera is placed (or held). The targets can be compact (as shown in the example of Figure 8A ), or can be larger (as shown in Figure 8C ).

[0141] Although the preferred embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. Many variations, changes, and alternatives will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. A variety of different combinations of the embodiments described herein are possible, and these combinations are considered to be part of the present disclosure. In addition, all features discussed in connection with any one of the embodiments herein can be readily applied to other embodiments herein. The appended claims are intended to define the scope of the invention and thus will cover methods and structures within the scope of these claims and their equivalents.

[0142] When a feature or element is referred to herein as being "on" another feature or element, it can be directly on the other feature or element or intervening features and / or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements. It will also be understood that when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element, or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements. Although described or shown with respect to one example, the features and elements so described or shown can be applied to other examples. Those skilled in the art will also recognize that references to a structure or feature that is "adjacent" to another feature can have portions that overlap or are located beneath the adjacent feature.

[0143] The terminology used herein is for the purpose of describing particular examples only and is not intended to limit the invention. For example, as used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated to " / ".

[0144] Spatial relative terms such as "under", "below", "lower", "over", "upper", etc. may be used herein to facilitate the description of the relationship of one element or feature to another or other elements or features as shown in the accompanying drawings. It should be understood that, in addition to the orientation depicted in the drawings, spatial relative terms are intended to encompass different orientations of the device in use or operation. For example, if the device in the figure is inverted, an element described as "under" or "beneath" another element or feature will be oriented "over" that other element or feature. Thus, the exemplary term "under" can include both upward and downward orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein can be interpreted accordingly. Similarly, unless specifically indicated otherwise, the terms "upwardly", "downwardly", "vertical", "horizontal", etc. used herein are for purposes of explanation only.

[0145] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context otherwise indicates. These terms may be used to distinguish one feature / element from another. Thus, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0146] Unless the context otherwise requires, throughout the specification and the appended claims, the word "comprise" and examples such as "comprises" and "comprising" mean that various components can be used together in methods and articles (e.g., compositions and devices including devices and methods). For example, the term "comprise" will be understood to mean including any of the recited elements or steps without excluding any other elements or steps.

[0147] Generally, any devices and methods described herein should be understood as inclusive, but all or subsets of components and / or steps may alternatively be exclusive and may be phrased as "consisting of" various components, steps, subcomponents or substeps, or alternatively "consisting essentially of" various components, steps, subcomponents or substeps.

[0148] As used in this specification and the claims, including as used in the examples, unless otherwise expressly specified, all numbers may be read as if prefaced by the word "about" or "approximately" even if the word does not expressly appear. When describing size and / or position, the phrase "about" or "approximately" may be used to indicate that the value and / or position described is within a reasonable expectation range of the value and / or position. For example, a numerical value may be + / -0.1%, + / -1%, + / -2%, + / -5%, + / -10% etc. of a specified value (or value range). Any numerical value given herein should also be understood to include the approximate or approximate value of that value unless the context otherwise indicates. For example, if the value "10" is disclosed, then "about 10" is also disclosed. Any numerical range described herein is intended to include all sub-ranges subsumed therein. It should also be understood that when a value is disclosed, values "less than or equal to" that value, values "greater than or equal to" that value, and possible ranges between the values are also disclosed as would be appropriately understood by a person skilled in the art. For example, if the value "X" is disclosed, then "less than or equal to X" and "greater than or equal to X" are also disclosed (where X is a numerical value). It should also be understood that throughout the application, data is provided in a variety of different formats, and this data represents ranges of endpoints and starting points, and any combination of data points. For example, if a particular data point "10" and a particular data point "15" are disclosed, then it is understood that greater than, greater than or equal to, less than, less than or equal to, equal to 10 and 15, and between 10 and 15 are all considered disclosed. It should also be understood that each unit number between two particular unit numbers is also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13 and 14 are also disclosed.

[0149] Although the various illustrative examples are described above, many changes among the various examples can be made without departing from the scope of the invention as described in the claims. For example, in alternative examples, the order of execution of the various method steps can often be changed, and in other alternative examples, one or more method steps can be completely skipped. Optional features of the various apparatus and system examples can be included in some examples and not included in other examples. Accordingly, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as described in the claims.

[0150] The examples and figures included herein illustrate specific examples by way of illustration and not limitation, in which separate subject matter may be practiced. As previously mentioned, other examples may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. These examples of the inventive subject matter may be referred to herein individually or collectively by the term "invention" for convenience only, and are not intended to voluntarily limit the scope of the present application to any single invention or inventive concept if in fact more than one invention or inventive concept is disclosed. Thus, although specific examples have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific examples shown. The present disclosure is intended to cover any and all modifications or variations of various examples. Combinations of the above examples and other examples not specifically described herein will be apparent to those skilled in the art after reading the above description.

Claims

1. A method, comprising: Receiving one or more input two-dimensional (2D) images of a patient's current teeth captured by a camera device, the one or more input 2D images corresponding to one or more occlusal views of the patient's current teeth in the current configuration of the patient's maxilla and / or mandible; Accessing one or more rendered 2D images representing target positions of the patient's current teeth in a planned treatment phase, wherein the one or more rendered 2D images are based on virtual camera parameters corresponding to the one or more input 2D images; Determining one or more rotational differences between the patient's current teeth and the planned treatment phase by comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images; and Outputting a difference indicator that indicates the one or more rotational differences between the patient's current teeth and the planned treatment phase.

2. The method according to claim 1 further comprises: Estimating translational movement of the patient's current teeth between the one or more input 2D images and the one or more rendered 2D images using the centers of the teeth in the one or more input 2D images and the one or more rendered 2D images.

3. The method according to claim 1, wherein, The one or more rendered 2D images are rendered based on a three-dimensional (3D) model of the patient's maxilla and / or mandible in the planned treatment phase.

4. The method according to claim 1, wherein Rendering the one or more rendered 2D images using the virtual camera parameters corresponding to the one or more input 2D images.

5. The method according to claim 1 further comprises: Generating profiles of the one or more input 2D images and the one or more rendered 2D images by segmenting the one or more input 2D images and the one or more rendered 2D images.

6. The method according to claim 1 further comprises: Estimating translational movement of the patient's current teeth by comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images.

7. The method according to claim 1, wherein Comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images includes matching tooth profile projections, matching cusps, and / or using a trained network.

8. The method according to claim 1, wherein The one or more input 2D images are frames of a video.

9. The method according to claim 1 further comprises: Generating the one or more rendered 2D images.

10. The method according to claim 1, wherein For each of the one or more input 2D images, the virtual camera parameters include parameters corresponding to the position of the camera device in space used to capture the input 2D image.

11. The method according to claim 1, wherein Determining the one or more rotational differences includes estimating the actual rotational movement of one or more of the patient's current teeth by comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images.

12. The method according to claim 11, wherein, The estimated rotational movement of the teeth corresponds to the tooth rotation required to align the teeth in the one or more input 2D images with the teeth in the one or more rendered 2D images.

13. A system, comprising: One or more processors; And A memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, the method comprising: Receiving one or more input two-dimensional (2D) images of a patient's current teeth captured by a camera device, the one or more input 2D images corresponding to one or more occlusal views of the patient's current teeth in the current configuration of the patient's maxilla and / or mandible; Accessing one or more rendered 2D images representing the target positions of the patient's current teeth in a planned treatment phase, wherein the one or more rendered 2D images are based on virtual camera parameters corresponding to the one or more input 2D images; Determining one or more rotational differences between the patient's current teeth and the planned treatment phase by comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images; and Outputting a difference indicator that indicates the one or more rotational differences between the patient's current teeth and the planned treatment phase.

14. The system according to claim 13, further comprising: An attachment for the camera device, the attachment including one or more optical components.

15. The system according to claim 14, wherein The one or more optical components include a wide-angle lens.

16. The system according to claim 13, further comprising: The camera device configured to capture one or more occlusal views of the patient's current teeth.

17. The system according to claim 13, wherein For each of the one or more input 2D images, the virtual camera parameters include parameters corresponding to the position of the camera device in space used to capture the input 2D image.

18. The system according to claim 13, wherein, Determining the one or more rotational differences includes estimating an actual rotational movement of one or more of the patient's current teeth by comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images.

19. The system according to claim 18, wherein, The estimated rotational movement of the teeth corresponds to the tooth rotation required to align the teeth in the one or more input 2D images with the teeth in the one or more rendered 2D images.

20. The system according to claim 13, wherein, Comparing the profiles of corresponding teeth in the one or more input 2D images with the profiles of corresponding teeth in the one or more rendered 2D images includes matching tooth profile projections, matching cusps, and / or using a trained network.

21. A non-transitory computer-readable medium having stored thereon computer program instructions, the computer program instructions comprising: Receiving one or more input two-dimensional (2D) images of a patient's current teeth captured by a camera device, the one or more input 2D images corresponding to one or more occlusal views of the patient's current teeth in the current configuration of the patient's maxilla and / or mandible; Accessing one or more rendered 2D images representing the target positions of the patient's current teeth in a planned treatment phase, wherein the one or more rendered 2D images are based on virtual camera parameters corresponding to the one or more input 2D images; Determine one or more rotational differences between the current teeth of the patient and the planned treatment phase by comparing the profiles of the corresponding teeth of the one or more input 2D images with the profiles of the corresponding teeth of the one or more rendered 2D images; and Output a difference indicator that indicates the one or more rotational differences between the current teeth of the patient and the planned treatment phase.

22. The non-transitory computer-readable medium according to claim 21, wherein, The one or more input 2D images are frames of a video.

Images