Processing 2D intraoral images and rendering novel views of a patient

Abstract

The systems, devices, and methods disclosed herein relate to systems, methods, and devices for enabling a patient to capture intraoral images and for processing these images to assist a dentist or orthodontist in evaluating the status of a patient's teeth and treatment plan or progress.

Description

Related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 561,123, filed March 4, 2024, pursuant to 35 USC § 119(e), the entire contents of which are incorporated herein by reference. Background Technology

[0002] Intraoral images are used to assist dentists or orthodontists in determining the conditions of a patient's mouth, including tooth placement, alignment, and gingival condition. Intraoral images can help evaluate overall dental monitoring and track the progress of treatment plans designed to move specific teeth to the desired position and alignment.

[0003] However, the conventional methods of capturing and using such images have several drawbacks. These drawbacks include the need for patients to visit a dentist's (or orthodontist's) office to obtain the images. A possible solution to this problem is to ask patients to provide images captured using a mobile phone or similar device, thus eliminating the need to visit the dentist's office in person.

[0004] While this is more convenient for patients, the quality of the captured images may be affected, and in some cases, the images may be unusable. One way to address this is to use a buccal retractor, which can be used to pull the patient's cheeks and lips away from the teeth so that the images show their teeth more clearly. The patient is then asked to take multiple photographs of their teeth from different angles, and these photographs are provided to the dentist for evaluation.

[0005] Unfortunately, this approach also has drawbacks, as monitoring treatment plans to adjust one or more teeth in a patient's teeth requires consistent and relatively high-quality images. This can be difficult for patients using their own devices due to the lack of a common viewpoint for image capture (i.e., a viewpoint based on a consistent or known camera position relative to the mouth, camera rotation or other perspective changes during image capture, or alignment of the camera optics with the mouth) and varying image capture conditions (such as differences in ambient lighting, camera focus, camera or patient movement, or other camera settings). Furthermore, current capture procedures typically involve single-image acquisition, thus excluding the possibility of fusing information from multiple images to mitigate temporal issues that could adversely affect image quality.

[0006] Potential changes in camera position, camera settings, and environmental conditions during image capture attempts can render images unusable and / or unsuitable for determining the information a dentist needs. This is because a dentist may need to know the actual tooth location (rather than the apparent location affected by camera movement or differences in camera position during image capture), the actual gingival tissue color, or other indications of the current state of the patient's teeth and gums. Furthermore, the use of automated image processing techniques typically requires consistent image capture conditions or at least a known set of parameters describing these conditions so that images can be adjusted or normalized before evaluation.

[0007] Systems, apparatus, and methods are desired to enable patients to more effectively capture and process intraoral images to assist dentists or orthodontists in evaluating the condition of a patient's teeth and in developing or progressing treatment plans. Various embodiments of this disclosure individually and collectively address this and other objectives. Summary of the Invention

[0008] In some embodiments, the systems, apparatuses, and methods disclosed and / or described herein relate to systems, methods, and apparatuses for enabling a patient to capture intraoral images (e.g., photographs, videos) and process these images and videos to assist a dentist, orthodontist, or other dental professional in evaluating the condition of a patient's dentition, including teeth, occlusion, masticatory movements, temporomandibular joint connectivity, and the formulation or progress of treatment plans. In one embodiment, the captured images are derived from video clips obtained by the patient.

[0009] In some embodiments, the captured images are used as the basis for a reconstructed representation of the current state of the patient's teeth by "fusion" information obtained from multiple images. The reconstruction can be used to generate novel images of designated areas of the mouth and / or images taken from a desired perspective or environmental conditions. This allows dentists to better determine the current state of the patient's teeth and evaluate and / or refine treatment plans. The described and / or disclosed methods also provide dentists or orthodontists with the ability to obtain a consistent view of the patient's teeth as they monitor treatment plans over time.

[0010] In one embodiment, as part of developing a reconstructed or fused representation, images provided by the patient can be evaluated to determine whether they are suitable for developing a reconstructed representation. In one embodiment, the evaluation may involve image quality assessment, filtering based on one or more features, combining one or more images, or determining whether one or more images can be classified by a trained model.

[0011] In one embodiment, this disclosure relates to a method for capturing intraoral images of a patient, processing the captured images to generate reliable and consistent images or sets of images of the patient's teeth and gingiva, and using one or more of the generated images to evaluate the current state of the patient's teeth and / or gingiva. Based on this information, the progress of moving teeth to the desired position and alignment can be determined.

[0012] The disclosed and / or described image capture and processing technologies enable patients to capture sets of images directly or from video using mobile devices and provide these images to a dentist or orthodontist for the development or monitoring of progress in a treatment program aimed at moving the patient’s teeth to the desired position and alignment.

[0013] In some embodiments, a dentist or orthodontist may receive captured images or videos, generate an image set based on the video, apply one or more image processing techniques to evaluate the suitability of using the images, and then use one or more suitable images as input to a process for reconstructing a representation of the patient's mouth and teeth. The reconstructed representation can then be used to generate additional images depicting the patient's teeth from a desired or baseline perspective to assist the dentist or orthodontist in determining the current state of the teeth and, based on this, determining the next steps or changes in the treatment plan.

[0014] Images provided by the patient and / or generated from the reconstructed representation can also be used as input for training a model, or as input for a trained model that operates as a classifier to identify an aspect of the patient's mouth or teeth. Such models can be used for diagnostic, monitoring, or visualization functions based on intraoral images, where they benefit from consistent, high-quality data sources.

[0015] In some embodiments, the images may be one or more captured videos, which can be used as the basis for developing a reconstructed representation of the current state of the patient's teeth by "fusion" information obtained from the one or more videos. The reconstruction can be used to generate novel videos of designated areas of the mouth and / or videos taken from desired perspectives or environmental conditions. This allows dentists to better determine the current state of the patient's dentition and to evaluate and / or refine treatment plans. The described and / or disclosed methods also provide dentists or orthodontists with the ability to obtain a consistent view of the patient's dentition as they monitor treatment plans over time.

[0016] In one embodiment, as part of developing a reconstructed or fused representation, videos provided by patients can be evaluated to determine whether they are suitable for developing a reconstructed representation. In one embodiment, the evaluation may include image quality assessment of the videos, filtering videos based on one or more features, combining one or more videos, or determining whether one or more videos can be classified by a trained model.

[0017] In one embodiment, this disclosure relates to a method for capturing intraoral video of a patient, processing the captured video to generate a reliable and consistent video or set of videos of the patient's oral cavity, and using one or more of the generated videos to evaluate the current state of the patient's oral cavity, including teeth, occlusion, masticatory movements, temporomandibular joint connection, and other aspects of oral health. Based on this information, progress in moving teeth to desired positions and alignments, temporomandibular joint connection or position, etc., can be determined.

[0018] The disclosed and / or described video capture and processing technologies enable patients to use mobile devices to capture one or more videos and provide these videos to a dentist or orthodontist for use in developing or monitoring the progress of a treatment plan.

[0019] In some embodiments, a dentist or orthodontist may receive captured video, apply one or more image processing techniques to evaluate the suitability of using the video (such as each frame of the video), and then use one or more suitable videos, portions of videos, etc., as input to a process for reconstructing a representation of movement within the patient's oral cavity. The reconstructed representation can then be used to generate additional video depicting the patient's oral cavity from a desired or baseline perspective to assist the dentist or orthodontist in determining the current state of the oral cavity and, based on this, determining the next steps or changes in the treatment plan.

[0020] Videos provided by the patient and / or generated from the reconstructed representation can also be used as input for training a model, or as input for a trained model that operates as a classifier to identify an aspect of the patient's mouth or teeth, including the jaw and teeth. Such models can be used for diagnostic, monitoring, or visualization functions based on intraoral video, where they benefit from consistent, high-quality data sources.

[0021] In one embodiment, examples of the disclosed method may include the following steps, stages, functions, or operations:

[0022] Images (multiple) or videos are acquired using a mobile device. This can be performed by a patient using their phone's camera. In one embodiment, desired timing (such as the time of acquisition or the time interval between captured images), location, environmental conditions (light, temperature), camera settings, and / or other characteristics of the captured images or videos can be determined and stored by an application or set of instructions installed on the patient's device. In one embodiment, a user interface or audio clip can assist the patient in positioning the camera to capture a desired set of images (e.g., photos, videos) (e.g., by providing guidance on proper camera positioning for the purpose of acquiring the desired images or videos, such as how or where to move the camera, such as panning or rotating). In one embodiment, an application installed on the mobile device can control the operation of the camera to assist in capturing one or more images at a specific camera setting (e.g., by setting the depth of focus (or focus distance), aperture, focal length, or illumination, as a non-limiting example). In one embodiment, the orientation of the mobile device or other characteristics related to the images or videos can be determined and provided to a dentist or orthodontist along with the images or videos (e.g., in the form of metadata and / or sensor data). In one embodiment, sensor data (such as data from a light sensor or gyroscope in a mobile device) can be used to guide a patient when locating the mobile device or changing the environmental conditions in which images are collected. In another embodiment, sensor data can be used during image processing to correct or modify them to remove artifacts or other undesirable aspects.

[0023] The video and / or images are provided to the patient directly or via access to a remote platform, such as a multi-tenant or SaaS platform, by a dentist or orthodontist. In the case of a remote platform, each dentist or orthodontist may be associated with an account on the SaaS platform. Such a SaaS platform may include one or more services or applications for processing and / or evaluating the received images or videos. In one embodiment, certain image processing functions may be performed on a mobile device. In one embodiment, processing the captured images or videos may include some form of anonymization, encoding, or encryption before transmission to the remote platform or system. If a patient captures video, the video may then be processed to generate one or more images. As a non-limiting example, this may include sampling or filtering.

[0024] Images (e.g., photographs, videos) may be evaluated or otherwise filtered to obtain a desired set of images for the reconstruction process. As further disclosed and / or described herein, this processing stage may include one or more of the following for the purpose of evaluating (and in some cases, correcting) image quality (still images, a video frame, or multiple video frames): detecting poor or inadequate image quality (such as by determining the degree of blur in an image or video frame or whether the initial image is uncorrectable); performing comparisons with previously obtained images to assist in identifying changes in tooth position or alignment (or misalignment) between captures and / or to assist in evaluating the quality of images provided by the patient. In one embodiment, this may be implemented as a form of supervised learning in which a trained model generates an indication of the relative quality or utility of images (as a non-limiting example). In one embodiment, the dataset used to train the image quality assessment model may be obtained by comparing labels or annotations assigned to images by multiple processes or techniques to identify annotation errors and using that information as part of the training data for the model.

[0025] Then, the filtered or otherwise evaluated image (e.g., photograph, video) can be used as input to one or more reconstruction processes to generate a baseline representation for synthesizing one or more novel views as still images or moving videos. Examples of image processing techniques that can be used to generate reconstructed representations (or fusions) include, but are not limited to: differentiable volume rendering, such as Neural Radiation Field (NeRF), Structure from Motion, and Gaussian Splatting, intelligent image stitching, or fine-tuned “personalized” diffusion models.

[0026] The reconstructed representation is then used to generate one or more images or videos that represent aspects of the patient's teeth, jaw, or other parts of the oral cavity as viewed from the dentist's or orthodontist's desired perspective, location, environmental conditions, or other aspects of interest. This allows the dentist or orthodontist to view the patient's teeth from a different perspective than that in images (e.g., photographs, videos) obtained from the patient. It also provides baseline images and / or videos for the dentist or orthodontist to use when comparing image or video data obtained at different times.

[0027] The generated one or more views are evaluated to determine the current state of the patient's teeth and treatment progress compared to the treatment plan. This process can be used to monitor the progress of the treatment plan. A trained model can be used to create a visual representation of the patient's dental anatomy, such as the patient's teeth, from one or more static or dynamic views, including novel views. The visual representation of the current state of the teeth can be evaluated to determine whether treatment is progressing as planned, such as by comparison with a model of the patient's dentition in the treatment plan. In some embodiments, the visual representation of the current state of the teeth can be evaluated to make a diagnosis.

[0028] Machine learning models can be trained to act as classifiers using multiple generated images (e.g., photos, videos). This is an optional stage, but includes: training the model as a classifier using multiple generated images for diagnosis, treatment planning, or determining the state or changes in state of a patient's teeth during training. In one embodiment, images used as part of the training dataset can be obtained from one or more of the following sources: reconstructed representations disclosed and / or described herein when operated under multiple conditions or input parameters; and image sets (e.g., photo sets, video sets) generated by successive blurring processes, in which it is advantageous to determine the change in blur level between images after successive application of blur filters, and if the change indicates that the original image is sufficiently sharp, one or more blurred images from the original and blurred images can be used as part of the training dataset for the machine learning model.

[0029] The trained model can be operated to perform diagnostic, evaluation, or other functions, such as helping to generate updated treatment plans and associated dental appliances, such as orthodontic appliances.

[0030] Trained models can be used to aid in dental visualization, diagnosis, monitoring, and other dental functions. For example, a model can be used to create a visual representation of a patient's dental anatomy (such as the patient's teeth) from one or more static or dynamic views, including novel views. A dental professional's model can monitor changes in a patient's gingiva over time from the same perspective (such as a novel view) and can make a diagnosis based on, for example, changes in gingival recession. A dental professional can monitor changes in a patient's occlusion, masticatory movements, or temporomandibular joint connections over time from the same novel view perspective and can make a diagnosis based on, for example, different jaw movements and / or initial occlusal contacts. Treatment plans, including updated dental appliances, can be developed based on the diagnosis or evaluation.

[0031] In one embodiment, this disclosure relates to a system for capturing intraoral images (e.g., photographs, videos) of a patient, processing the captured images and / or generating reliable and consistent images or image sets (e.g., photo sets, video sets) of the patient's teeth, jaws, and gingiva, and using the generated images or image sets to evaluate the current state of the patient's tooth position, temporomandibular joint connection, occlusion, masticatory movements, or other aspects of the oral cavity. The system may include a set of computer-executable instructions stored in (or on) memory or data storage elements (such as non-transitory computer-readable media) and one or more electronic processors or coprocessors. When executed by the processor or coprocessor, the instructions cause the processor or coprocessor (or devices to which they are part) to perform a set of operations implementing embodiments of one or more of the disclosed methods.

[0032] In one embodiment, this disclosure relates to a non-transitory computer-readable medium containing a set of computer-executable instructions, wherein when the set of instructions is executed by one or more electronic processors or coprocessors, the processors or coprocessors (or devices in which they are part) perform a set of operations implementing embodiments of one or more of the disclosed methods.

[0033] In some embodiments, the systems and methods disclosed herein may be provided through a SaaS or multi-tenant platform. The platform provides access to multiple entities, each with a separate account associated with a data storage device. For example, each account may correspond to a dentist or orthodontist, a patient, an entity, a set or class of entities, a set or type of patients, an insurance company, or an organization. Each account may access one or more services, wherein the service set is instantiated within its account, and implements one or more of the methods or functions disclosed and / or described herein.

[0034] Upon review of the detailed embodiments and the accompanying drawings, other objects and advantages of the disclosed systems, apparatuses, and methods will be apparent to those skilled in the art. Throughout the drawings, the same reference numerals and descriptions indicate similar but not necessarily identical elements. While the embodiments disclosed or described herein are readily adaptable to various modifications and alternatives, specific embodiments are illustrated by way of example in the drawings and described in detail herein. However, the embodiments of this disclosure are not limited to the exemplary or specific forms described. Rather, this disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0035] As used herein, the terms “invention,” “the invention,” “this invention,” “this disclosure,” or “the disclosure” are intended to refer broadly to all subject matter disclosed in this document, the drawings or figures, and the claims. Statements containing these terms do not limit the meaning or scope of the disclosed subject matter or the claims. The various embodiments covered by this disclosure are defined by the claims, not by this summary. This summary is a high-level overview of various aspects of this disclosure and introduces some of the concepts further described in the following detailed description section. This summary is not intended to identify key, necessary, or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to appropriate portions of the specification throughout, any or all drawings or figures, and each claim. Attached Figure Description

[0036] The embodiments of this disclosure will be described with reference to the accompanying drawings, in which:

[0037] Figure 1(a) is a flowchart or diagram illustrating a process, method, operation or function that may be performed in an implementation of an embodiment of the disclosed system and method;

[0038] Figure 1(b) is a flowchart or diagram illustrating a process, method, operation, or function that can be performed to evaluate the progress of a treatment regimen in an implementation of an embodiment of the disclosed system and method.

[0039] Figure 1(c) is a flowchart or diagram illustrating a process, method, operation or function that can be performed for image capture or image quality assessment in an implementation of an embodiment of the disclosed system and method.

[0040] Figure 1(d) is a view of the user interface used to display the generated image and the original image;

[0041] Figure 1(e) is a flowchart or diagram illustrating a process, method, operation or function that can be performed for fuzzy image quality assessment in an implementation of an embodiment of the disclosed system and method.

[0042] Figure 1(f) is a flowchart or diagram illustrating a process, method, operation, or function that can be performed in an implementation of an embodiment of the disclosed system and method to generate a novel view using a differentiable volume rendering model.

[0043] Figure 1(g) is a flowchart or diagram illustrating a process, method, operation, or function that can be performed in an implementation of an embodiment of the disclosed system and method for generating and manipulating a deformable 3D model.

[0044] Figure 2This is a diagram illustrating elements or components that may be present in a computer device or system configured to implement methods, processes, functions, or operations according to some embodiments of the systems, apparatuses, and methods disclosed herein; and

[0045] Figures 3 to 5 This is a diagram illustrating the architecture of a multi-tenant or SaaS platform that can be used to implement embodiments of the systems and methods disclosed herein.

[0046] Note that the same reference numerals are used throughout this disclosure and in the accompanying drawings to refer to similar parts and features. Detailed Implementation

[0047] Capturing high-quality, consistent image data (such as still images or videos suitable for clinical evaluation and dental treatment planning) over time presents numerous challenges due to variations in lighting, the capture equipment used, changes in dental conditions, and other factors. These challenges can be exacerbated when patients capture and provide image data. As discussed in this paper, we provide solutions and improvements for capturing and processing digital image data, offering numerous advantages. For example, digital image data can be captured at different orientations, positions, focal lengths, and different focusing distances or depths of focus. It is even possible to capture digital image data at inconsistent orientations, positions, focal lengths, and depths of focus. Complete dental models can be generated from digital image data. The quality of image data or portions thereof can be evaluated and assigned quality metrics. Image data can be selected for generating complete dental models, articular connection models, or other models of the oral cavity based on its quality, including focus, appropriate exposure, or other factors as discussed in this paper.

[0048] The systems and methods discussed in this paper contribute to more consistent progress tracking in both 2D and 3D image spaces of static and dynamic (e.g., moving) subjects, as well as in treatment planning and other dental applications. They improve the operation of dental computational systems and processes by helping to generate high-quality, accurate data that would otherwise be unavailable.

[0049] In some embodiments, the systems, apparatuses, and methods disclosed and / or described herein relate to systems, methods, and apparatuses for enabling patients to capture intraoral images (e.g., photographs, videos) and process these images to assist dentists, orthodontists, or other dental professionals in evaluating the state of a patient's oral health (including teeth, jaws, etc.) and in the development or progress of treatment plans.

[0050] For example, in some embodiments, captured images (e.g., photographs, videos) are used as the basis for a reconstructed representation of the current state of a patient's teeth by "fusion" information obtained from multiple images. The reconstruction can be used to generate novel images of designated areas of the mouth and / or images taken from desired viewpoints, patient movement, or environmental conditions. This allows dentists to better determine the current state of a patient's dentition and to evaluate and / or refine treatment plans. The described and / or disclosed methods also provide dentists or orthodontists with the ability to obtain a consistent view of a patient's dentition as they monitor treatment plans over time.

[0051] Figure 1(a) is a diagram illustrating a set of 100 processes, methods, operations, or functions that may be performed in an implementation of an embodiment of the disclosed system and method. As shown, an embodiment may include one or more steps or stages.

[0052] At box 102, images or videos of the patient's oral cavity are acquired using a mobile device. These images or videos may include hard tissues such as teeth, soft tissues such as gums, lips, and / or cheeks. In some embodiments, the images or videos may include extraoral features or anatomical structures, such as facial features or anatomical structures, including the nose, eyes, etc. In one embodiment, timing (such as the time of acquisition or the time interval between captured images), location, environmental conditions (light, temperature), camera settings, and / or other characteristics of the captured images or videos may be determined and stored by an application or set of instructions installed on the patient's device. In some embodiments, the images may include 2D images (such as digital images captured using an image sensor such as a CMOS or CCD image sensor) or image data (such as digital data describing the light captured by the image sensor). The light may be visible light reflected from the surface of the patient's tissue or visible light reflected from tissue beneath the surface, such as infrared light that can penetrate the patient's tissue. In some embodiments, the light may be X-rays or other wavelengths of light, and the image data may record the light passing through the patient's tissue. In some embodiments, the images may include 3D image data, such as point clouds representing surface data of the patient's dentition, or 3D meshes representing surface data. In some embodiments, image data may include 3D volumetric data, such as CBCT data, CT scan data, or other volumetric data. Two-dimensional data may be represented by a 2D arrangement of pixels (such as a 2D array). Three-dimensional data may be represented by a 2D or 3D arrangement of voxels (such as a 3D array).

[0053] In one embodiment, a user interface or audio clip can assist the patient in positioning the camera to capture a desired set of images, such as by providing guidance on the proper positioning of the camera for the purpose of obtaining the desired images or video. For example, if the camera is too close, the guidance may include instructions for moving the camera further away from the patient's dental arch. If the camera is tilted, the guidance may include instructions for rotating the camera. If the camera is moving too fast or too slow, the guidance may include instructions for slowing down, keeping the camera stationary, or moving the camera along the dental arch of the patient's teeth.

[0054] In one embodiment, the user interface or audio clips may assist the patient in moving the camera or various aspects of the patient's oral cavity, such as the patient's jaw, lips, etc., to capture a desired set of images (e.g., photos, videos) of the dynamic patient movement during the movement (e.g., by providing guidance on appropriate camera or anatomical positioning, such as how or where to move the camera, jaw, lips, etc., for the purpose of obtaining the desired images or videos).

[0055] In one embodiment, as a non-limiting example, an application installed on a mobile device can control the operation of a camera to assist in capturing one or more images (e.g., photos, videos) under specific camera settings (such as by setting a focus distance, focal length, or lighting, such as by automatically turning on lights).

[0056] In one embodiment, the position and orientation of the mobile device (such as relative to the patient's dentition) or other characteristics related to the image or video can be determined and provided to a dentist or orthodontist along with the image or video (such as in the form of metadata and / or sensor data). In one embodiment, sensor data (such as data from a light sensor or gyroscope in the mobile device) can be used to provide guidance to the patient when positioning the mobile device or changing the environmental conditions for collecting images, such as providing instruction as discussed herein. In one embodiment, sensor data can be used during image processing to correct or modify them to remove artifacts or other undesirable aspects.

[0057] At box 104, images or videos can be provided to a dentist or orthodontist for evaluation. For example, a mobile device can send images or videos (and related information) to a remote server that can store the images or videos and enable the dentist or orthodontist to subsequently access them. As another example, a mobile device can send images directly to the dentist or orthodontist's system.

[0058] In some embodiments, if a patient captures video, that video can be processed later to generate one or more images. For example, at box 105, the video can be processed to extract images. As a non-limiting example, the processing may also include sampling or filtering.

[0059] At box 106, images (e.g., photographs, videos) may be evaluated or otherwise filtered to obtain a desired set of images for a reconstruction process, such as reconstruction of a novel view of the patient's dentition. As further disclosed and / or described herein, this processing stage may include: evaluating (and in some cases, correcting) image quality. Evaluating or correcting image quality may include: detecting poor or inadequate image quality (e.g., by determining the degree of blurring in the image or whether the initial image is uncorrectable), performing a comparison with previously obtained images to assist in identifying changes in occlusion, masticatory movements, temporomandibular joint connections, tooth position, or alignment (or misalignment) between images, and / or assisting in evaluating the quality of images provided by the patient.

[0060] In one embodiment, evaluating or correcting image quality can be implemented as a form of supervised learning, wherein a trained model generates an indication of the relative quality or utility of an image (e.g., a photograph, video) (as a non-limiting example). In one embodiment, the dataset used to train the image quality evaluation model can be obtained by comparing labels or annotations assigned to image data (such as images or videos) by multiple processes or techniques to identify labeling errors and using that information as part of the model's training data.

[0061] At box 108, filtered or otherwise evaluated images (e.g., photographs, videos) are used as input to one or more reconstruction processes to generate a baseline for one or more novel static or dynamic views of the patient’s dentition.

[0062] Examples of image processing techniques that can be used to generate reconstructed representations (or fusions) include, but are not limited to, differentiable volume rendering techniques such as neural radiation fields and Gaussian splashing, image stitching, motion reconstruction structures, and finely tuned personalized diffusion models. The use of such image processing techniques is described in more detail below.

[0063] At box 110, one or more novel static or dynamic views are generated based on a reconstructed representation or a baseline representation of the patient's teeth. The reconstructed representation is then used to generate one or more images or videos representing the patient's oral cavity as viewed from a desired perspective, location, environmental conditions, movement, facial or jaw position, or other aspects of interest to the dentist or orthodontist. This allows the dentist or orthodontist to view the patient's oral cavity from other angles and from locations different from those in the images or videos obtained from the patient. This also provides one or more baseline images (e.g., photographs, videos) for the dentist or orthodontist to use when comparing images obtained at different times.

[0064] At box 112, one or more generated static or dynamic views are evaluated to determine the desired treatment plan. This step can be used to prepare an initial treatment plan. The treatment plan may include one or more treatment phases. A treatment phase may be an incremental repositioning phase of orthodontic treatment used to move one or more teeth in the patient's teeth from an initial tooth arrangement toward a target arrangement. For example, a treatment phase can be generated by determining (e.g., based on one or more generated views) an initial tooth arrangement indicated by a digital representation, determining a target tooth arrangement, and determining a movement path for one or more teeth in the initial arrangement to achieve the target tooth arrangement. The movement path may be optimized based on minimizing the total distance of movement, preventing collisions between teeth, avoiding more difficult tooth movements, or other suitable criteria. A treatment phase may be an incremental jaw repositioning phase of orthodontic treatment used to move the patient's jaw from an initial arrangement toward a target arrangement, such as by mandibular advancement or retraction. The treatment plan may also include restorative treatments such as veneers, crowns, bridges, or other restorative treatments.

[0065] At box 113, one or more static or dynamic views generated are evaluated to determine the current state of the patient's teeth and progress compared to the treatment plan. This process step can be used to monitor the progress of the treatment plan. Progress tracking can be performed at any point during treatment. In some embodiments, progress tracking may correspond to the patient completing a pre-planned treatment phase, such as fitting a crown or other prosthesis, completing an orthodontic treatment phase that may include wearing orthodontic appliances for a period of time. For example, once the treatment plan is determined, a first set of one or more appliances is applied to the patient in a first treatment phase, which may include one or more phases. Progress tracking may be performed after the last appliance in the first set is applied to the patient. In some embodiments, progress tracking may be performed after indicating that the patient's treatment progress has not followed the expected or planned progress tracking, such as if the appliance used for the treatment phase does not fit the patient's dental arch. For example, the appliance may remain too tight on the dental arch, making it difficult for the patient to remove the appliance, or the appliance may remain too loose on the dental arch, causing part of the appliance to detach from the dental arch.

[0066] At box 116, a machine learning model can be trained to act as a classifier using multiple generated images (e.g., photos, videos). This can include using multiple generated images to train the model as a classifier for diagnosis, treatment planning, or determining the state or changes in state of a patient's oral cavity during training. In one embodiment, the images used as part of the training dataset can be obtained from one or more sources. This can include using multiple generated images to train the model as a classifier for diagnosis, treatment planning, or determining the state or changes in state of a patient's oral cavity during training. In one embodiment, the images used as part of the training dataset can be obtained from one or more of the following sources: reconstructed representations disclosed and / or described herein, existing sets of images and annotations that have been evaluated to determine their annotation accuracy when operating under multiple conditions or input parameters, and / or sets of images generated by a successive blurring process—in which case it is beneficial to determine changes in the degree of blur between images after successive application of blur filters—if the change indicates that the original image is sufficiently sharp, then one or more blurred images from the original and blurred images can be used as part of the training dataset for the machine learning model.

[0067] At box 117, trained models can be used to aid in dental visualization, diagnosis, monitoring, and other dental functions. For example, as discussed herein, models can be used to create visual representations of a patient's dental anatomy, such as the current state of the patient's teeth from one or more dynamic or static views. In some embodiments, images can be used to aid diagnostic applications. Dental diagnosis includes identifying and assisting in the treatment of various oral health problems, including diagnosing health issues of the teeth, gingiva, and other structures in the mouth. Dental monitoring may include tracking the progress and status of a patient's dental health and orthodontic treatment over time. For example, trained models can be used to generate multiple static or dynamic visualizations of a patient's dentition over time from the same perspective (including field of view and focal length), such as based on multiple imaging sessions taken at intervals over a period of time. Dental professionals can monitor changes in the patient's gingiva over time from this same perspective and can make diagnoses based on, for example, changes in gingival recession. Dental professionals can monitor changes in a patient's occlusion, masticatory movements, or temporomandibular joint connections over time from the same novel viewpoint and can make diagnoses based on, for example, different jaw movements and / or initial occlusal contacts.

[0068] As described, one or more reconstruction processes can be used to “fuse” multiple images (e.g., photographs, videos) and generate a baseline for synthesizing one or more novel static or dynamic views of a patient’s oral cavity, including jaw movement, teeth, and gums. Several such processes are available and can be utilized. Examples include differentiable volumetric rendering techniques such as neural radiation fields and Gaussian splashing, intelligent image stitching, and finely tuned personalized diffusion modeling.

[0069] In some embodiments, images or videos of a patient's dentition can be used to generate a neural radiation field of the patient's dentition. A neural radiation field is a method for assembling realistic 3D scenes from 2D images (e.g., photographs, videos). A neural radiation field can represent a scene as a continuous function, such as a 5-dimensional function (x, y, z, coordinates defining the location in the scene, and two angles defining the viewing direction) that maps 3D coordinates and 2D viewing orientation to color and density at that point. This approach allows for photorealistic renderings of static or dynamic subjects from novel viewpoints (viewpoints not captured in the 2D images used to generate the neural radiation field).

[0070] In some embodiments, the neural radiation field uses a neural network to represent a scene, such as a dentition, which includes the patient's teeth and gums. The neural network is trained on images and / or videos of the patient's dentition, such as images and / or videos acquired at box 102. During training, the network learns to predict the color and density of points in the scene such that when these points are rendered from the same viewpoint as the training images or videos, they closely match the original images. This process may include using gradient descent to optimize the network's parameters. In some embodiments, additional images, videos, or other data can be used to train the neural network. For example, position and orientation data captured when capturing images or videos can also be used to train the network. In some embodiments, scan data or 3D models of the patient's teeth or general teeth can be used to train the model. Such dentition-specific data and / or position and orientation data can contribute to a more accurate neural network and accelerate the generation of the neural network.

[0071] After training a neural network, it can be used to generate images (e.g., photos, videos) from novel viewpoints. Volumetric rendering can be used to generate these images. When using volumetric rendering, light rays are projected into the scene from a novel viewpoint. Color samples are taken at points along each ray (e.g., in each voxel of a 3D scene). Using a neural network, each pixel is generated based on the sampled rays (e.g., by assigning color properties to pixels) for each point or pixel in a generated 2D image or video frame.

[0072] As discussed in this paper, neural radiation fields can be used to effectively fill gaps in the raw image data captured during progress tracking image acquisition at box 102. As discussed in this paper, patients or even dentists may not be able to capture progress images (e.g., photos, videos) from the same camera position and orientation or the same subject movement during each stage of progress tracking. Neural radiation fields can be used to allow the generation of consistent images from the same viewpoint over time, even if the originally acquired images or videos were not captured from the same viewpoint. This capability is particularly valuable for intraoral imaging because it can correct for patient capture errors and generate images seamlessly aligned with a consistent viewpoint, while also correcting for patient image capture errors.

[0073] Static neural radiation fields can generate novel views of static scenes captured from multiple viewpoints. These fields are particularly useful for reconstructing scenes that remain unchanged throughout the imaging cycle. By leveraging the continuity and consistency of static scenes, static NeRF achieves a high level of detail and accuracy in the resulting images. In some embodiments, dynamic neural radiation fields can be used to extend the capabilities of static NeRF by incorporating time as an additional input variable, thereby enabling the rendering of time-varying 3D scenes with deformations (rigid or even non-rigid) from sparse monocular or multiview input views. These dynamic neural radiation fields can be used to generate novel views from dynamic scenes, including video frames in which the subject, scene, or elements within the scene move during capture. Dynamic neural radiation fields can generate video (or still images) of subject movement based on the novel views. By incorporating time as an additional input variable, dynamic neural radiation fields are extended to dynamic scenes. This enables the rendering of time-varying 3D scenes with non-rigid deformations from sparse monocular or multiview input views.

[0074] Dynamic Neural Radiation Fields (NeRF) decomposes the learning process into two neural networks: a canonical network and a deformable network. The canonical network is trained to learn the appearance and geometry of the scene in a canonical (e.g., reference) configuration, while the deformable network is trained to model the transformations between the dynamic scene and the canonical space at different times. This approach allows Dynamic NeRF to generate temporally coherent renderings of novel views, adapting to changes in the scene over time. The Dynamic Neural Radiation Fields model is trained to learn how each point in the scene moves over time. Given one or more desired novel views, the Dynamic Neural Radiation Fields model uses the canonical and deformable networks to generate temporally coherent renderings of the novel views.

[0075] For monocular or multi-view applications, deformable field learning can use a deformable network to map each point at a specific time to its corresponding canonical space representation. This deformable network is then used to predict one or more displacement vectors for that point. The canonical network models the radiation and density of a dynamic scene in a fixed (e.g., canonical) space to produce the color and volume density of points within the canonical space.

[0076] During the rendering process of a novel view at a specific time, the deformable network maps each point into a canonical space. The canonical network then generates color and density values ​​for the mapped points, and finally, volumetric rendering is used to synthesize one or more final images of the video.

[0077] In some embodiments, depth information can be generated from the input video and used in conjunction with the differentiable volume rendering described herein. The depth information can be generated from a monocular video based on a deformation field trained to map points from different video frames into a canonical space. During training, an initial depth estimate is made for each frame in the video. Depth scales are then applied to correct for depth scale discrepancies by segmenting static and dynamic regions in the video, reprojecting cross-frame depth values ​​to a common reference frame, and applying a scale factor relevant to the cross-frame depth values ​​for the arrival position in each frame. The deformation field is then generated based on the depth scale across multiple frames.

[0078] Dynamic neural radiation fields can aid dental treatment in several ways. For example, they may help reconstruct and analyze 3D jaw movement based on captured video data, allowing clinicians to generate novel viewpoints and simulate orthodontic treatments that include aesthetic modifications. This capability can be used for remote dental evaluations, treatment planning, and patient education, where capturing a clinically preferred perspective from patients who record their own jaw movement videos can be challenging.

[0079] One challenge in dental assessment is the inconsistency of camera angles when patients record jaw movement videos using personal devices. Even recordings by dental professionals may not be consistent with or capture from the clinically preferred viewpoint. These recordings may vary in distance, angle, and stability, resulting in a suboptimal view for clinical analysis. Dynamic NeRF addresses this issue by reconstructing videos of the patient's jaw captured at different angles, distances, and other parameters from a consistent and novel viewpoint. This allows the system to render novel videos from the clinically preferred viewpoint, such as frontal occlusion or left or right-side views, providing dental professionals with the optimal perspective for assessing jaw alignment, occlusion, temporomandibular joint function, and other dental evaluations without requiring patients to re-record videos, record videos under clinical supervision, or make in-person visits.

[0080] In addition to capturing individual instances of jaw movement, clinicians can monitor a patient's jaw function over weeks or months to track treatment progress. However, videos captured at different times can be inconsistent in terms of camera positioning, lighting, and distance, making direct comparisons difficult. Dynamic NeRF overcomes these limitations by normalizing each record to a standardized clinical perspective through novel view generation, allowing for consistent visualization of jaw movements over time. This enables dental professionals to accurately compare changes before, during, and after treatment, detect deviations in mandibular movement, and assess progress in orthodontic, prosthetic, or occlusal treatments.

[0081] In some embodiments, Dynamic NeRF can be used to evaluate modifications to a patient's dentition. Dynamic NeRF allows clinicians to generate synthetic images of modified tooth arrangements (such as those achieved through orthodontic adjustments, veneers, or restorations), and then use these images to render realistic motion videos. Dental professionals can use the generation of novel views with modified teeth or dentition to evaluate functional adaptations, showing how different dental treatments may affect a patient's occlusion and jaw movement over time. Patients can use them to better visualize the potential outcomes of treatment, helping them make informed decisions about their dental care.

[0082] For at least some clinical evaluations or applications, evaluation of real-world images (e.g., photographs, videos) (as opposed to synthesized images) can be used. Referring to Figure 1(f), method 600 is illustrated. Method 600 may begin at box 610, where multiple images 612 of a patient's dentition are captured from one or more of the following: multiple viewpoints, focal lengths, depths of focus, etc. At box 620, a NeRF model is optimized and / or trained based on input image data (e.g., photographs, videos) and / or other data. At box 630, a dental professional can review a novel view 632 of a static or dynamic subject synthesized from a neural radiation field to monitor the patient's dentition. In some embodiments, at box 640, one or more original images or videos 612 captured from one or more perspectives closest to the novel view may be presented to the dental professional for review (e.g., if a position on the dentition or jaw movement, etc., is of interest or otherwise requires further review). For example, the system disclosed herein can present synthesized novel views of static or dynamic subjects and can also identify a set of original images depicting the novel view from a perspective within a threshold degree of similarity (e.g., in terms of angles and coordinates of features within the scene). Additionally, the uncertainty associated with the novel view can be provided to dental professionals to allow them to determine whether they should review the original captured data at a specific location on the patient's dentition.

[0083] Figure 1(d) depicts a user interface 160 of a novel view 162 of a static or dynamic subject, such as shown on a display, which is based on a reconstruction representation, such as a neural radiation field of the original image 164 captured from the closest position and orientation as depicted in the novel view 162 (or other reconstruction representation methods described herein, such as motion recovery structures, Gaussian splashes, etc.).

[0084] In some embodiments, a neural radiation field can be used to generate a 3D model of a patient's dentition. For example, multiple novel views can be generated from the neural radiation field, and these novel views can be used to generate a 3D model using photogrammetry, structure of motion restoration (SfM), or other methods.

[0085] In some embodiments, Gaussian splashing can be used to generate novel static or dynamic views of a patient's dentition. Gaussian splashing may produce results similar to those of NeRF, but there are some differences and advantages. Unlike NeRF, Gaussian splashing does not require neural network components, and rendering using Gaussian splashing may be faster. A Gaussian splash is a set of points in the space of a scene, represented as a 3D Gaussian splash with position, rotation, and scale (each with mean and standard deviation). Each point can have a "color" represented by third-order spherical harmonics, allowing the color to change based on the view orientation. During rendering to a novel 2D image or video (such as a frame of a video), the points are represented in image space as a 2D Gaussian splash, such as a 2D array of pixels. This method handles specular highlights and reflections that are common in intraoral images due to saliva reflection well and may be particularly well-suited for this use case.

[0086] Dynamic Gaussian splashing can be used to render novel static or dynamic views of dynamic scenes. In a dynamic Gaussian splashing model, each point in space within the scene retains the same scale, color, and opacity over time, similar to non-dynamic Gaussian splashing. However, in dynamic Gaussian splashing, each Gaussian also has a position and rotation in 3D space, which are tracked and move over time (such as within the time frame of the original captured video). Additionally, each Gaussian may have a background log-odds ratio (logit). The background log-odds ratio is a value that can be used to help the model distinguish between Gaussians that are part of a dynamic object or subject and those that are part of a static background in the captured video. A high background log-odds ratio indicates that the Gaussian is part of a static background of the scene, while a low or negative value indicates that the Gaussian is part of a dynamic part of the scene. In some embodiments, the value of the background log-odds ratio is the probability that the Gaussian is part of the background. This means that while the visual characteristics of each point remain constant, their position and orientation are updated frame-by-frame to reflect movement within the scene. In essence, dynamic Gaussian splashing can create a continuous and realistic representation of a moving subject or other element in a video by combining static properties with dynamic position data.

[0087] When generating a dynamic Gaussian splash model, Gaussian parameters are initialized from a sparse point cloud at the first time point. Then, in subsequent video frames, the position and rotation parameters of the Gaussian are updated based on the changing scene, while scale, opacity, and color remain constant. By maintaining a constant scale, color, and opacity over time, dynamic Gaussian splashing enforces consistency in color, size, and opacity during novel view generation. When rendering a novel view at a specific time using the dynamic Gaussian splash model, the Gaussian splash uses the parameters from that specific time. For example, a Gaussian splash representing a novel view of the scene at time T1 can be generated based on the position and rotation parameters at time T1 in the model, and a Gaussian splash representing a novel view of the scene at time T2 can be generated based on the position and rotation parameters at time T2. Rendering one or more novel views of a dynamic scene over a period of time produces a series of images that can be used to form a video.

[0088] The systems and methods described herein describe the generation of novel static and dynamic views. While some embodiments describe the use of Gaussian splashing to generate novel views, and others describe the use of NeRF to generate novel views, it should be understood that NeRF can be used alternatively in the context of the discussion of the use of Gaussian splashing, and vice versa.

[0089] In some embodiments, generating novel views using Gaussian splashing may include: capturing a scene, such as by capturing 2D images, videos, etc. For example, a mobile device (e.g., a smartphone or other camera device) may be used to capture such images or videos. In some embodiments, 3D image data may also be generated. For example, 3D scanning, photogrammetry, or depth sensing may be used during scene capture. Key features or points may be extracted from the scene. These features may be 3D points in space and include location, color values, texture information, etc. Each of these features may be represented using Gaussian splashing of location, color values, texture information, etc. The parameters of the Gaussian (mean and distribution) may be determined based on the desired smoothness or blur level.

[0090] When generating a novel view (i.e., an image or video from an original, uncaptured viewpoint), the system calculates the contribution of each feature to each pixel in the novel view. This is done by projecting the features onto the new view plane using Gaussian splashing. Contributions from different features accumulate at each pixel. Gaussian splashing allows features to blend smoothly together, helping to avoid discontinuities. The accumulated values ​​at each pixel can then be used to generate the final image or video frame, providing a realistic and continuous representation of a static or dynamic scene from the new viewpoint.

[0091] For example, dental professionals can use novel views to assess the progress of dental treatment. Novel views can, for instance, allow for more direct comparisons between images (e.g., photographs, videos) taken at two different times (where their angles / positions may vary), as further discussed herein (e.g., with regard to Figure 1(b)). As another example, dental professionals may be able to generate desired views that may not have been captured before to better assess a region (e.g., a protruding part of a tooth), or to generate a set of views around a region to gain a three-dimensional understanding of that region.

[0092] As noted above, in clinical evaluations or applications requiring the evaluation of real-world images, dental professionals can first review novel 2D images (e.g., photographs, videos) synthesized by Gaussian sputtering to monitor a patient's dentition, and subsequently review (or request) one or more original captured images that most closely approximate the synthesized novel view. If a location on the dentition raises concern or warrants further review, one or more original images (e.g., photographs, videos) captured from the closest or near-closest set of viewpoints compared to the novel view can be presented to the dental professional for review. Additionally, the uncertainties associated with the novel view can be provided to the dental professional to allow them to determine whether the original captured data for a specific location on the patient's dentition should be viewed.

[0093] In some embodiments, Gaussian splashing can be used to generate a 3D model of a patient's dentition. For example, multiple novel views can be generated from Gaussian splashing. These views can be used to generate a 3D model using photogrammetry, kinematic reconstruction, or other methods.

[0094] In some embodiments, image stitching can be used to generate novel images (e.g., still images or videos) with a larger field of view (such as a panoramic view). Image stitching can use a combination of one or more different methods to generate novel images. Features in the original captured images (still images or videos) are matched. Methods for achieving this may include classical algorithms (such as SIFT) or machine learning-based methods (such as SuperGlue or LightGlue) to match keypoints across frames. The images are then transformed and stitched together to create a panoramic view based on the keypoints. In some embodiments, color filters are applied to each transformed image to provide consistent colors in the images. Color filters can be used to mitigate inconsistencies caused by differences in image capture conditions (such as images captured under different lighting or exposure conditions).

[0095] In some embodiments, a finely tuned, personalized diffusion model can be used to generate novel views. This approach involves fine-tuning the diffusion model using intraoral images of the patient and then generating novel perspectives from that model. This approach uses a diffusion model trained in two parts. In the first part, the diffusion model can be trained based on images of a dentition that is not the patient's dentition. For example, the images could be images of a generic dentition with common teeth, gums, and other tissues from multiple different viewpoints. In some embodiments, a 3D model of the generic dentition (such as 2D rendered views of the dentition from multiple viewpoints) can be used additionally or alternatively to train the model. Viewpoints and other aspects of the training data can be labeled to include information such as the location of teeth, gums, and other tissues within the image and / or the distance and angle of the viewpoint. In the second part (which may occur after the first part), the model can be trained using labeled images of the patient's actual dentition. After training the model using the first and second parts, highly specific images are synthesized in various scenes, including different viewpoints.

[0096] Once one or more reconstruction procedures are applied, the reconstructed representation can be used to generate one or more novel views of the patient's teeth, gums, lips, palate, etc. Parameters or descriptions of the desired view (such as a "right cheek view") can be specified by the user of the reconstruction procedure and used to cause the procedure to generate one or more images, such as a right cheek view of the patient's dentition from a specific angle, distance, and field of view. This allows dentists or orthodontists to better visualize the patient's mouth and teeth and to provide a common view or perspective for comparative purposes that may be lacking in images provided by the patient at different times. Several methods also exist to quantify the areas of uncertainty in the novel views to help viewers determine the extent to which they rely on them.

[0097] In some embodiments, the structure of objects in a scene can be determined based on the relative motion of a subject as images are taken from different locations or viewpoints (such as different angles and distances). The structure derived from the motion model can be generated from a series of 2D images or videos captured from different angles and locations around the dentition. These images include: overlapping fields of view, capturing portions of the same location of the dentition from multiple images captured from different angles and locations.

[0098] Next, keypoints in each image are identified. Keypoints in an image can identify locations within the image corresponding to unique features, such as the edges of teeth and / or gums; color; or the surfaces of teeth and / or gums, cusps, markings on adjacent regions, etc. Keypoints and their corresponding features can then be matched across the set of images to determine the location of features in one image in other images within the set.

[0099] Then, the camera pose, such as position and angle, is determined based on the relative positions of key points in the image. In some embodiments, a 3D model of the dentition (such as a generic or patient dentition) can be used to help determine the camera pose when capturing the image. For example, multiple 2D projections of the 3D model of the dentition can be generated at various virtual camera positions and orientations. The camera pose when capturing the image can be determined based on matching teeth or other features in the 2D projection with corresponding features in the captured image.

[0100] In some embodiments, captured images can be analyzed or processed to identify teeth in the images. For example, a neural network or other classifier can identify teeth in an image. Initially, the camera pose can be estimated based on the teeth or other anatomical structures identified in the image. The camera pose can then be refined based on key points in the image.

[0101] Triangulation can then be used to determine the location of features in the image, and a point cloud can be generated. In some embodiments, the generation of the point cloud can occur during (e.g., concurrently with) the camera pose determination process.

[0102] The point cloud can then undergo bundle adjustment, where the 3D coordinates representing the scene geometry and the parameters of the camera that captured the image (e.g., position, orientation, and intrinsic parameters such as focal length and lens distortion) are refined. Bundle adjustment can include adjusting the 3D points of the camera pose and the camera parameters to minimize reprojection error, which is the difference between the observed image points and the projected points derived from the 3D model and camera parameters. Bundle adjustment can include least-squares optimization.

[0103] In some embodiments, motion reconstruction models and point clouds generated using motion reconstruction processes can be used to generate 2D images from novel viewpoints not found in the original image.

[0104] In some embodiments, the 2D images generated using the motion-restoring structural process may include noise or other defects, such as cavities or missing information. A machine learning model, such as a generative adversarial network or a diffusion model, can be trained on patient dentistry data, which may include captured raw videos or images and / or 3D scan data of the patient or general dentistry data. The machine learning model can then be used to correct defects, such as by filling cavities or other missing information.

[0105] Figure 1(b) is a flowchart or diagram illustrating a method that can be performed in an implementation of an embodiment of the disclosed system and method for evaluating the progress of a treatment regimen. This flowchart presents one example use of the techniques or methods disclosed and / or described herein, and other example uses are possible and can be used to develop and / or monitor the progress of a patient's treatment regimen.

[0106] As shown in Figure 1(b), in one embodiment, a process or method for monitoring the progress of a patient's treatment regimen may include one or more of the following steps, stages, operations, or functions. In some embodiments, at box 120, multiple images (e.g., photographs, videos) of the patient's dentition (including teeth and gums) may be received. These images may have been captured at a first time T1 (e.g., during the first phase of dental treatment) using any suitable device (e.g., a mobile device, a portable scanner capable of capturing images). In some embodiments, image data of a static or dynamic subject or scene, as well as one or more other data related to the capture, such as camera position, orientation, and other settings, may be received. In some embodiments, the images or videos may be captured by the patient using their mobile device (e.g., as part of a virtual monitoring program), and the set of images may be associated with data or metadata describing camera settings, camera positioning and orientation, and, in some cases, environmental data. The images and associated data or metadata may be provided to a remote platform (in one embodiment) for further processing and evaluation.

[0107] In some embodiments, images (e.g., photographs, videos) captured at different times (such as at time T1 and at time T2 after T1) can be used. While some anatomical features may move between time T1 and T2, non-moving anatomical features captured in images at T1 and T2 can be used to align images taken at different times. In some embodiments, features not expected to move can be used to align images. For example, in orthodontic treatment programs, some teeth (such as distal molars) may remain stationary and act as anchorage teeth during some treatment phases, while anterior teeth (such as incisors) are repositioned. Such anchorage teeth or other teeth that are not yet planned to move during or between T1 and T2 can be used to align images captured at T1 and T2. In some embodiments, non-moving features and their relative positions within the image can be used to derive the position and orientation of the camera or other imaging device at the time the image was captured.

[0108] Non-mobile features include features that have not been planned for movement during the treatment plan, and may also include untreated teeth (such as during restorative surgery). When capturing dynamic subjects in a scene (such as capturing occlusion, chewing, temporomandibular joint connection, changes in facial expression, etc.), the subjects may be moving, but the non-mobile features discussed in this article can be used to align images or videos.

[0109] At box 122, the received image (e.g., a photograph, video) can be filtered or sampled to create a set of images of sufficient quality (e.g., input to a trained model, evaluate blur, combine multiple images, or focus stacking). One or more techniques can be used to evaluate the image quality of the received image. For example, captured images can be input to a trained model to evaluate image quality. The trained model can output a quality metric for each image, an indication of which images have acceptable quality, or an image with acceptable quality. In some embodiments, the quality of portions of an image can be evaluated using a trained model. The trained model can indicate which portions of the image have acceptable or unacceptable quality, assign a quality metric to each portion, or otherwise indicate which locations in the image are acceptable and / or unacceptable. Similarly, blur in an image or portions thereof can be evaluated. For example, captured images can be input to a trained model to evaluate image blur. The trained model can output a blur or sharpness metric for each image, an indication of which images have acceptable blur or sharpness, or an image with acceptable blur. In some embodiments, the blurring of a portion of an image can be evaluated using a trained model. The trained model can indicate which portions of the image have acceptable or unacceptable blurring, can assign a blur metric to each portion, or can otherwise indicate which locations in the image are acceptable and / or unacceptable.

[0110] In some embodiments, images or portions thereof (such as portions with acceptable quality and / or blurriness) may be combined, for example by focus stacking, or the view of the image may be expanded to include more of the patient's anatomy to produce an image or set of images with features suitable for use. Images may be processed to enhance features, contrast, remove glare, or alter other aspects of the image based on associated data or metadata.

[0111] At box 124, a reconstructed representation of the patient's dentition can be created. As described elsewhere in this document, a reconstructed representation of the patient's mouth can be generated using one of several techniques or approaches, including, for example, differentiable volume rendering approaches such as NeRF and Gaussian sputtering.

[0112] At box 126, images (e.g., photographs, videos) of the patient's teeth can be captured at time T2, where T2 follows T1, and images (e.g., photographs, videos) of the patient's teeth can be received with or without indicating camera position, orientation, and settings. This may include capturing images of the patient's mouth using a set of associated camera and / or environmental data or metadata. In some embodiments, camera position and orientation may be determined based on image data, rather than provided with the images.

[0113] The previously constructed representation of the patient's mouth can then be used to generate appropriate images (e.g., photos, videos) for comparison with the most recently provided images. For example, at box 128, a reconstructed representation of the patient's teeth can be accessed. As another example, a dynamic reconstructed video representation of the patient's open and closed jaws can be accessed. The previously generated representations can be stored locally on mobile devices and / or remote platforms.

[0114] At box 130, an image (e.g., photograph, video) of the patient's teeth can be generated using a reconstructed representation based on one or more of the camera position, orientation, and settings of the image captured at time T2. The reconstructed representation is used to generate novel images (e.g., photographs, videos) where the camera position, camera settings, and / or environmental conditions are similar to those associated with the image taken at time T2 using one or more of the methods discussed herein.

[0115] At box 132, compare an image of the patient's teeth generated using the reconstructed representation (e.g., a photograph, video) with an image taken at time T2. Compare the image generated using the reconstructed representation with an image taken at time T2 using a human or a trained model to identify differences or any regions of interest in the images. As suggested, this can be performed manually and / or with the aid of a trained model that identifies differences between the two images.

[0116] Other types of views can be generated and comparisons made. For example, in some embodiments, the original image (e.g., a photograph, video) captured at time T1 and the synthesized image (e.g., a photograph, video) generated based on the image captured at time T2 (e.g., a photograph, video) can be used to identify changes in a patient's anatomy for dental diagnosis, tracking, or treatment, wherein the T2 image is synthesized to correspond to the same location and orientation as the original T1 image. In some embodiments, the synthesized image generated based on the image captured at time T1 and the original image captured at time T2 can be used to identify changes in a patient's anatomy for dental diagnosis, tracking, or treatment, wherein the T1 image is synthesized to correspond to the same location and orientation as the original T2 image.

[0117] In some embodiments, a first composite image (e.g., a photograph or video) can be generated based on an image captured at time T1, and a second composite image can be generated based on an image captured at time T2. The camera position, orientation, field of view, etc., of both the first and second images can be identical. The first and second composite images can be used to identify changes in a patient's anatomy for dental diagnosis, monitoring, or treatment.

[0118] In some embodiments, the synthesized image may not be generated based on a virtual camera view with focal length, field of view, depth of focus, or other camera characteristics. For example, in some embodiments, a synthesized view or novel view can be generated as a panoramic image that unrolls the dental arch and provides visibility of all teeth in a single image. Such panoramic or unwrapped images may be helpful for comparison, evaluation, diagnosis, and tracking.

[0119] At box 134, based on comparison, the treatment plan may be modified or other treatments may be determined to be necessary or optimal. Modifications or further treatment may then be recommended to dental professionals and / or patients (e.g., via notifications on the interface of a software application). For example, if teeth or jaws do not move as expected, route correction (e.g., using a new set of appliances) may be recommended. Intervention may be recommended if gingival defects or other dental or oral health problems are identified. In some embodiments, temporomandibular joint connection, masticatory movements, or occlusal function may not be as expected after restorative work is performed. Based on comparison, the dentist or orthodontist may modify or maintain the existing plan as appropriate.

[0120] In some embodiments, the position of the teeth in the captured images and / or reconstructed representations can be compared with the expected position of the teeth in a treatment plan. If the positions do not match sufficiently, orthodontic treatment may go astray. A new orthodontic treatment plan can be generated to move the teeth from their current position to the desired final position. Additionally, new orthodontic appliances, such as braces, can be fabricated. The fabricated braces can be worn by the patient.

[0121] As described, during an image or video capture process performed using a patient's mobile device, an application installed on the device can control various aspects of the device's camera operation. Doing so can improve the quality or utility of the captured images and compensate for limitations of the device's camera. For example, optical limitations such as a shorter focusing distance can result in a shallow depth of field, where only certain portions of the image are in focus at a given time. Furthermore, the camera's flash is typically positioned near the subject's teeth, which may mean that only certain portions of the image or video are properly exposed. Finally, to fully capture the dental arch of the mouth, the patient may need to take multiple images (e.g., photographs, videos), increasing the time and care required to obtain images.

[0122] To compensate for or correct for such limitations, in some embodiments, multiple images can be combined into a single synthetic image. In one such embodiment, an application on a mobile device can control the camera to capture a desired set of images. In one example use case, the mobile device captures multiple photos. One or more sets of images are either processed on the mobile device or uploaded to a server for processing. Once processed, one or more synthetic images can be generated, which can be analyzed by a dentist and optionally evaluated via AI / ML-based mechanisms, or used as input to generate one or more reconstruction models among those discussed herein, such as differentiable volumetric rendering models including neural radiation fields and Gaussian splashes, motion recovery structures, diffusion, etc.

[0123] Referring to Figure 1(c), pathways for generating synthetic images from multiple captured images are described, and these pathways may include focus stacking, high dynamic range (HDR) images with tone mapping, and / or generating panoramic images of dental arches. While references are made to photographs or images, this disclosure can be applied to videos, such as, for example, video frames. For example, videos of a moving subject can be captured simultaneously from different viewpoints to allow for stitching together of a video panorama at sufficiently high frame rates and with different exposure or focus values, thereby enabling focus stacking or HDR video generation.

[0124] Focus stacking refers to the process used to compensate for or correct the depth-of-field (DoF) limitations of many mobile device cameras. Depth of field (DoF) refers to the range of distances within acceptable focus when the camera is focused at a specific distance. An approximation of the camera's true depth of field can be expressed as: , in u It is the distance between the subject and the person being tested. N It is the f-number or "aperture value" that indicates the focal ratio of the lens when taking an image. f It's the focal length. c It's the circle of confusion. Due to the inherent characteristics of mobile device cameras, the circle of confusion (which is proportional to the size of the camera sensor) is typically very small. With reasonable assumptions about the relevant parameters, when focusing on an object more than 4 inches away (a rough distance that can capture an image of the oral cavity), the depth of field can be as small as 0.5 inches, meaning that objects between 3.75 and 4.25 inches from the camera lens might be in "reasonable" focus. When the camera is closer to the teeth, the depth of field can be even smaller, such as less than 1 / 4 or 1 / 8 inch.

[0125] To address this limitation, in one embodiment, the camera of the device used to capture dental images (e.g., a patient's mobile device) can be configured to collect multiple images at different focal lengths. The images can then be aligned using point-matching methods to find suitable transformations (such as affine, homography, or linear motion, for example) that correlate the images. Once the images are aligned, they can be combined using a focus-stacking algorithm, such as a complex wavelet-based approach. Other approaches can also be included as part of this approach, such as identifying which image has the largest variance in the observed edges. By focusing and stacking multiple images taken from the same or similar viewpoints, a single composite image can be generated from that viewpoint, where the depth of field is greater than any single image used to generate the composite image. This process can be repeated for images taken from multiple viewpoints. The composite image from multiple viewpoints can then be used as input to one of the reconstruction methods discussed herein.

[0126] In some embodiments, high dynamic range (HDR) images with tone mapping can be used to address some of the camera limitations discussed above. In these embodiments, the camera device can capture multiple images at different photographic exposures while capturing a view, and a composite image can be created from these images. In some embodiments, in addition to collecting images with different depths of focus, or instead of collecting images with different depths of focus, the device is operated (typically via an installed application or instruction set) to collect images at different exposure sensitivities. Because the camera is typically positioned closer to some teeth in an image compared to other teeth in the image, the camera's automatic exposure algorithm can be set so that the camera is used to properly expose only the nearest teeth (e.g., teeth within a threshold radius or range), while the rest of the image is often underexposed. By making the exposure sensitivity and / or exposure parameters, such as shutter speed and aperture, different, this process can capture multiple images, each with a different portion of the dental cavity and a desired exposure.

[0127] After aligning the images, a single composite image can be created using most (if not all) of the caries cavities with appropriate (or at least suitable) exposure. This improves the applicability of multiple images for dental evaluation via AI / ML algorithms or for direct visual evaluation by trained clinicians. This approach optimizes tone mapping so that each part of the image is properly exposed. This process can be repeated for images taken from multiple viewpoints. The composite image from multiple viewpoints can then be used as input to one of the reconstruction methods discussed in this paper.

[0128] In some embodiments, the methods and systems disclosed herein can be configured to capture a series of photographs taken along a patient's dental arch to capture images of the teeth along the arch. Once the photographs are captured, a single panoramic image can be constructed to provide the dentist with a single image (e.g., a single sequential image) showing, for example, the entire dental arch (e.g., the buccal sides of all patient teeth, the lingual sides of all patient teeth, and / or the occlusal sides of all patient teeth).

[0129] In some embodiments, the combined image can be a focus-stacked and high dynamic range tone-mapped image, wherein multiple images with different focuses and exposures from the same or similar viewpoints can be combined using the process described herein. This process can be repeated for images taken from multiple viewpoints. The combined image from the multiple viewpoints can then be used as input to one of the reconstruction methods discussed herein.

[0130] In some embodiments, posterior teeth that may not be clearly visible are captured at a quality sufficient to generate reconstructions of some or all of the posterior teeth. In some embodiments, the reconstruction may be based on the anterior teeth. To generate a complete set of teeth for reconstruction, a 3D model of the patient's teeth (such as a model from a 3D scan of the patient's teeth) may be aligned with the reconstructed anterior teeth. The position and orientation of the posterior teeth in the aligned 3D model can be used for the posterior teeth in the reconstruction.

[0131] Figure 1(c) is a flowchart or diagram illustrating a process, method, operation, or function that can be performed for image capture or image quality assessment of an image (e.g., still or video) in an implementation of embodiments of the disclosed systems and methods. This flowchart presents one example use of the techniques or approaches disclosed and / or described herein, and other example uses are possible and can be used to develop datasets for training models or evaluating the status of treatment protocols by creating reconstructed representations of a patient's mouth and teeth. Although images are referenced, this disclosure can be used with still or video images of static or dynamic subjects.

[0132] As shown in the figure, image capture may include one or more of the following steps, stages, operations, or functions. At box 140, an application for controlling image capture in a mobile device can be installed. This application can be downloaded from a remote platform and installed by the patient on their mobile device. In one embodiment, the application may generate a user interface and / or audio to assist the patient in capturing the desired image.

[0133] When instructed, the patient can use the camera to capture one or more images, such as still images or videos. In one embodiment, an installed application can control the mobile device camera as follows: At box 142, the application is used to control the camera to capture multiple images of the patient's dentition at different focal lengths (depths of field) from one or more viewpoints along the dental arch. At box 142, the application is used to control the camera to capture multiple images of the patient's dentition at different exposures from one or more viewpoints along the dental arch. In box 146, the application is used to control the camera to capture multiple images of the patient's dentition along the dental arch (panoramic view).

[0134] At box 148, one or more captured images are provided to a remote platform for processing and / or evaluation. Images can be processed to generate composite images, such as focus-specified images and / or high dynamic range images and / or panoramic images. The captured images and / or composite images can be used to create a reconstructed representation of the patient's dentition. Given sufficient images of sufficient quality, a reconstructed representation of the patient's mouth and teeth, as disclosed and / or described, is created.

[0135] As disclosed and / or described herein, various embodiments may perform image quality assessments to determine whether images provided by a patient are suitable for use as part of a dataset for training models and / or evaluating treatment protocols. As shown, image quality assessment may include one or more of the following steps, stages, operations, or functions.

[0136] At box 150, initiate quality assessment processing for the captured images. At box 152, capture images of the patient's dentition as still images, video images, or a combination of still and video images.

[0137] One or more quality assessment processes or operations can be performed on each item in a captured photo or a combination of captured photos or videos. For example, at box 154, an image can be input into a trained quality assessment model. The model output can be a quality metric or evaluation.

[0138] In another example, the image can be blurred gradually and iteratively, such as at box 156. For each blurring iteration, a comparison can be made between the blurriness of the image in the current iteration and the blurriness of the image in a previous or original iteration. The greater the reduction in relative blurriness between iterations (such as the first iteration and the original image), the sharper the image. As discussed in this paper, a blur metric can be determined based on the relative blurriness of the image after one or more iterations.

[0139] The successive blurring technique described herein is used. In this process, the original image is successively blurred or filtered (as suggested in Figure 1(c)). Each iteration of the blurred image is evaluated to determine a characteristic metric. It has been found that if the original image is relatively more blurred and therefore has lower image quality, the value of the metric changes less between successive iterations.

[0140] Based on this evaluation, the original, sharper image can be used with greater confidence as part of the reconstructed representation and / or as part of the training dataset.

[0141] Sufficient quality images can also be used as a starting point to generate multiple related images by inputting constantly changing parameters or instruction machines into an image generator, such as an image generator based on generative techniques and / or reconstruction approaches disclosed and / or described herein.

[0142] Non-limiting examples of suitable blur filters and blur metrics can include processing images with a Gaussian blur filter, such as: •

[0143] Then, a Laplacian filter may be applied to the blurred filter, such as: •

[0144] This produces a Laplacian of Gaussian filter, such as: • ,

[0145] In these equations, I Indicates pixel intensity. x and y These are the columns and rows of each pixel in the image. The Laplacian of Gauss is a representation of the blurriness of a given pixel within a specific blur iteration. Examples of blur measures for a pixel can include the terminal Laplacian of Gauss, the value of the Laplacian of Gauss after a certain number of iterations (such as 10, 20, or 100 iterations), the slope of the Laplacian of Gauss curve at the first one or a few iterations, or the area under the Laplacian of Gauss curve after a certain number of iterations (such as 10, 20, or 100 iterations).

[0146] Blur metrics can be evaluated for a single pixel or multiple pixels to determine the relative blur of the single pixel or multiple pixels, or the relative blur of a single pixel compared to another single pixel. In some embodiments, the blur metric can compare one region (a group of pixels, such as a group of adjacent pixels) of an image with another region of the image, such as by averaging the blur metric for each pixel in each respective region. In some embodiments, the relative blur of multiple images can be compared based on the average of the blur metric for each respective image.

[0147] Figure 1(e) illustrates a process 170 for iteratively blurring an image to determine its blurriness, as discussed herein, and an associated feature curve 180, which is a curve representing a blurriness measurement as a function of iteration for an example image. At box 172, an initial image A is provided. In the column of box 174, the image is blurred; and in the column of box 176, a blur metric for each feature in the image is determined. In some embodiments, the features in the image may be anatomical features, such as teeth, adjacent regions between teeth, a portion of the image, a portion of an anatomical feature, or a single pixel or group of pixels in the image. In the column of box 178, the blur metric for each feature in the image is aggregated. Blur metrics such as aggregated blur metrics are shown in plot 180, where the horizontal axis represents iteration and the vertical axis is the blur metric, such as the aggregated blur metric from column 178. Each row in process 170 represents an iteration of the blurring process. Each time the image is further blurred, a blur metric and an aggregated blur metric for the blurred image are determined, and the resulting blur metric is shown as plotted on curve 180.

[0148] In some embodiments, the blur metric and the iterative method described above can be used to train an image quality machine learning model, such as the model used at box 154 in Figure 1(c). For example, the machine learning model can be trained using an image set and corresponding blur metrics or curves, as shown and described herein (such as with respect to Figure 1(e)). In some embodiments, a human assessment of image blur, such as a blur score on a scale of 1 to 10, may also be included. A new image can then be provided to the trained machine learning model, for example. The model's output can be a blur metric or blur score for the new image, such as on a scale of 1 to 10, or a Boolean value, such as an indication of whether the new image is sharp enough to be used or whether the new image is blurry enough to be unusable.

[0149] Additionally or conversely, supervised (e.g., logistic regression, trees, random forests, with or without PCA) or unsupervised classification (e.g., clustering) models can be used to identify blurry images compared to sharp images.

[0150] As shown in box 155, another image quality assessment method can be used. For example, protocol-based image quality assessment can be used. A set of previously obtained images may contain one or more types of features within the labeled images. Such an labeled dataset may be available from an operator providing services for the functionality disclosed and / or described herein, who has access to images captured by multiple patients (which may be used after appropriate anonymization). Such images and associated labels may be generated from one or more of the following: procedural annotation models, human annotators, or other validation sources suitable for this purpose. In some embodiments, the labels may also include confidence values ​​that indicate the annotator's confidence in the presence of the labeled features at the labeled locations in the image.

[0151] Generally, identifying photos or regions of photos with image quality issues is a subjective, labor-intensive, and often task-dependent process (because an image considered blurry for one task may be suitable for another). Supervised machine learning uses large amounts of labeled data to train models, and in many cases, images have multiple ground truth sources. As mentioned, in one embodiment, indications of consensus / disagreement among these different sources can be used to train the model to identify labeling errors and, in some cases, to estimate / predict image quality.

[0152] Image quality assessment based on consensus among multiple verification sources can provide one or more benefits. In some embodiments, image quality assessment allows image quality to be estimated as a mask across the image. As further described herein, an image can be partitioned or segmented into regions close to features already labeled by multiple verification sources. For example, bounding boxes or regions can be formed around each of one or more attachments (or some other features already labeled by multiple verification sources), and disagreement on the presence of each attachment can indicate only the quality of the bounding box or region where the attachment is located. A mask can then be applied to the corresponding image, indicating which bounding regions are likely above a threshold quality level (e.g., consensus among multiple verification sources is sufficient) and which bounding regions are likely below a threshold quality level (e.g., consensus among multiple verification sources is insufficient). In various embodiments, the overall image quality can be estimated based on a relative amount (e.g., ratio, percentage) of unmasked regions (e.g., regions above a threshold quality level) compared to masked regions (e.g., regions above a threshold quality level) in the image. In other embodiments, unmasked regions can be relied upon (e.g., for training machine learning models, for determining image acceptability, etc.), and masked regions can be discarded or distrusted.

[0153] Although attachments are discussed herein, they represent only one type of instrument structure. Image quality assessment based on consensus among multiple verification sources may include annotations of other instrument structures, such as buttons, aids, or other structures located on a patient's teeth or other tissues. In some embodiments, the instrument structure may be located on the instrument. For example, the structure may be a protrusion or groove placed on the instrument, with the explicit purpose of identifying good and bad images or assessing the quality of an image or a portion thereof.

[0154] In some embodiments, a mask that indicates the quality across an image can be created. For example, each location in the mask (such as a pixel of the mask) can be assigned a value based on the relative quality of the image at that location.

[0155] In some embodiments, labeled images can be used to train machine learning or artificial intelligence algorithms. For example, images can be labeled by multiple people for the presence of attachments, where consensus among the labelers indicates the quality of the image or a portion of the image, while disagreement among the labelers indicates that the image or a portion of the image is of poor or insufficient quality. Then, even without labels, models trained using these labeled images can be used to determine the quality of images.

[0156] In some embodiments, image quality assessment allows for the determination of quality by utilizing existing information, such as annotations in attachments that may be part of individual patient treatment analysis / monitoring. In some embodiments, this information can be utilized without requiring additional time or effort for quality review.

[0157] In some embodiments, image quality assessment allows for customized image quality estimates for a specific application. For example, in some embodiments, an assessment of consensus regarding a task can be used for a specific application. The example provided herein is a task of annotating attachments in an image by applying image quality assessment. Here, the task of annotating attachments in an image helps determine the location of attachments in the image. However, when multiple images of the same image or the same dentition are annotated with attachment locations, the consensus or disagreement between them can be an indication of the quality of one or more images, where consensus indicates a higher relative quality than disagreement.

[0158] Currently, there is no feasible way to perform image quality estimation across different image types. Training such a model is possible, but it is expected to require slow, expensive, and error-prone annotation work. Annotating image quality across images (such as providing a mask of image quality for each pixel value or group of pixels) could be useful, but may not be feasible. However, the following disclosures and / or descriptions address these drawbacks.

[0159] Image quality has a significant impact on annotation accuracy and consensus among annotators. If labels are based on the entire image (e.g., image classification), images with discrepancies between labels generated by different processes are more likely to be generated from lower-quality images. If labels are based on regions (e.g., they are processed as bounding boxes or segmentation labels), image regions containing discrepancies are more likely to be generated due to lower image quality in those regions.

[0160] In some cases, one or more of the following comparison types can be used to assess label consensus or disagreement: • Compare labels for the same image across different annotators. This consensus among annotators can be used to determine the quality of an image or its parts, as discussed herein. For example, labels can be added to an image or part to identify features or attributes of the image by multiple annotators. Consensus or disagreement among labels for an image or its parts can be an indicator of the quality of the image or its parts, where consensus indicates a higher relative quality than disagreement. • Compare labels with trusted verification sources (such as known patient treatment protocols). In some embodiments, trusted verification sources can be used to compare labeled images. For example, a treatment protocol may indicate the location of attachments or other features in or on the patient's dentition. Annotators can use the locations, attachments, or other features in the image to label photographs of the patient's dentition. If the labeled image matches the treatment protocol data, the image is likely to have acceptable high quality. If the labeled image does not match the treatment protocol, the image is likely to have unacceptable or low quality. • Compare labels (such as labels from human annotators) with machine learning model predictions, where predictions are obtained from models trained on different datasets or using cross-validation techniques. For example, in cross-validation of a machine learning model used to predict tooth numbers. In this case, a mismatch between model predictions and label predictions might be due to model prediction errors rather than labeling errors. If this occurs, the model might learn to predict images that are difficult to segment (because each tooth or pixel in an image labeled with tooth numbers might be part of the segmentation process) as having lower quality. For example, this mismatch between labels and predictions might occur in a dentition with severe anterior occlusion.

[0161] In each of these cases, the image or image region where there is disagreement between the different annotation evaluations is labeled in this way. The machine learning model is then trained to learn these annotation "errors." Note that in implementing this approach, care should be taken to ensure that the annotation errors are not confused with some other aspect of the image.

[0162] As an example, mixed dentitions may have more labeling errors and lower image quality for certain tasks, especially if the images are taken of children. Care should also be taken to ensure that the validation source is free from bias that could affect the assessment of predicted image quality. An example of this might be using cross-validation of a machine learning model to predict tooth numbers as the true source. In this case, label prediction mismatch is often due to model prediction error rather than labeling error. If this occurs, the model may learn to predict images that are difficult to segment as having lower quality (e.g., severe anterior occlusion).

[0163] An example of how this approach can be used in practice includes evaluating consensus among validation sources (e.g., human annotators, procedural annotation models, known treatment protocols for the relevant patient) in determining the presence of dentition or orthodontic attachments, other aids (e.g., buttons, levers), or other features. Attachments are sometimes small and often tooth-colored, making them almost invisible if the image quality is insufficient. Even when photographs are provided as photo sets with multiple angles and 3D models with attachments are available to annotators, many photo sets may have one or more teeth where attachments are not visible, and therefore may not be identified as having such attachments, even if they may actually be present on the patient's teeth. In some embodiments, the determination can be evaluated against the patient's treatment protocol to determine whether the treatment protocol is "in consensus" with the annotator. For example, an annotator may not label a particular tooth of the patient as having an attachment, but the patient's treatment protocol may indicate that the attachment should be present on the tooth. Although the attachment may have fallen off, the "disagreement" between the annotator and the treatment protocol may indicate that the image is of lower quality (or at least the portion of the image near the attachment area is of lower quality), for example, inferring that the annotator did not see the attachment due to lower image quality. As another example, a similar comparison can be made between identified discrepancies among multiple annotators (e.g., human annotators and / or procedural annotators). Similarly, a discrepancy might be seen as evidence of lower image quality (either overall or in portions of the image near attachments or other relevant features).

[0164] Similarly, teeth in regions of relatively high-quality images are likely to have attachments that match the treatment plan, and thus can be said to be more "in consensus" with the treatment plan. Likewise, consensus among multiple validation sources (e.g., multiple human annotators and / or procedural annotators) can be an indication of relatively high quality. If regions with high and low quality images can be identified in the image, such as by comparing how well the tooth attachment labels match the treatment plan, masks showing relatively high and low quality regions across the image can be created. By repeating this operation across all attachment labels, an image / image quality mask pair can be constructed, which can then be used to train a semantic segmentation model. The semantic segmentation model is then trained to predict where low-quality regions are likely to be located in the image.

[0165] In some embodiments, during the training and use of quality assessments employing these methods, the possibility of attachments falling off during treatment can be considered to avoid confusing the visual presence of missing attachments with improper image labeling. One way to mitigate this problem is by using observations where teeth more distant in an image are typically blurrier than those closer to the camera (often the focus point). Because of this, when an attachment label is missing from a tooth, that tooth and any teeth further away in the image may be labeled as being in a lower-quality area of ​​the image.

[0166] Similarly, images of teeth with all attachments labeled (such as those determined based on consensus with treatment protocols and / or other sources of validation, such as the labeler) can be considered to be in areas with relatively high-quality images. In a similar manner, teeth further forward than these will also be considered to be in areas of higher image quality. Some teeth may lack attachments and may be located between a tooth lacking attachment labels and another tooth with attachment labels. In such cases, the image quality of that tooth may not be able to be determined with sufficient accuracy.

[0167] In one embodiment, pixels of teeth in regions of relatively high quality can be labeled with '1', while pixels of teeth in regions of relatively low quality can be labeled with '-1'. Areas outside the teeth, as well as teeth in regions of unknown / undefined quality, may be labeled with '0'. The neural network can receive an image as input and generate a mask of the same size as the input. The network's parameters can learn to output higher (or lower) values ​​for regions of the image with higher (or lower) relative quality.

[0168] A similar approach can be used when an image has already been labeled by multiple annotators. In this case, images (or regions of images) where annotators disagree are more likely to come from lower-quality images / regions. However, having multiple annotators label a single image is not cost-effective. An alternative approach is to use labeled data to train a cross-validated machine learning (ML) model. The model's predictions for the "hold-out" data can be compared to its labels. In this case, regions with higher levels of inaccurate predictions may indicate lower-quality images or regions of lower-quality images.

[0169] In one embodiment, multiple techniques disclosed and / or described may be used together. For example, images from different camera orientations and positions may be used to generate a model of a patient's mouth and teeth using appropriate reconstruction techniques. This may involve using multiple focal lengths to capture multiple images. Then, evaluating blur metrics and / or other quality metrics may be used to determine whether one or more images require focus stacking, and, if instructed, focus stacking may be applied to the images. Next, the quality of one or more focus-stacked images is compared with that of non-focus-stacked images, and the images most useful for a particular training or inference task are selected.

[0170] At box 158, an image set of acceptable quality can be used to train or generate one or more models discussed in this paper, such as for generating reconstructed representations.

[0171] Figure 1(g) depicts a flowchart or diagram illustrating a process, method, operation, or function that can be executed to generate and manipulate deformable 3D models using dynamic differentiable volume rendering processes, such as the dynamic NeRF and dynamic Gaussian splashing processes discussed herein. Dynamic differentiable volume rendering is a rendering technique that extends differentiable volume rendering by incorporating time-based variations, thereby allowing the rendering of dynamically evolving scenes. Dynamic differentiable volume rendering captures changes in a scene over time, thus enabling novel view representations of dynamic scenes. Deformable 3D models allow for parametric representations that can be one or more 3D objects, such as a patient's teeth and jaw, which can be modified by adjusting predefined parameters, such as those discussed below.

[0172] At frame 710, video is captured of the patient moving their jaw and face through various expressions, postures, and / or positions. In some embodiments, positions may include positions 750, which include mid-occlusion, right lateral occlusion, left lateral occlusion, retruded occlusion, protruding occlusion, and open occlusion. Expressions may include facial expressions such as neutral expressions, open-mouth smiles, closed-mouth smiles, etc. Postures may include facial or head postures.

[0173] At box 720, a dynamically differentiable volume rendering model is generated. This dynamically differentiable volume rendering model can be generated using any of the methods described herein. Compared to the model described herein, this model can be generated using additional parameters. For example, additional parameters such as Gaussian parameters can be assigned to each pixel at each time point in the video. These additional parameters can include parameters related to expression, pose, and / or position.

[0174] At box 730, desired views, poses, expressions, positions, etc., can be received or evaluated. For example, in some embodiments, standard views, poses, expressions, positions, etc., may be previously generated using a general model; a 3D model of the patient's teeth, jaw, and other anatomical structures; or another model may be received for generating novel views. In some embodiments, desired views, poses, positions, expressions, etc., may be derived from video or still images of standard views, poses, expressions, etc., of another person or patient. View, pose, position, and expression parameters may be generated based on views, poses, positions, expressions, etc., in models, images, etc. In some embodiments, standard or generated views, poses, expressions, positions, etc., from models, videos, etc., can be evaluated, and one or more sets of parameters can be selected for generating novel views.

[0175] At box 740, view, pose, position, and expression parameters can be used as input to the dynamically differentiable volumetric rendering model of box 720 to generate a novel view 760 based on the patient video captured at box 710, using the desired view, pose, position, and expression from box 730. The novel view 760 can be an image or video showing the patient moving their jaw from a standard viewpoint in a standard movement. For example, the camera used to capture the video at box 710 may not be in a clinically relevant position and / or the patient may not have moved or assumed a clinically relevant pose. The dynamically differentiable volumetric rendering model of box 720 and the view, pose, position, and expression from box 730 can be used to generate clinically relevant data.

[0176] Figure 2 This is an illustration of elements or components that may exist in a computer device or system configured to implement methods, processes, functions, or operations according to embodiments of the systems and methods disclosed herein. As noted, in some embodiments, the systems and methods may be implemented in the form of means including processing elements and a set of executable instructions. The executable instructions may be part of a software application and are arranged in a software architecture.

[0177] Generally, embodiments can be implemented using a set of software instructions designed to be executed by processing elements (such as GPUs, CPUs, TPUs, QPUs, microprocessors, processors, coprocessors, or controllers, as non-limiting examples) that are appropriately programmed. In complex applications or systems, such instructions are typically arranged as "modules," each of which typically performs a specific task, process, function, or operation. The operation of the entire set of modules can be controlled or coordinated by an operating system (OS) or other form of organizational platform.

[0178] Each application module or submodule may correspond to a specific function, method, process, or operation implemented by that module or submodule. Such function, method, process, or operation may include functions, methods, processes, or operations used to implement one or more aspects of the disclosed and / or described systems, apparatus, and methods.

[0179] Application modules and / or submodules may include suitable computer-executable code or instruction sets (e.g., those that may be executed by a processor, microprocessor, or CPU that is appropriately programmed), such as computer-executable code corresponding to a programming language. For example, programming language source code may be compiled into computer-executable code. Alternatively or additionally, the programming language may be an interpreted programming language, such as a scripting language.

[0180] A module may contain one or more sets of instructions for performing the functions and operations described in the accompanying drawings and specification, including the methods, operations, or functions described herein. These modules may include the illustrated modules, but may also include more or fewer modules than those illustrated. As mentioned, each module may contain a computer-executable instruction set. This instruction set may be executed by a programmed processor contained in a server, client device, network element, system, platform, or other component.

[0181] A module or submodule may contain instructions that are executed by processors included in one or more of a server, client device, network element, system, platform, or other component. Therefore, in some embodiments, multiple electronic processors (each part of a separate device, server, or system) may be responsible for executing all or part of the software instructions contained in the illustrated module or submodule. Thus, although Figure 2 The diagram illustrates a set of modules that perform multiple functions or operations together, but these functions or operations may be performed by different devices or system elements, wherein some modules / submodules (or instructions contained in those modules / submodules) are associated with those devices or system elements.

[0182] like Figure 2As shown, system 200 may represent a server or other form of computing or data processing system, platform, or device. Each module (or submodule) 202 contains an executable instruction set, wherein when the instruction set is executed by one or more suitable electronic processors (such as the electronic processor indicated by "physical processor 230" in the figure), system (or server, platform, or device) 200 operates to perform a particular process, operation, function, or method.

[0183] Module 202 is stored in (non-transitory) memory 220, which typically includes an operating system module 204 containing instructions for accessing and controlling the execution of instructions contained in other modules (and other functions). Module 202 stored in memory 220 is accessed via a "bus" or communication line 216 for transferring data and executing instructions. The bus or communication line 216 also allows processor(s) 230 to communicate with the module to access and execute instruction sets. The bus or communication line 216 also allows processor(s) 230 to interact with other elements of system 200, such as input or output devices 222, communication elements 224 for exchanging data and information with devices outside system 200, and additional memory devices 226.

[0184] For example, module 202 may contain computer-executable instructions that, when executed by a programmed processor, cause the processor or a device implementing it to perform a process, method, function, or operation.

[0185] Module 206 can provide the patient with an application for controlling the mobile device's camera. In one embodiment, the application can be obtained by the patient downloading it from a remote platform or website.

[0186] Module 208 can acquire images(s) or videos captured by a mobile device camera controlled by an installed application. In one embodiment, timing (such as acquisition time or time interval between captured images), location, environmental conditions (light, temperature), camera settings (focal length, exposure, illumination, aperture, focus distance) and / or other characteristics of the captured images or videos are expected to be controlled and / or stored by the application.

[0187] In one embodiment, the location, orientation, or other characteristics (multiple) of the mobile device associated with the image or video can be determined and provided to a dentist or orthodontist along with the image or video (such as in the form of metadata and / or mobile device sensor data). The video and / or images are provided to the patient's dentist or orthodontist directly or via access to a remote platform (such as a multi-tenant or SaaS platform). If the patient captures video, the video can then be processed to generate one or more images. As a non-limiting example, this may include sampling or filtering.

[0188] Module 210 can process, evaluate, or otherwise filter images to obtain a set of images of sufficient quality for use in the reconstruction process. As further disclosed and / or described herein, this processing phase may involve one or more of the following for the purpose of evaluating (and in some cases, correcting) image quality: detecting poor or inadequate image quality (such as by determining the degree of blurring in the image or whether the initial image is uncorrectable); performing comparisons with previously obtained images to assist in identifying changes in tooth position or alignment (or misalignment) between images and / or to assist in evaluating the quality of images provided by the patient; and using a trained machine learning model to assign values ​​to image quality or relative image quality.

[0189] Module 212 can use filtered or otherwise evaluated images as input to one or more reconstruction processes to generate baseline representations for synthesizing one or more novel views. Examples of image processing techniques that can be used to generate reconstructed representations (or fusions) include, but are not limited to: neural radiation fields, Gaussian splashing, image stitching, motion reconstruction structures, and finely tuned personalized diffusion models.

[0190] Module 213 can generate one or more novel views based on the reconstructed representation or baseline. The reconstructed representation can be used to generate one or more images that represent the patient's teeth as viewed from the angle, position, environmental conditions, or other aspects of interest desired by a dentist or orthodontist;

[0191] Module 214 can evaluate one or more generated views to determine the current state of the patient's teeth and the progress of the treatment plan. This process can be used to monitor the progress of the treatment plan. This step or stage is typically performed manually by a dentist or orthodontist, but can also be performed wholly or partially by a trained model.

[0192] Module 215 can use multiple generated images to train a machine learning model to act as a classifier. This is an optional stage, but includes: using multiple generated images to train the model as a classifier for diagnosis, treatment planning, or determining the state or changes in state of a patient's teeth during training. In one embodiment, the images used as part of the training dataset can be obtained from one or more of the following sources: reconstructed representations disclosed and / or described herein when operating under multiple conditions or input parameters, existing sets of images and annotations that have been evaluated to determine their annotation accuracy, and / or sets of images generated by a successive blurring process—in which case it is beneficial to determine changes in the degree of blur between images after successive application of blur filters—if the change indicates that the original image is sufficiently sharp, one or more blurred images from the original and blurred images can be used as part of the training dataset for the machine learning model. Module 215 can also operate the trained model to aid in tooth visualization, diagnosis, monitoring, and other dental functions. For example, as discussed herein, the model can be used to create visual representations of a patient's dental anatomy, such as the current state of the patient's teeth from one or more views. In some embodiments, the images can be used to aid in diagnostic applications. Dental diagnosis includes identifying and assisting in the treatment of various oral health problems, including diagnosing health issues of the teeth, gums, and other structures in the mouth. Dental monitoring may include tracking the progress and status of a patient's dental health and orthodontic treatment over time. For example, a trained model can be used to generate multiple visualizations of a patient's dentition over time from the same perspective (including field of view and focal length), such as based on multiple imaging sessions taken at intervals over a period of time. Dental professionals can monitor changes in the patient's gums over time from this same perspective and can make diagnoses based on, for example, changes in gingival recession. As mentioned, in some embodiments, the systems and methods disclosed and / or described herein may be provided through a Software as a Service (SaaS) or multi-tenant platform. This platform provides access to multiple entities, each with a separate account associated with a data storage device. For example, each account may correspond to a dentist or orthodontist, a patient, an entity, a set or class of entities, a set or type of patients, an insurance company, or an organization. Each account may access one or more services, wherein the set of services is instantiated within its account, and implements one or more of the methods or functions disclosed and / or described herein.

[0193] Figure 3 This is a diagram illustrating a SaaS system that can be implemented according to embodiments of the present disclosure. Figure 4 This is a diagram illustrating elements or components of an example operating environment in which embodiments of the present disclosure may be implemented. Figure 5 The illustration shows an embodiment of this disclosure that can be implemented therein. Figure 4A diagram showing additional details of the components or parts of a multi-tenant distributed computing service platform.

[0194] In some embodiments, the systems or services(s) disclosed and / or described herein may be implemented as microservices, processes, workflows, or functions that execute in response to requests. Microservices, processes, workflows, or functions may be executed by servers, data processing elements, platforms, or systems. In some embodiments, services may be provided by a service platform located in the "cloud." In such embodiments, the platform is accessible via APIs and SDKs.

[0195] The described document processing and evaluation services can be provided as microservices within the platform to each user or company. The interfaces of these microservices can be defined using REST and GraphQL endpoints. The management console allows users or administrators secure access to the underlying request and response data, manage accounts and access, and, in some cases, modify processing workflows or configurations.

[0196] Note that, although Figures 3 to 5 The illustration depicts a multi-tenant or SaaS architecture that can be used to deliver business-related or other applications and services to multiple accounts / users. However, this architecture can also be used to deliver other types of data processing services and provide access to other applications. For example, this architecture can be used to provide document processing and evaluation processes disclosed and / or described herein.

[0197] Although in some embodiments, Figures 3 to 5 The platform or system of the type illustrated may be operated by a third-party provider, but in other embodiments, the platform may be operated by a provider and different sources may use the platform to provide applications or services to users.

[0198] Figure 3 This is a diagram illustrating an embodiment of a system 300 in which embodiments of the present disclosure may be implemented or through which the services disclosed and / or described herein may be accessed. Taking advantage of the business service system hosted by an Application Service Provider (ASP), such as a multi-tenant data processing platform, users of the service, as a non-limiting example, may include individuals, businesses, shops, or organizations. Users may access the service using a suitable client, including but not limited to desktop computers, laptop computers, tablet computers, scanners, or smartphones. Generally, client devices capable of accessing the Internet may be used to provide requests for services (such as document processing) or text messages. Users interface with the service platform across the Internet 308 or another suitable communication network or combination of networks. Non-limiting examples of suitable client devices include desktop computers 303, smartphones 304, tablet computers 305, or laptop computers 306.

[0199] The system 310, which can be hosted by a third party, may include, for example: Figure 3 The coupled service set 312 and web interface server 314 are shown. It should be understood that either or both of service 312 and web interface server 314 can be implemented on one or more different hardware systems and components, even... Figure 3 It is represented as a single unit.

[0200] As a non-limiting example, service 312 may include one or more processes, functions, or operations for providing a patient with an application to assist in controlling a set of images or videos captured by a mobile device camera, processing the set of images or videos captured by the patient's mobile device camera, selecting a set of images for a reconstruction process, generating one or more novel views from the reconstruction process, and using the generated novel views to develop and / or assess the progress of the patient's treatment plan.

[0201] In some embodiments, the set of applications or services available to a user may include one or more applications or services that perform the functions and methods disclosed and / or described herein. As an example, in some embodiments, the set of applications, functions, operations, or services available through platform or system 310 may include: • Account management services 316, such as: The process or service of authenticating persons or entities requesting data processing services (such as credentials, purchase receipts, or verification that customers have been authorized by the company to use the services provided by the platform). The process or service of receiving requests to process images or video sets; ο An optional process or service used to generate a price for a requested service or to charge a fee in accordance with a service contract; ο Generate a container or instantiated process or service for the requested process for the user / client, where instantiation can be customized for a specific company; and Other forms of account management services; • Processes or services, 318 sets, such as: Provide / download an application to patients to control the camera on their mobile devices; ο Acquire images (multiple) or videos captured by the mobile device's camera, controlled by the application; To process, evaluate, or filter images to determine a set with sufficient image quality; A baseline representation is generated through a reconstruction process using an image set of sufficient quality. Use baseline / reconstruction representations to generate one or more "novel" views; ο Evaluate the generated views (multiple) to determine the expected treatment plan and / or the progress of the current plan; Train ML models to act as classifiers for diagnosis and treatment planning; • Implement service 320, such as: This enables data processing and service providers and / or platforms to implement and configure processes and services provided to users.

[0202] Figure 3 The platform or system shown can be hosted on a distributed computing system consisting of at least one (but typically multiple) "servers". A server is a physical computer dedicated to providing a data storage and execution environment for one or more software applications or services designed to serve the needs of users on other computers communicating with the server (e.g., via a public network such as the Internet). The server and the services it provides can be referred to as a "host", and the remote computer and the software application running on the served remote computer can be referred to as a "client". Depending on the computing services (multiple) provided by the server, it can be called a database server, data storage server, file server, mail server, print server, or web server (as an example).

[0203] Figure 4 This is a diagram illustrating elements or components of an example operating environment 400 in which embodiments of the present disclosure may be implemented. As shown, multiple clients 402, integrated and / or incorporated into multiple computing devices, can communicate with a multi-tenant service platform 408 via one or more networks 414. For example, clients may be integrated and / or incorporated into client applications (e.g., software) implemented or executed at least in part by one or more computing devices. Examples of suitable computing devices include personal computers, server computers 404, desktop computers 406, laptop computers 407, notebook computers, tablet computers or personal digital assistants (PDAs) 410, smartphones 412, cellular phones, and consumer electronic devices that integrate one or more computing device components, such as one or more electronic processors, microprocessors, central processing units (CPUs), or controllers. Examples of suitable networks 414 include networks utilizing wired and / or wireless communication technologies and networks operating according to any suitable networking and / or communication protocol (e.g., the Internet).

[0204] The distributed computing service / platform (which may also be referred to as a multi-tenant data processing platform) 408 may include multiple processing layers, including a user interface level 416, an application server level 420, and a data storage level 424. The user interface level 416 may maintain multiple user interfaces 417, including graphical user interfaces and / or web-based interfaces. The user interfaces may include a default user interface for the service, which provides access to applications and data for users or "tenants" of the service (depicted as "service UI" in the diagram); and one or more user interfaces (e.g., represented in the diagram as "tenant A UI", ..., "tenant Z UI" and accessible via one or more APIs), which have been specialized / customized according to user-specific requirements.

[0205] The default user interface may include user interface components that enable tenants to access and use the functions and capabilities provided by the service platform. As a non-limiting example, this may include accessing tenant data, initiating the instantiation of a specific application, or causing the execution of a specific data processing operation. Each application server or processing level 422 shown in the figure may be implemented using a set of computers and / or components including computer servers and processors, and may perform various functions, methods, processes, or operations as determined by the execution of a software application or instruction set. Data storage level 424 may include one or more data repositories, which may include a service data repository 425 and one or more tenant data repositories 426. The data repositories may be implemented using suitable data storage technologies, including relational database management systems (RDBMS) based on Structured Query Language (SQL).

[0206] Service platform 408 may be multi-tenant and can be operated by entities to provide a set of business-related or other data processing applications, data storage, and functionality to multiple tenants. For example, applications and functionality may include providing web-based access to functionality used by the enterprise to serve end users, allowing users with a browser and internet or intranet connectivity to view, enter, process, or modify certain types of information. Such functionality or applications are typically implemented by one or more software code / instruction modules, which are maintained and executed by one or more servers 422, which are part of the platform's application server layer 420. (See also: [link to relevant documentation]). Figure 3 As pointed out, Figure 4 The platform system shown can be hosted on a distributed computing system consisting of at least one (but typically multiple) "servers".

[0207] As mentioned, enterprises can utilize platforms or systems provided by third parties instead of building and maintaining such platforms or systems themselves. Third parties can implement business systems / platforms as described in the context of a multi-tenant platform, where users are provided with individual instantiations of the enterprise's data processing workflows (such as image processing and usage disclosed and / or described herein), with each company / enterprise representing a tenant of the platform. One advantage of such multi-tenant platforms is that each tenant can customize an instance of the data processing workflow according to that tenant's specific business needs or operational methods. Furthermore, each tenant can be an enterprise or entity using the multi-tenant platform to provide business services and functionality to multiple users.

[0208] Figure 5 The illustration shows an embodiment of this disclosure that can be implemented therein. Figure 4 The diagram illustrates additional details of the components or parts of a multi-tenant distributed computing service platform. Generally, embodiments can be implemented using a set of software instructions designed to be executed by processing elements (such as CPUs, microprocessors, processors, controllers, or computing devices) programmed in a suitable manner. In complex systems, such instructions are typically arranged as "modules," each performing a specific task, process, function, or operation. The operation of the entire set of modules can be controlled or coordinated by an operating system (OS) or other form of organizational platform.

[0209] Figure 5 The example architecture 500 of the illustrated multi-tenant distributed computing service platform includes a user interface layer or level 502 with one or more user interfaces 503. Examples of such user interfaces include graphical user interfaces and application programming interfaces (APIs). Each user interface may include one or more interface elements 504. For example, a user can interact with interface elements to access functionality and / or data provided by the application and / or data storage layer of the example architecture. Examples of graphical user interface elements include buttons, menus, checkboxes, drop-down lists, scroll bars, sliders, spinners, text boxes, icons, labels, progress bars, status bars, toolbars, windows, hyperlinks, and dialog boxes. Application programming interfaces can be local or remote and may include interface elements such as parameterized program calls, programmable objects, and messaging protocols.

[0210] Application layer 510 may include one or more application modules 511, each application module 511 having one or more associated submodules 512. Each application module 511 or submodule 512 may correspond to a function, method, process, or operation implemented by that module or submodule (e.g., a function or process related to providing data processing and other services to users of the platform). Such function, method, process, or operation may include functions, methods, processes, or operations used to implement one or more aspects of the disclosed system and methods, such as one or more processes, operations, or functions disclosed and / or described with reference to the specification and drawings: • Provide patients with / allow an application to control the camera on their mobile devices; • Acquire images (multiple) or videos captured by the mobile device's camera, controlled by the application; • Process, evaluate, or filter images to determine an image set with sufficient quality; • Generate a baseline representation through a reconstruction process using an image set of sufficient quality; • Generate one or more “novel” views using baseline / reconstruction representation; • Evaluate the generated views (multiple) to determine the expected treatment plan and / or the progress of the current plan; • Train ML models to act as classifiers for diagnosis and treatment planning;

[0211] Application modules and / or submodules may include any suitable computer-executable code or instruction set (e.g., as that which may be executed by a processor, microprocessor, or CPU that is appropriately programmed), such as computer-executable code corresponding to a programming language. For example, programming language source code can be compiled into computer-executable code. Alternatively or additionally, the programming language may be an interpreted programming language, such as a scripting language. Each application server (e.g., such as...) Figure 4 The element (represented by 422) may include each application module. Alternatively, different application servers may include different sets of application modules. Such sets may be disjoint or overlapping.

[0212] Data storage layer 520 may include one or more data objects 522, each data object 522 having one or more data object components 521, such as attributes and / or behaviors. For example, a data object may correspond to a table in a relational database, and data object components may correspond to columns or fields of such a table. Alternatively or additionally, a data object may correspond to a data record with fields and associated services. Alternatively or additionally, a data object may correspond to a persistent instance of a programmatic data object, such as structure and category. Each data repository in the data storage layer may include each data object. Alternatively, different data repositories may include different sets of data objects. Such sets may be disjoint or overlapping.

[0213] Notice, Figures 3 to 5 The example computing environment depicted is not intended to be a limiting example. Other environments that may be implemented in whole or in part in the embodiments include devices (including mobile devices), software applications, systems, apparatuses, networks, SaaS platforms, IaaS (Infrastructure as a Service) platforms, or other configurable components that can be used by multiple users for data entry, data processing, application execution, or data viewing (as non-limiting examples).

[0214] The embodiments disclosed and / or described herein can be implemented in a modular or integrated manner using computer software in the form of control logic. Based on the disclosure and teachings provided herein, those skilled in the art will understand and appreciate other ways and / or methods of implementing the invention using hardware and combinations of hardware and software.

[0215] This disclosure includes the following terms and embodiments:

[0216] Clause 1. A method for performing orthodontic treatment, comprising: receiving an orthodontic treatment plan to move a patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan including a plurality of treatment phases, each of the plurality of treatment phases including a corresponding dental arrangement associated with the patient's teeth; receiving one or more first images of the patient's teeth captured at a first time when the patient's teeth are in the first dental arrangement, the first images having been captured from one or more first camera positions, orientations, or settings; receiving one or more second images of the patient's teeth captured at a second time when the patient's teeth are in the second dental arrangement, the second time being different from the first time, the second images having been captured from one or more second camera positions, orientations, or settings; generating a novel image of the teeth in the first dental arrangement or the second dental arrangement based on one or more of the first images or the second images, the novel image representing the teeth in the first dental arrangement or the second dental arrangement as viewed from a camera position, orientation, or setting different from the camera position, orientation, or setting of the one or more first images or the second images.

[0217] Clause 2. The method according to Clause 1 further includes: displaying the novel image and one or more of the first or second images used to generate the novel image.

[0218] Clause 3. The method according to Clause 1, wherein the novel image and one or more of the first or second images used to generate the novel image are simultaneously displayed.

[0219] Clause 4. The method according to Clause 1, wherein the novel image is generated as an image viewed from a position and orientation that matches the position and orientation of one or more first images or one or more second images.

[0220] Clause 5. The method according to Clause 1, wherein the novel image is generated as an image viewed from a position and orientation that does not match the position and orientation of the one or more first images or the one or more second images.

[0221] Clause 6. The method according to Clause 1 further comprises: accessing an orthodontic treatment plan to move the patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan comprising a plurality of treatment phases, each of the plurality of treatment phases being associated with a corresponding dental arrangement of the patient's teeth; and evaluating the progress of the treatment plan by comparing the novel image with one or more of the first or second images used to generate the novel image.

[0222] Clause 7. The method according to Clause 1, wherein generating a novel image of the tooth in the first dental arrangement or the second dental arrangement based on one or more of the first image or the second image, the novel image representing the tooth in the first dental arrangement or the second dental arrangement as viewed from a camera position and orientation different from that of the one or more first images or the second images, further comprises: using one or more of the following: neural radiation field, Gaussian splashing, smart image stitching, Gaussian splashing, or a finely tuned diffusion model.

[0223] Clause 8. The method according to Clause 1, wherein the first time is before the implementation of the treatment plan, and the second time is after at least a portion of the implementation of the treatment plan.

[0224] Clause 9. The method according to Clause 1, wherein the different camera positions and orientations correspond to the position and orientation of one of the first images or the second images.

[0225] Clause 10. The method according to Clause 1, wherein the one or more first images or the one or more second images are obtained from a video provided by the patient.

[0226] Clause 11. The method according to Clause 1, wherein before generating a novel image of the tooth in the first dental arrangement or the second dental arrangement based on one or more of the first image or the second image, the method includes: evaluating the quality of the first image or the one or more of the second images.

[0227] Clause 12. The method according to Clause 7, wherein evaluating the quality of one or more of the first image or the second image further comprises: using a trained model to evaluate the quality.

[0228] Clause 13. The method according to Clause 1, wherein the first image of the tooth is captured at multiple exposure settings.

[0229] Clause 14. The method according to Clause 1, wherein the first image of the teeth is captured in multiple focus settings.

[0230] Clause 15. The method according to Clause 7, wherein evaluating the quality of one or more of the first image or the second image further comprises: determining a blur metric.

[0231] Clause 16. The method according to Clause 8 further includes: displaying the uncaptured 2D image of the tooth in the second dental arrangement.

[0232] Clause 17. The method according to Clause 1, wherein the first time is during the first treatment phase and the second time is during the second treatment phase.

[0233] Clause 18. The method according to Clause 1, wherein the treatment phase includes the time prior to the commencement of the orthodontic treatment.

[0234] Clause 19. The method according to Clause 1, wherein the second 2D image comprises a plurality of still images.

[0235] Clause 20. The method according to Clause 1, wherein the second image includes video.

[0236] Clause 21. The method according to Clause 1 further comprises: selecting an image from the second image, wherein a first camera position and orientation among the plurality of camera positions and orientations associated with the image selected from the first image more closely corresponds to a first position and orientation among the positions and orientations associated with the first image compared to other camera positions and orientations among the plurality of camera positions and orientations.

[0237] Clause 22. The method according to Clause 21 further includes: displaying the image selected from the first image while displaying the second image.

[0238] Clause 23. The method according to Clause 1, wherein generating the second image comprises: generating a Gaussian splash image based on the second image.

[0239] Clause 24. The method according to Clause 1, wherein generating the second image comprises: generating a Gaussian sputtered 3D representation of the patient's tooth based on the second image; and rendering a 2D image of the 3D representation of the tooth based on the first camera position and orientation associated with the first image to generate the second 2D image of the tooth.

[0240] Clause 25. The method according to Clause 1, wherein generating the second 2D image comprises: generating a neural radiation field of the patient's tooth based on the second 2D image; and rendering a 2D image of a 3D representation of the tooth using the neural radiation field based on the first camera position and orientation associated with the first 2D image to generate the second 2D image of the tooth.

[0241] Clause 26. The method according to Clause 1 further includes: determining that the position and orientation of the second camera are different from the position and orientation of the first camera.

[0242] Clause 27. An orthodontic treatment method, the method comprising: receiving an orthodontic treatment plan to move a patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan including a plurality of treatment phases, each of the plurality of treatment phases including a corresponding arrangement of the patient's teeth; receiving a first 2D image of the patient's teeth captured at a first time; receiving a second 2D image of the patient's teeth captured at a second time, the second time corresponding to a phase of the orthodontic treatment plan, the second 2D image being captured from a plurality of camera positions and orientations and having a plurality of image characteristics; selecting a subset of the second 2D images based on the image characteristics; and combining the subset of the second 2D images to generate a third 2D image, the third 2D image including a plurality of 2D images at the plurality of camera positions and orientations; and comparing the position of the patient's teeth in the first 2D image with the position of the patient's teeth in the third 2D image.

[0243] Clause 28. The method according to Clause 27, wherein said plurality of image characteristics includes image exposure.

[0244] Clause 29. The method according to Clause 28, wherein combining the subset of the second 2D image comprises: aligning the subset of the 2D image; and tone mapping the 2D image, wherein the plurality of 2D images at the plurality of camera positions and orientations comprises a plurality of tone-mapped 2D images at the plurality of camera positions and orientations.

[0245] Clause 30. The method according to Clause 27, wherein said plurality of image features includes an image focal plane.

[0246] Clause 31. The method according to Clause 30, wherein combining the subset of the second 2D image comprises: aligning the subset of the 2D image; and combining the focused portions of the subset of the second 2D image to generate the plurality of 2D images at the plurality of camera positions and orientations.

[0247] Clause 32. The method according to Clause 27, wherein said plurality of image features includes depth of field.

[0248] Clause 33. The method according to Clause 32, wherein combining the subset of the second 2D image comprises: aligning the subset of the 2D image; and combining the focused portions of the subset of the second 2D image to generate the plurality of 2D images at the plurality of camera positions and orientations.

[0249] Clause 34. The method described in Clause 27, wherein the first time is before the commencement of dental treatment and the second time is after the commencement of treatment.

[0250] Clause 35. An orthodontic treatment method, the method comprising: receiving a first 2D image of the patient's teeth, the first 2D image being captured from a plurality of camera positions and orientations and corresponding to a stage of the orthodontic treatment plan; evaluating the quality of the first 2D image by iteratively: altering an aspect of each of the first 2D images; determining a metric for the altered aspect of a plurality of portions of each of the first 2D images; aggregating the determined metric for each of the plurality of portions of each of the first 2D images to generate an aggregated metric; and determining a quality metric of the image based on the aggregated metric in the iterations.

[0251] Clause 36. The method according to Clause 35, wherein: the aspect is the blurriness of the image.

[0252] Clause 37. The method according to Clause 36, wherein: the change includes applying a blur filter to the image.

[0253] Clause 38. The method according to Clause 37, wherein: the blur filter is a Gaussian blur filter.

[0254] Clause 39. The method described in accordance with Clause 37, wherein: the metric is a fuzzy metric.

[0255] Clause 40. The method according to Clause 39, wherein determining the measure of the altered aspect of each of the plurality of portions of the first 2D image comprises: applying a Laplacian filter to the blurred image.

[0256] Clause 41. The method according to Clause 40, wherein determining the quality metric of the image based on the aggregated metric within the iteration comprises: determining the slope of the aggregated blur metric within the first two iterations.

[0257] Clause 42. The method according to Clause 40, wherein determining the quality metric of the image based on the aggregated metric within the iteration comprises: determining the value of the aggregated blur metric after at least 10 iterations.

[0258] Clause 43. The method according to Clause 40, wherein determining the quality metric of the image based on the aggregated metric within the iteration comprises: determining the value of the area of ​​the aggregated blur metric relative to the curve of the iteration after at least 10 iterations.

[0259] Clause 44. An orthodontic treatment method, the method comprising: receiving a first 2D image of the teeth of the patient; receiving labels from a plurality of annotators indicating the location of dental appliance structures in the first 2D image; comparing the location of each of the plurality of dental appliance structures from each of the plurality of annotators; and determining an image quality metric for each of the first 2D images based on the comparison.

[0260] Clause 45. The method according to Clause 44, wherein the comparison comprises: comparing the presence of dental appliance structures on a particular tooth of each of the annotators.

[0261] Clause 46. The method according to Clause 45, wherein the determination of the image quality metric is based on the degree of consensus on the presence or absence of dental appliance structures on a particular tooth for each of the annotators.

[0262] Clause 47. The method described in Clause 46, wherein a greater consensus indicates higher image quality.

[0263] Clause 48. The method according to Clause 47, wherein determining an image quality metric for each of the first 2D images based on the comparison comprises: determining an image quality metric for a plurality of portions of each of the first 2D images based on the comparison.

[0264] Clause 49. The method described in Clause 45, wherein the labels from multiple labelers include labels for dental appliance structures on anterior and posterior teeth.

[0265] Clause 50. The method according to Clause 49, wherein determining the image quality metric of the plurality of portions of each of the first 2D images in the first 2D images based on the comparison comprises: determining the degree of consensus on the presence of dental appliance structures on the anterior and posterior teeth.

[0266] Clause 51. The method according to Clause 50, wherein the comparison of the position of each of the plurality of dental appliance structures from each of the plurality of annotators comprises: comparing the position of each of the plurality of dental appliance structures from each of the plurality of annotators with the position of the appliance structure in the first phase of the orthodontic treatment plan.

[0267] Clause 52. The method according to Clause 45, wherein the quality metric of the portion of the image of the anterior teeth is based on the degree of consensus regarding the presence or absence of dental appliance structures on the anterior teeth, and the quality metric of the portion of the image of the posterior teeth is based on the degree of consensus regarding the presence or absence of dental appliance structures on the posterior teeth.

[0268] Clause 53. The method described in Clause 44, wherein the appliance structure is an accessory.

[0269] Clause 54. The method according to Clause 44, wherein the device structure is a structure attached to the patient's tissue.

[0270] Clause 55. The method according to Clause 44, wherein the appliance structure is a structure on an orthodontic appliance worn by the patient when the first 2D image is captured.

[0271] Clause 56. The method according to Clause 44, wherein the label includes a confidence value that indicates the labeler's confidence in determining that the appliance structure is located at the labeled location.

[0272] Clause 57. The method described in Clause 44, wherein the annotator is a human or an algorithm.

[0273] Clause 58. A method for performing orthodontic treatment, comprising: receiving a first set of images of a patient's mouth captured using a mobile device; evaluating the first set of images to generate a second set of images for a reconstruction process; using the second set of images as input to the reconstruction process to generate a baseline representation for generating one or more novel views of the patient's mouth; generating one or more novel views based on the generated baseline representation; and evaluating the generated one or more novel views to determine a treatment plan required by the patient or to determine the current state of the patient's teeth and to monitor the progress of the treatment plan.

[0274] Clause 59. The method according to Clause 58, wherein instead of the image set, video is captured using the mobile device, and the method further comprises: processing the video to generate the first image set.

[0275] Clause 60. The method described in Clause 58 further includes: training a machine learning model using a plurality of generated views as a classifier operation; and operating the trained model to perform diagnostic or evaluation functions.

[0276] Clause 61. The method described in Clause 58, wherein the reconstruction process includes one or more of the following: neural radiation field (NeRF), Gaussian splashing, image stitching, or diffusion model.

[0277] Clause 62. The method according to Clause 58 further includes: using an application installed on the mobile device to control the operation of the camera or mobile device during the capture of the image set.

[0278] Clause 63. The method described in Clause 62, wherein the application controls one or more of the following: camera depth of field, camera exposure settings, or lighting.

[0279] Clause 64. The method according to Clause 62, wherein the application provides a user interface display or audio clip to assist the patient in capturing the image.

[0280] Clause 65. The method according to Clause 64, wherein sensor data from the mobile device is used to provide the patient with guidance on locating the mobile device or changing the environment in which the images were collected.

[0281] Clause 66. The method according to Clause 65, wherein the sensor data is data from one or more of the optical sensors or gyroscopes in the mobile device.

[0282] Clause 67. The method according to Clause 65, wherein the sensor data applicable to each image is associated with the image.

[0283] Clause 68. The method according to Clause 67, wherein the sensor data is used to adjust the appearance of one or more images in the image.

[0284] Clause 69. The method according to Clause 58, wherein evaluating the provided images to generate a second image set for the reconstruction process further comprises: determining the quality of one or more images among the provided images; and removing images of insufficient quality before generating the second image set.

[0285] Clause 70. The method according to Clause 69, wherein determining the quality of one or more of the provided images includes one or more of the following: determining the degree of blur of the image; performing a comparison with a previously obtained image; or operating a trained model to generate an indication of the relative quality or utility of the image.

[0286] Clause 71. The method according to Clause 58 further comprises: receiving an image of the patient's mouth, the image being captured at a later time compared to the first image set; accessing a stored version of the baseline representation; using the baseline representation to generate a novel view of the patient's mouth, the generated novel view having camera position, camera orientation, camera settings, or lighting characteristics of the received image; comparing the generated novel view with the received image; and determining, based on the comparison, whether to maintain or change the patient's treatment regimen.

[0287] Clause 72. The method described in Clause 71, wherein the determination of whether to maintain or change the patient’s treatment plan is performed by one or more of a dentist, orthodontist, or trained model.

[0288] Clause 73. The method according to Clause 70, wherein the trained model is trained using an image set and associated labels, and further, wherein the accuracy of the labels is determined using a model trained to identify the error when comparing the image set and associated labels with more than one single source.

[0289] Clause 74. A system for performing orthodontic treatment, comprising: an application for controlling one or more aspects of a camera of a mobile device; a remote platform or server for receiving a first set of images of a patient's mouth captured by said device, and generating a second set of images for a reconstruction process in response to evaluating the first set of images; generating a baseline representation using the second set of images as input to the reconstruction process for generating one or more novel views of the patient's mouth; generating one or more novel views based on said generated baseline representation; and evaluating said generated one or more novel views to determine the treatment plan required by the patient or to determine the current state of the patient's teeth and to monitor the progress of the treatment plan.

[0290] Clause 75. The system according to Clause 74, wherein the application controls one or more of the following: camera depth of field, camera exposure settings, use of panorama settings, or lighting.

[0291] Clause 76. The system pursuant to Clause 75, wherein the application provides a user interface display or audio clip to assist the patient in capturing the image.

[0292] Clause 77. The system of claim 74, wherein the remote platform or server further operates to: receive an image of the patient's mouth, the image being captured at a later time compared to the first image set; access a stored version of the baseline representation; use the baseline representation to generate a novel view of the patient's mouth, the generated novel view having camera position, camera orientation, camera settings, or lighting characteristics of the received image; compare the generated novel view with the received image; and determine, based on the comparison, whether to maintain or change the patient's treatment plan.

[0293] Clause 78. A method for performing orthodontic treatment, comprising: receiving a first set of images of a patient's mouth captured using a mobile device; evaluating the first set of images to generate a second set of images for a reconstruction process; using the second set of images as input to the reconstruction process to generate a baseline representation for generating one or more novel views of the patient's mouth; generating one or more novel views based on the generated baseline representation; and evaluating the generated one or more novel views to determine a treatment plan required for the patient or to determine the current state of the patient's teeth and to monitor the progress of the treatment plan.

[0294] Clause 79. A method for orthodontic treatment, comprising: receiving a first plurality of images of a patient’s dentition, the first plurality of images being captured at a plurality of different locations, orientations, and focal distances; determining the quality of each of the images in the first plurality of images; determining, based on the quality of each of the images, to combine one or more images captured at different focal distances; and combining the one or more images captured at different focal distances to generate a focused stacked image.

[0295] Clause 80. The method according to Clause 79, wherein the focused stacked image is an image from the first image set according to any one of Clauses 1 to 78, one or more second images of the patient's teeth, one or more first images of the patient's teeth, a first 2D image, or a second 2D image.

[0296] Clause 81. The method according to Clause 79 further comprises: comparing the quality of the focused stacked image with the one or more images taken at different focus distances; and selecting the focused stacked image or the one or more images taken at different focus distances based on the comparison of the quality.

[0297] Clause 82. A system for performing orthodontic treatment, comprising a processor; and a memory including instructions that, when executed by the processor, cause the system to: access one or more first image data of the patient's teeth captured at a first time when the patient's teeth are in a first dental arrangement, the first image data having been captured from one or more first camera positions, orientations, or settings; access one or more second image data of the patient's teeth captured at a second time when the patient's teeth are in a second dental arrangement, the second time being different from the first time, the second image data having been captured from one or more second camera positions, orientations, or settings; and generate novel image data of the teeth in the first dental arrangement or the second dental arrangement based on one or more of the first image data or the second image data, the novel image data representing the teeth in the first dental arrangement or the second dental arrangement as viewed from a camera position, orientation, or setting different from the one or more first image data or the second image data.

[0298] Clause 83. The system for performing orthodontic treatment as described in Clause 82, wherein the instructions, when executed by the processor, further cause the system to: receive an orthodontic treatment plan to move a patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan comprising a plurality of treatment phases, each of the plurality of treatment phases comprising a corresponding dental arrangement associated with the patient's teeth.

[0299] Clause 84. The system for performing orthodontic treatment as described in Clause 82, wherein the first image data and the second image data are a first still image and a second still image.

[0300] Clause 85. The system for performing orthodontic treatment as described in Clause 84, wherein the novel image data is a new still image.

[0301] Clause 86. The system for performing orthodontic treatment as described in Clause 85, wherein the instructions, when executed by the processor, also cause the system to: simultaneously display the new still image and one or more of the first or second still images used to generate the new still image.

[0302] Clause 87. The system for performing orthodontic treatment as described in Clause 86, wherein the new still image is generated as an image viewed from a position and orientation that matches the position and orientation of one of the first still image or the second still image.

[0303] Clause 88. The system for performing orthodontic treatment as described in Clause 86, wherein the new still image is generated as an image viewed from a position and orientation that does not match the position and orientation of the first still image or the second still image.

[0304] Clause 89. The system for performing orthodontic treatment as described in Clause 82, wherein the first image data and the second image data are a first video and a second video.

[0305] Clause 90. The system for performing orthodontic treatment as described in Clause 89, wherein the novel image data is a new video.

[0306] Clause 91. A system for performing orthodontic treatment as described in Clause 90, wherein the instructions, when executed by the processor, also cause the system to: simultaneously display the new video and one or more of the first or second video used to generate the new video.

[0307] Clause 92. The system for performing orthodontic treatment as described in Clause 91, wherein the new video is generated as a video viewed from a position and orientation matching the position and orientation of the first video or the second video.

[0308] Clause 93. The system for performing orthodontic treatment as described in Clause 91, wherein the new video is generated as a video viewed from a position and orientation that does not match the position and orientation of the first video or the second video.

[0309] Clause 94. A system for performing orthodontic treatment as described in Clause 82, wherein the instructions, when executed by the processor, further cause the system to: access an orthodontic treatment plan to move a patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan comprising a plurality of treatment phases, each of the plurality of treatment phases being associated with a corresponding dental arrangement of the patient's teeth; and evaluate the progress of the orthodontic treatment plan by comparing the novel image data with one or more of the first or second image data used to generate the novel image data.

[0310] Clause 95. A system for performing orthodontic treatment as described in Clause 82, wherein novel image data of the teeth in the first or second dental arrangement is generated based on one or more of the first or second image data, the novel image data representing the teeth in the first or second dental arrangement as viewed from a camera position and orientation different from that of the one or more first or second image data, further employing one or more of the following: neural radiation field, Gaussian splashing, intelligent image stitching, Gaussian splashing, or a finely tuned diffusion model.

[0311] Clause 96. A system for performing orthodontic treatment as described in Clause 83, wherein the first time is before the implementation of the orthodontic treatment plan, and the second time is after at least a portion of the implementation of the orthodontic treatment plan.

[0312] Clause 97. A method for performing orthodontic treatment, comprising: capturing one or more first image data of the patient's teeth at a first time when the patient's teeth are in a first dental arrangement, the first image data having been captured from one or more first camera positions, orientations, or settings; capturing one or more second image data of the patient's teeth at a second time when the patient's teeth are in a second dental arrangement, the second time being different from the first time, the second image data having been captured from one or more second camera positions, orientations, or settings; transmitting the first image data and the second image data to a remote server device for processing; and receiving novel image data of the teeth in the first dental arrangement and the second dental arrangement from the remote server device, wherein the novel image data is generated based on the first image data or the second image data, the novel image data representing the teeth in the first dental arrangement or the second dental arrangement as viewed from a camera position, orientation, or setting different from the one or more first image data or second image data.

[0313] Clause 98. The method of Clause 97, wherein the novel image data and one or more of the first or second image data used to generate the novel image data are simultaneously displayed.

[0314] Clause 99. The method according to Clause 98, wherein the novel image data is generated as an image viewed from a position and orientation that matches the position and orientation of one or more of the first image data or one or more of the second image data.

[0315] Clause 100. The method according to Clause 97, wherein the first time is before the implementation of the orthodontic treatment plan, and the second time is after at least a portion of the implementation of the orthodontic treatment plan.

[0316] Clause 101. The method according to Clause 97, wherein the different camera positions and orientations correspond to the position and orientation of one of the image data in the first image data or the second image data.

[0317] The software components, processes, or functions disclosed and / or described in this application can be implemented as software code to be executed by a processor using conventional or object-oriented techniques using a suitable computer language such as Python, Java, JavaScript, C, C++, or Perl. The software code can be stored as a series of instructions or commands in (or on) a non-transitory computer-readable medium such as random access memory (RAM), read-only memory (ROM), magnetic media such as a hard disk drive, or optical media such as a CD-ROM. In this context, a non-transitory computer-readable medium is a medium suitable for storing data or instructions other than transient waveforms. Such a computer-readable medium can reside on or within a single computing device and can exist on or within different computing devices within a system or network.

[0318] According to one example implementation, the processing element or processor, as used herein, can be a central processing unit (CPU) or conceptualized as a CPU (such as a virtual machine). In this example implementation, the CPU or a device in which the CPU can be incorporated can be coupled, connected, and / or communicate with one or more peripheral devices (such as a display). In another example implementation, the processing element or processor can be incorporated into a mobile computing device (such as a smartphone or tablet).

[0319] Non-transitory computer-readable storage media as referred to herein may include a number of physical drive units, such as redundant arrays of independent disks (RAID), flash memory, USB flash drives, external hard drives, thumb drives, pen drives, key drives, high-density digital versatile optical disc (HD-DVD) drives, internal hard drives, Blu-ray disc drives or holographic digital data storage (HDDS) drives, synchronous dynamic random access memory (SDRAM), or similar devices or memory forms based on similar technologies. Such computer-readable storage media allow processing elements or processors to access computer-executable processing steps and applications stored on removable and non-removable storage media to unload data from or upload data to a device. As mentioned, with respect to the embodiments disclosed and / or described herein, non-transitory computer-readable media may include structures, techniques, or methods other than transient waveforms or similar media.

[0320] Example embodiments of this disclosure are described herein with reference to system block diagrams and / or flowcharts or process diagrams of functions, operations, processes, or methods. One or more blocks in a block diagram, or one or more stages or steps in a flowchart or process diagram, and combinations of blocks in a block diagram and combinations of stages or steps in a flowchart or process diagram, may be implemented by computer-executable program instructions. In some embodiments, one or more blocks, or stages or steps, may not necessarily need to be performed in the presented order, or may not need to be performed at all.

[0321] Specific examples of computer-executable program instructions that can be loaded onto a general-purpose computer, special-purpose computer, processor, or other programmable data processing apparatus to produce a machine. Instructions that execute on a computer, processor, or other programmable data processing apparatus create components for implementing one or more of the functions, operations, processes, or methods disclosed and / or described herein. Computer program instructions can be stored in (or on) a computer-readable storage medium that can direct a computer or other programmable data processing apparatus to operate in a particular manner such that the instructions stored in (or on) the computer-readable storage medium produce an article of art comprising instruction components that, when executed, implement one or more of the functions, operations, processes, or methods disclosed and / or described herein.

[0322] While embodiments of the present disclosure have been described in conjunction with what are currently considered to be the most practical approaches and techniques, the embodiments are not limited to the disclosed implementations. Rather, the disclosed implementations are intended to include and cover modifications and equivalent arrangements encompassed within the scope of the appended claims. Although specific terminology has been used herein, it is used in a general and descriptive sense only and not for limiting purposes.

[0323] This written description uses examples to illustrate one or more embodiments of this disclosure and enables those skilled in the art to practice the disclosed methods and techniques, including making and using the apparatus or system and performing related methods. The patentability scope of this disclosure is defined by the claims and may include other examples that would occur to those skilled in the art. Other examples are intended to be within the scope of the claims if they have the same structural and / or functional elements as the literal description of the claims or if they include structural and / or functional elements that are slightly different from the literal description of the claims.

[0324] All references cited herein, including publications, patent applications and patents, are incorporated herein by reference to the same extent that each reference is individually and specifically indicated to be incorporated by reference and / or fully elaborated herein.

[0325] Unless otherwise indicated herein or clearly contradicted by the context, the terms “a” and “an” and “the” and similar references used in the specification and claims shall be interpreted to cover both the singular and plural. Unless otherwise indicated, the terms “having,” “including,” “containing,” and similar references in the specification and claims shall be interpreted as open-ended terms (e.g., meaning “including but not limited to”).

[0326] Unless otherwise indicated herein, statements of value ranges herein are intended as shorthand for each individual value that inclusively falls within the range, and each individual value is incorporated into the specification as it is individually stated herein. Unless otherwise indicated herein or otherwise obviously contradicted by the context, the method steps or stages disclosed and / or described herein may be performed in any suitable order.

[0327] Unless otherwise indicated, the use of examples or exemplary language (e.g., "such as") in this document is intended to better illustrate embodiments of this disclosure and does not limit the scope of the claims. No language in the specification should be construed as indicating that any unclaimed element is essential to every embodiment of this disclosure.

[0328] As used herein (i.e., in the claims, drawings, and description), the term “or” is used inclusively to refer to alternative and combination items.

[0329] Different arrangements of elements, structures, components, or steps illustrated in the accompanying drawings or described herein, as well as components and steps not shown or described, are possible. Similarly, some features and sub-combinations are useful and can be employed without reference to other features and sub-combinations. Embodiments have been described for illustrative and non-limiting purposes, and alternative embodiments may become apparent to the reader of this specification. Therefore, this disclosure is not limited to the embodiments described in the specification or depicted in the accompanying drawings, and modifications can be made without departing from the scope of the appended claims.

[0330] This document describes in detail one or more embodiments of the disclosed subject matter to satisfy legal requirements, but such description does not limit the scope of the claims. The claimed subject matter may be embodied in other ways, may include different elements or steps, and may be used in combination with other prior art or subsequently developed techniques. Unless the order of the steps or the arrangement of the elements is expressly indicated as required, this description should not be construed as implying any desired order or arrangement of the steps or elements in or between.

[0331] Embodiments of this disclosure are described more fully herein with reference to the accompanying drawings, which form part of this disclosure and illustrate exemplary embodiments that can be used to practice this disclosure. However, this disclosure may be embodied in different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to satisfy legal requirements and to communicate the scope of this disclosure to those skilled in the art.

[0332] The subject matter of this disclosure may be embodied in whole or in part as a system, one or more methods, or one or more devices. Embodiments may take the form of hardware implementations, software implementations, or a combination of software and hardware aspects. For example, in some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented by one or more suitable processing elements (such as processors, microprocessors, coprocessors, CPUs, GPUs, TPUs, QPUs, or controllers, as non-limiting examples) that are part of a client device, server, network element, remote platform (such as a SaaS platform), “cloud” service, or other form of computing or data processing system, device, or platform.

[0333] One or more processing elements can be programmed using a set of executable instructions (e.g., software instructions), which can be stored in (or on) one or more suitable non-transitory data storage elements. In some embodiments, the instruction set can be delivered to a user via an application that transmits or executes the instruction set (e.g., via a network, such as the Internet). In some embodiments, end users can utilize the instruction set or application by accessing a SaaS platform or services provided through such a platform.

[0334] In some embodiments, the systems and methods disclosed herein may be provided through a SaaS or multi-tenant platform. The platform provides access to multiple entities, each with a separate account associated with a data storage device. For example, each account may correspond to a dentist or orthodontist, a patient, an entity, a set or class of entities, a set or class of patients, an insurance company, or an organization. Each account may access one or more services, wherein the service set is instantiated within its account, and implements one or more methods or functions disclosed and / or described herein.

[0335] In some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented in dedicated hardware, such as programmable gate arrays, application-specific integrated circuits (ASICs), etc. Note that embodiments of the methods of the present invention may be implemented as an application program, a subroutine as part of a larger application program, a "plugin," a functional extension of a data processing system or platform, or other suitable form. Therefore, the following detailed description should not be considered limiting.

Claims

1. A system for performing orthodontic treatment, comprising: processor; as well as The memory includes instructions that, when executed by the processor, cause the system to: Access one or more first image data of the patient's teeth captured at a first moment when the patient's teeth are in the first dental arrangement, the first image data having been captured from one or more first camera positions, orientations, or settings; Access one or more second image data of the patient's teeth captured at a second time when the patient's teeth are in a second dental arrangement, the second time being different from the first time, the second image data having been captured from one or more second camera positions, orientations, or settings; as well as Based on one or more of the first image data or the second image data, novel image data is generated of the teeth in the first dental arrangement or the second dental arrangement, the novel image data representing the teeth in the first dental arrangement or the second dental arrangement as viewed from a camera position, orientation, or setting different from that of the one or more first image data or the second image data.

2. The system for performing orthodontic treatment according to claim 1, wherein the instructions, when executed by the processor, further cause the system to: Receive an orthodontic treatment plan to move a patient's teeth from a first dental arrangement toward a second dental arrangement, the orthodontic treatment plan comprising multiple treatment phases, each of the multiple treatment phases comprising a corresponding dental arrangement associated with the patient's teeth.

3. The system for performing orthodontic treatment according to claim 1, wherein the first image data and the second image data are a first still image and a second still image.

4. The system for performing orthodontic treatment according to claim 3, wherein the novel image data is a new still image.

5. The system for performing orthodontic treatment according to claim 4, wherein the instructions, when executed by the processor, further cause the system to: Simultaneously display the new still image and one or more of the first or second still images used to generate the new still image.

6. The system for performing orthodontic treatment according to claim 5, wherein the new still image is generated as an image viewed from a position and orientation that matches the position and orientation of one of the first still image or the second still image.

7. The system for performing orthodontic treatment according to claim 5, wherein the new still image is generated as an image viewed from a position and orientation that does not match the position and orientation of the first still image or the second still image.

8. The system for performing orthodontic treatment according to claim 1, wherein the first image data and the second image data are a first video and a second video.

9. The system for performing orthodontic treatment according to claim 8, wherein the novel image data is a new video.

10. The system for performing orthodontic treatment according to claim 9, wherein the instructions, when executed by the processor, further cause the system to: Simultaneously display the new video as well as one or more videos from the first or second video used to generate the new video.

11. The system for performing orthodontic treatment according to claim 10, wherein the new video is generated as a video viewed from a position and orientation matching the position and orientation of the first video or the second video.

12. The system for performing orthodontic treatment according to claim 10, wherein the new video is generated as a video viewed from a position and orientation that does not match the position and orientation of the first video or the second video.

13. The system for performing orthodontic treatment according to claim 1, wherein the instructions, when executed by the processor, further cause the system to: Accessing an orthodontic treatment plan to move the patient's teeth from a first dental arrangement to a second dental arrangement, the orthodontic treatment plan comprising multiple treatment phases, each of which is associated with a corresponding dental arrangement of the patient's teeth; and The progress of the orthodontic treatment plan is evaluated by comparing the novel image data with one or more of the first or second image data used to generate the novel image data.

14. The system for performing orthodontic treatment according to claim 1, wherein the novel image data of the teeth in the first or second dental arrangement is generated based on one or more of the first or second image data, the novel image data representing the teeth in the first or second dental arrangement as viewed from a camera position and orientation different from that of the one or more first or second image data, further using one or more of the following: neural radiation field, Gaussian splashing, intelligent image stitching, Gaussian splashing, or a finely tuned diffusion model.

15. The system for performing orthodontic treatment according to claim 2, wherein the first time is before the orthodontic treatment plan is implemented, and the second time is after at least a portion of the orthodontic treatment plan is implemented.

16. A method for performing orthodontic treatment, comprising: Capture one or more first image data of the patient's teeth at a first moment when the patient's teeth are in the first dental arrangement, the first image data having been captured from one or more first camera positions, orientations, or settings; Capture one or more second image data of the patient's teeth at a second time when the patient's teeth are in a second dental arrangement, the second time being different from the first time, the second image data having been captured from one or more second camera positions, orientations, or settings; The first image data and the second image data are sent to a remote server device for processing. as well as The remote server device receives novel image data of the tooth located in the first dental arrangement or the second dental arrangement, wherein the novel image data is generated based on the first image data and the second image data, and the novel image data represents the tooth located in the first dental arrangement or the second dental arrangement viewed from a camera position, orientation, or setting different from the camera position, orientation, or setting of the one or more first image data or the second image data.

17. The method of claim 16, wherein the novel image data and one or more of the first or second image data used to generate the novel image data are simultaneously displayed.

18. The method of claim 17, wherein the novel image data is generated as an image viewed from a position and orientation that matches the position and orientation of one or more of the first image data or one or more of the second image data.

19. The method of claim 16, wherein the first time is before the orthodontic treatment plan is implemented, and the second time is after at least a portion of the orthodontic treatment plan is implemented.

20. The method of claim 16, wherein the different camera positions and orientations correspond to the position and orientation of one of the image data in the first image data or the second image data.

Images